Many of these are good ideas, several of them have been raised before, there are
far more ideas that we know are good than time to implement them.
On Tue, 5 Aug 2014, Eugene Istomin wrote:
Hello Rainer,
day-to-day we are moving to high-performance flexible log-parsing/delivery
solution, thanks to Rsyslog :)
I have some feature ideas:
*I. Multi-line normalization*
Proc filesystem have well-known multi-line syntax:
for ex.,
cat /proc/meminfo
MemTotal: 4010180 kB
MemFree: 1396880 kB
Buffers: 87444 kB
Cached: 1377868 kB
SwapCached: 5736 kB
...
Easy to see the key/value tokens standing per-line. Could Rsyslog &
normalization be able to work this way:
1) open /proc/meminfo (how many bytes to read may be im(file?) parameter)
2) mmnormalize will parse multi-line string & makes JSON according to rulebase
variables
imfile already has the ability to read a couple multi-line formats (every line
after the first starts with whitespace and blank line between logs)
tcp octet mode has the ability to deliver message containing embedded newlines
from one machine to another
But overall, rsyslog (like all log processing software) has a lot of places
where it's assumed that each line is a separate log.
Handling a multi-line log like this gets tricky, how do you tell where one log
ends and the next begins? can you conclude that a log has finished being written
and can be processed before the next log is written? If the log is written
infrequently, this can be a _really_ long delay, so which timestamp do you put
on the log message, when it starts or when it 'ends'?
In general, your best bet is to have an external program scrape your multi-line
input and convert it to a single line before handing it to rsyslog (escape the
newlines, tabs, and other special characters so that you can get them back if
needed)
*II. Adding a mmnormalize char-count field type*
Let's parse DRBD status string:
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
There are special flags in the end of DRBD status line: "r-----", each flag use
only one character.
Is it possible to add a new field type with ability to match not by delimiter
but by chars count:
Field type: 'char-count'
Matches: One or more characters, up to the lenth given in extra
data.
Extra data: Number
Example: %field_name:char-count:1%
I think this is a great idea that I haven't seen mentioned before.
*III. External field mapping*
Let we have a JSON:
{
"acl_history": " PTR != HELO=+30; SPAMD=-2.5;",
"acl_reason": "Accepted with score 30",
"message_id": "1XEZGA-0006zm-Lb",
}
After mmjsonparse in some syslog gate we can use JSON data as $!VARs.
Would be cool if this gate would be able to map one keys to another:
acl_history => CmdbEximAclHistory
acl_reason => CmdbEximAclReason
message_id => CmdbEximMessageId
{
"CmdbEximAclHistory": " PTR != HELO=+30; SPAMD=-2.5;",
"CmdbEximAclReason": "Accepted with score 30",
"CmdbEximMessageId": "1XEZGA-0006zm-Lb"
}
Mapping can be stored in external file.
This feature leads to simplify code mapping in loganalyse side.
why not just change your parser?
Or, you can do
set $!CmdbEximAclHistory = $!acl_history;
unset $!acl_history;
I think the mapping file adds significant complexity without a lot of value.
*IV. Dynamic variables in actions/rulesets*
The mail idea is to use variables from parsed JSON, like
## DEFAULT ES INDEX ##
template(name="es_index-srvdate" type="list") {
property(name="$!msg_class" )
constant(value="-")
property(name="$!msg_view" )
constant(value="-")
property(name="$year")
constant(value=".")
property(name="$month")
constant(value=".")
property(name="$day")
}
Right now is impossible to use dynamic variables as action target/port/etc:
Work:
ruleset(name="relp_cee" queue.filename="relp_cee" ) {
action(type="omrelp" Template="cee" Target="core" Port="20514")
}
Does not work :
ruleset(name="relp_cee" queue.filename="$!_msg_template" ) {
action(type="omrelp" Template="$!_msg_template" Target="$!_msg_next_ip"
Port="$!_msg_next_port")
}
This has been requested many times, the problem is that evaluating the variable
every time for all of these fields would be a very significant slowdown as well
as adding a lot of complexity.
You have to ask, is it really that useful to have the these things be variables?
how many different ones are you going to have? and why would you change them?
It may be better to have multiple rulesets, each differing only by name and
destination and select between which ruleset to run based on a if..then..else
tree instead (remember that if the rulesets are defined in external files and
then included, they can be auto-generated from a template, they just would
require a restart to take effect).
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
