On Wednesday, August 06, 2014 06:24:01 PM David Lang wrote:
> Many of these are good ideas, several of them have been raised before,
there
> are far more ideas that we know are good than time to implement them.
>
> On Tue, 5 Aug 2014, Eugene Istomin wrote:
> > Hello Rainer,
> >
> > day-to-day we are moving to high-performance flexible log-
parsing/delivery
> > solution, thanks to Rsyslog :) I have some feature ideas:
> >
> >
> > *I. Multi-line normalization*
> >
> > Proc filesystem have well-known multi-line syntax:
> >
> > for ex.,
> > cat /proc/meminfo
> > MemTotal: 4010180 kB
> > MemFree: 1396880 kB
> > Buffers: 87444 kB
> > Cached: 1377868 kB
> > SwapCached: 5736 kB
> > ...
> >
> > Easy to see the key/value tokens standing per-line. Could Rsyslog &
> > normalization be able to work this way:
> >
> > 1) open /proc/meminfo (how many bytes to read may be im(file?)
parameter)
> > 2) mmnormalize will parse multi-line string & makes JSON according to
> > rulebase variables
> imfile already has the ability to read a couple multi-line formats (every
> line after the first starts with whitespace and blank line between logs)
>
> tcp octet mode has the ability to deliver message containing embedded
> newlines from one machine to another
>
> But overall, rsyslog (like all log processing software) has a lot of places
> where it's assumed that each line is a separate log.
>
> Handling a multi-line log like this gets tricky, how do you tell where one
> log ends and the next begins? can you conclude that a log has finished
> being written and can be processed before the next log is written? If the
> log is written infrequently, this can be a _really_ long delay, so which
> timestamp do you put on the log message, when it starts or when it
'ends'?
>
> In general, your best bet is to have an external program scrape your
> multi-line input and convert it to a single line before handing it to
> rsyslog (escape the newlines, tabs, and other special characters so that
> you can get them back if needed)
Ok, clear. If so, this preparser also could create CEE JSON.
>
> > *II. Adding a mmnormalize char-count field type*
> >
> > Let's parse DRBD status string:
> > 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
> >
> > There are special flags in the end of DRBD status line: "r-----", each
> > flag use only one character. Is it possible to add a new field type with
> > ability to match not by delimiter but by chars count:
> >
> > Field type: 'char-count'
> > Matches: One or more characters, up to the
lenth given in extra data.
> > Extra data: Number
> > Example: %field_name:char-count:1%
>
> I think this is a great idea that I haven't seen mentioned before.
Thanks, this field type is really needed.
>
> > *III. External field mapping*
> >
> > Let we have a JSON:
> > {
> >
> > "acl_history": " PTR != HELO=+30; SPAMD=-2.5;",
> > "acl_reason": "Accepted with score 30",
> > "message_id": "1XEZGA-0006zm-Lb",
> >
> > }
> >
> > After mmjsonparse in some syslog gate we can use JSON data as
$!VARs.
> > Would be cool if this gate would be able to map one keys to another:
> >
> > acl_history => CmdbEximAclHistory
> > acl_reason => CmdbEximAclReason
> > message_id => CmdbEximMessageId
> >
> > {
> >
> > "CmdbEximAclHistory": " PTR != HELO=+30; SPAMD=-2.5;",
> > "CmdbEximAclReason": "Accepted with score 30",
> > "CmdbEximMessageId": "1XEZGA-0006zm-Lb"
> >
> > }
> >
> > Mapping can be stored in external file.
> > This feature leads to simplify code mapping in loganalyse side.
>
> why not just change your parser?
>
> Or, you can do
> set $!CmdbEximAclHistory = $!acl_history;
> unset $!acl_history;
>
> I think the mapping file adds significant complexity without a lot of value.
We are thinking about Scala variables naming in our loggitoring
application, this mapping is one of approaches we discussing about. Yes,
others are "set"/"unset" and Scala-side vars remap.
> > *IV. Dynamic variables in actions/rulesets*
> >
> > The mail idea is to use variables from parsed JSON, like
> >
> > ## DEFAULT ES INDEX ##
> > template(name="es_index-srvdate" type="list") {
> >
> > property(name="$!msg_class" )
> >
> > constant(value="-")
> >
> > property(name="$!msg_view" )
> > constant(value="-")
> >
> > property(name="$year")
> >
> > constant(value=".")
> >
> > property(name="$month")
> > constant(value=".")
> >
> > property(name="$day")
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
