On 09/06/2018 11:57 AM, David Lang wrote:
On Thu, 6 Sep 2018, Rich Megginson wrote:
That is, if multiline specifically means records like this:
first line of recordA
recordA second line
recordA third line
....
first line of recordB
Perhaps Noriko and I mean something slightly different by "multiline" - a single message spread out in some way across multiple lines in a log file, not distinguished by the characters at
the start of the line.
There's already the aforementioned case of crio log file lines which are distinguished by
the F or the P in the "partial" field. docker json-file log files have a
similar problem:
{"time":"<TIMESTAMP_n>", "stream":"stdout", "log":"partial_log_0"}
{"time":"<TIMESTAMP_n+1>","stream":"stdout", "log":"partial_log_1"}
{"time":"<TIMESTAMP_n+2>", "stream":"stdout", "log":"rest_of_partial_log\n"}
In this case, the partial message is indicated by the absence of the trailing \n in the "log" string value, and the final part of the message has the trailing \n. I'm assuming we could
come up with an endmsg.regex value which would work in this case. Then we would get a single message string like this:
{"time":"<TIMESTAMP_n>", "stream":"stdout", "log":"partial_log_0"}{"time":"<TIMESTAMP_n+1>","stream":"stdout", "log":"partial_log_1"}{"time":"<TIMESTAMP_n+2>", "stream":"stdout",
"log":"rest_of_partial_log\n"}
Would we be able to use mmnormalize with repeat and foreach to parse this into
not on a single step, but what I would do is:
you have a repeating pattern, use the repeat option to parse those repeating
components into an array
Then you can use foreach on the array and work on the individual parts (which
would end up needing to be parsed as json in this example), and then append the
parts to one variable
using mmnormalize to create $.foo[]
set $.baz='';
foreach $.bar $.foo {
set $.baz=$.baz & $.bar
}
David Lang
Thanks.
Just to clarify - rsyslog does not currently support "multiline" docker json-file nor crio logs because there is no endmsg.regex - we would need to add support for that first in order to use
mmnormalize repeat and foreach as you have described above.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
