I think this would better be addressed inside the kernel. They need to define a precise format so that one knows how to match messages. IF that's not present, I think the results can be very surprising.
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Monday, July 11, 2011 9:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] combining multiline kernel messages into a > single message > > >I am just starting down this road, so please forgive my ignorance and > >any ill-conceived assumptions. > > > >I want to centralize logging of multiple hosts to a single host. > There > >is one artifact of doing so (and in fact it's not even particular to > >forwarding -- it seems to happen on a single node) that I want to > >resolve and that's the intermingling of log messages in the middle of > >what should be multiline kernel messages. Think stack traces. > > > >The kernel dumps a few dozen lines onto /dev/kmsg which in reality > >represent a single messages. It happens with the OOM killer but > >probably, the most common case is stack traces, which each line in the > >trace is logged as a separate syslog line with a date and time and > host > >stamp, etc. > > > >The problem is that it's possible for other messages to be printed in > >between these multiple lines/messages from the kernel. I'd like to > try > >to atomize these kernel messages so that they are guaranteed to be > >together in the log without other messages interspersed in them. > > > >Is this something that's realistic to achieve or is it fraught with to > >many problems to be able to do reliably? > > we do have some logic in the imfile to allow it to combine multiple > line > logs into one log entry, it has a problem right now that it doesn't > sanitize the newlines so when you forward it things will probably get > broken up again. > > similar logic could probably be added to the kmesg input, but it is > going > to be tricky. > > the problem that you have is how to clearly identify the beginning of a > new message. in imfile it can do three things > > 1. each line is a separate log > > 2. if a line starts with whitespace, it's part of a earlier log entry > > 3. log entries are separated by a blank line > > now, #2 and #3 both mean that the log entry cannot be sent until the > next > log entry arrives (otherwise you don't know if the next data to appear > is > part of the log entry you are working on or not) > > with messages from the kernel, if they have timestamps on the front of > them added by the kernel, it is going to be hard to tell if it is a > continuation or not, but without those timestampes (or if you teach > kmesg > input to skip them), I think that approach #2 will work. > > I would suggest dumping dmesg output from a few systems and try running > them through the imfile mode #2 and see how the results look. > > I think that the kernel is pretty consistant about having every new > message start at the beginning of a line, but I'm not sure that it > indents > everything (especially oops and stack trace messages) > > even if it doesn't indent them, you could add logic to the kmesg input > module to watch for, and combine the multi-line log messages, but that > could become very messy (and change from kernel version to kernel > version), if the simple indentation approach does not work, I would > suggest asking on the kernel mailing list before starting any other > work. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
