I am just starting down this road, so please forgive my ignorance and
any ill-conceived assumptions.
I want to centralize logging of multiple hosts to a single host. There
is one artifact of doing so (and in fact it's not even particular to
forwarding -- it seems to happen on a single node) that I want to
resolve and that's the intermingling of log messages in the middle of
what should be multiline kernel messages. Think stack traces.
The kernel dumps a few dozen lines onto /dev/kmsg which in reality
represent a single messages. It happens with the OOM killer but
probably, the most common case is stack traces, which each line in the
trace is logged as a separate syslog line with a date and time and host
stamp, etc.
The problem is that it's possible for other messages to be printed in
between these multiple lines/messages from the kernel. I'd like to try
to atomize these kernel messages so that they are guaranteed to be
together in the log without other messages interspersed in them.
Is this something that's realistic to achieve or is it fraught with to
many problems to be able to do reliably?
we do have some logic in the imfile to allow it to combine multiple line
logs into one log entry, it has a problem right now that it doesn't
sanitize the newlines so when you forward it things will probably get
broken up again.
similar logic could probably be added to the kmesg input, but it is going
to be tricky.
the problem that you have is how to clearly identify the beginning of a
new message. in imfile it can do three things
1. each line is a separate log
2. if a line starts with whitespace, it's part of a earlier log entry
3. log entries are separated by a blank line
now, #2 and #3 both mean that the log entry cannot be sent until the next
log entry arrives (otherwise you don't know if the next data to appear is
part of the log entry you are working on or not)
with messages from the kernel, if they have timestamps on the front of
them added by the kernel, it is going to be hard to tell if it is a
continuation or not, but without those timestampes (or if you teach kmesg
input to skip them), I think that approach #2 will work.
I would suggest dumping dmesg output from a few systems and try running
them through the imfile mode #2 and see how the results look.
I think that the kernel is pretty consistant about having every new
message start at the beginning of a line, but I'm not sure that it indents
everything (especially oops and stack trace messages)
even if it doesn't indent them, you could add logic to the kmesg input
module to watch for, and combine the multi-line log messages, but that
could become very messy (and change from kernel version to kernel
version), if the simple indentation approach does not work, I would
suggest asking on the kernel mailing list before starting any other work.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
