On Tue, 28 Apr 2015, [email protected] wrote:
Hi David,
Thanks for your response.
I have several distributed virtualized rsyslog servers with the same
configuration.
On all servers I have an undeterministical dying of rsyslog between once a day
and once a week.
Messages in kernel ringbuffer (dmesg) are:
INFO: task rs:main Q:Reg:2614 blocked for more than 120 seconds.
or
rs:main Q:Reg D 0000000000000000 0 2614 1 0x00000000
or
rs:main Q:Reg[19176]: segfault at 0 ip 00007f9e2c5e492a sp 00007f9e284fd418
error 4 in libc-2.12.so[7f9e2c565000+18a000]
or
rs:main Q:Reg[12532]: segfault at 7f2d00534c5a ip 00007f2d2b95f92a sp
00007f2d27878418 error 4 in libc-2.12.so[7f2d2b8e0000+18a000]
VMCIUtil: Updating context id from 0x694633da to 0x694633da on event 0.
This sounds like a system level issue (something is preventing rsyslog from
running for too long)
OK. Maybe the issue is related to the TCP message forwarding?
My OS versions were between red-hat 6.4 and 6.6.
no, this looks like a kernel scheduler bug. Since you are running RHEL, I would
suggest that you reach out to RedHat support, at least for the first "blocked
for more than 120 seconds" bug, adn probably for the others as well
Configuration looks like this:
----------------------------------------------------------------------
Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1" KeepAlive.Interval="2"
KeepAlive.Time="20" MaxSessions="5000")
Module (load="imudp")
Module (load="omudpspoof")
$MaxOpenFiles 9000
lookup_table(name="lookuptable" file="rsyslog.lookup")
set $!dst = lookup("lookuptable", $fromhost-ip);
$template raw,"%rawmsg%"
$template rel,"%fromhost% %fromhost-ip% %rawmsg%\n"
ruleset(name="typea"){
action (type="omudpspoof" target="loghost" port="514" template="raw")
}
I think this is invalid. the result is not a RFC compliant syslog message. It's
likely that whatever is recieving the logs from this is going to have some
problems.
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omudpspoof.html
What exactly do you think is invalid?
Actually except the described problems most the time everything works as I
expect it to do.
omudpspoof forwards the tcp packages internally to udp listener, so spoofing
works here fine.
oops, I was seeing this as the rel template, not the raw template. you should be
able to leave out the template and rsyslog will format the message and send it.
if then
stop
if then
stop
will work just as well
Thanks :)
Sometimes (unexpectedly when) I get on a chained rsyslog-server logevents like
this:
Original-Message: host1 1.2.3.4 Apr 21 11:39:43 host1 sshd[11600]: Accepted
publickey for user from 2.3.4.5 port 23869 ssh2
Message on changed rsyslog: .3.4 Apr 21 11:39:43 host1 sshd[11600]: Accepted
publickey for user from 2.3.4.5 port 23869 ssh2
Error-Log on chained system:
Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP message:
invalid octet count 0. [v8.8.0.ad1]
Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP message:
delimiter is not SP but has ASCII value 58. [v8.8.0.ad1]
which template is this using?
The first rsyslog server uses the config below, so template is: $template
rel,"%fromhost% %fromhost-ip% %rawmsg%\n"
The recieving rsyslog server uses this template and logs to file: $template
raw,"%rawmsg%"
Ok, if the sending server is using the template rel to send the message to the
second server you have a problem (this is what I thought was happening with the
omspoof config above)
a valid syslog message is
<###>timestamp hostname syslogtag[pid]: message
you are sending
hostname ip <###>timestamp hostname syslogtag[pid]: message
the receiving message is going to try to 'do the right thing' and correct for
the malformed message, but it's unlikely to be right all the time.
the fact that you are getting framing errors indicates that the sending server
is doing something very wrong
Do you have any idear / debugging concept?
In my lab, everything seems to be fine, so I see only the option to test in
production, what i definitively don't want to do...
check that you have the same version and gnutls version on both sender and
receiver.
I do not use TLS at all.
a good debugging tool is to write the files locally with the same format that
you use to send them remotely, that way if something wierd happens, you can look
at what was sent.
Yes, I was also thinking about this. Is there any tool/script that I can use to
"replay" stored messages
tcpreplay is a good tool for this.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
