On Tue, 28 Apr 2015, Rainer Gerhards wrote:
2015-04-28 18:45 GMT+02:00 David Lang <[email protected]>:
On Tue, 28 Apr 2015, [email protected] wrote:
Hi David,
Thanks for your response.
I have several distributed virtualized rsyslog servers with the same
configuration.
On all servers I have an undeterministical dying of rsyslog between once
a day and once a week.
Messages in kernel ringbuffer (dmesg) are:
INFO: task rs:main Q:Reg:2614 blocked for more than 120 seconds.
or
rs:main Q:Reg D 0000000000000000 0 2614 1 0x00000000
or
rs:main Q:Reg[19176]: segfault at 0 ip 00007f9e2c5e492a sp
00007f9e284fd418 error 4 in libc-2.12.so[7f9e2c565000+18a000]
or
rs:main Q:Reg[12532]: segfault at 7f2d00534c5a ip 00007f2d2b95f92a sp
00007f2d27878418 error 4 in libc-2.12.so[7f2d2b8e0000+18a000]
VMCIUtil: Updating context id from 0x694633da to 0x694633da on event 0.
This sounds like a system level issue (something is preventing rsyslog
from
running for too long)
OK. Maybe the issue is related to the TCP message forwarding?
My OS versions were between red-hat 6.4 and 6.6.
no, this looks like a kernel scheduler bug. Since you are running RHEL, I
would suggest that you reach out to RedHat support, at least for the first
"blocked for more than 120 seconds" bug, adn probably for the others as well
Configuration looks like this:
----------------------------------------------------------------------
Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1"
KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000")
Module (load="imudp")
Module (load="omudpspoof")
$MaxOpenFiles 9000
lookup_table(name="lookuptable" file="rsyslog.lookup")
set $!dst = lookup("lookuptable", $fromhost-ip);
$template raw,"%rawmsg%"
$template rel,"%fromhost% %fromhost-ip% %rawmsg%\n"
ruleset(name="typea"){
action (type="omudpspoof" target="loghost" port="514" template="raw")
}
I think this is invalid. the result is not a RFC compliant syslog
message. It's
likely that whatever is recieving the logs from this is going to have
some
problems.
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omudpspoof.html
What exactly do you think is invalid?
Actually except the described problems most the time everything works as I
expect it to do.
omudpspoof forwards the tcp packages internally to udp listener, so
spoofing works here fine.
oops, I was seeing this as the rel template, not the raw template. you
should be able to leave out the template and rsyslog will format the message
and send it.
if then
stop
if then
stop
will work just as well
Thanks :)
Sometimes (unexpectedly when) I get on a chained rsyslog-server
logevents like this:
Original-Message: host1 1.2.3.4 Apr 21 11:39:43 host1 sshd[11600]:
Accepted publickey for user from 2.3.4.5 port 23869 ssh2
Message on changed rsyslog: .3.4 Apr 21 11:39:43 host1 sshd[11600]:
Accepted publickey for user from 2.3.4.5 port 23869 ssh2
Error-Log on chained system:
Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP
message: invalid octet count 0. [v8.8.0.ad1]
Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP
message: delimiter is not SP but has ASCII value 58. [v8.8.0.ad1]
which template is this using?
The first rsyslog server uses the config below, so template is: $template
rel,"%fromhost% %fromhost-ip% %rawmsg%\n"
The recieving rsyslog server uses this template and logs to file:
$template raw,"%rawmsg%"
Ok, if the sending server is using the template rel to send the message to
the second server you have a problem (this is what I thought was happening
with the omspoof config above)
a valid syslog message is
<###>timestamp hostname syslogtag[pid]: message
you are sending
hostname ip <###>timestamp hostname syslogtag[pid]: message
the receiving message is going to try to 'do the right thing' and correct
for the malformed message, but it's unlikely to be right all the time.
the fact that you are getting framing errors indicates that the sending
server is doing something very wrong
I guess there is a numeric hostname. That would trigger octet-counted
framing, and that in turn could trigger the error message.
or just no DNS for the hostname, so that field contains an IP address instead.
That makes a lot of sense.
David Lang
Rainer
Do you have any idear / debugging concept?
In my lab, everything seems to be fine, so I see only the option to test
in production, what i definitively don't want to do...
check that you have the same version and gnutls version on both sender
and
receiver.
I do not use TLS at all.
a good debugging tool is to write the files locally with the same format
that
you use to send them remotely, that way if something wierd happens, you
can look
at what was sent.
Yes, I was also thinking about this. Is there any tool/script that I can
use to "replay" stored messages
tcpreplay is a good tool for this.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
