2015-04-28 18:45 GMT+02:00 David Lang <[email protected]>: > On Tue, 28 Apr 2015, [email protected] wrote: > >> Hi David, >> >> Thanks for your response. >> >>>> I have several distributed virtualized rsyslog servers with the same >>>> configuration. >>>> On all servers I have an undeterministical dying of rsyslog between once >>>> a day and once a week. >>>> >>>> Messages in kernel ringbuffer (dmesg) are: >>>> INFO: task rs:main Q:Reg:2614 blocked for more than 120 seconds. >>>> or >>>> rs:main Q:Reg D 0000000000000000 0 2614 1 0x00000000 >>>> or >>>> rs:main Q:Reg[19176]: segfault at 0 ip 00007f9e2c5e492a sp >>>> 00007f9e284fd418 error 4 in libc-2.12.so[7f9e2c565000+18a000] >>>> or >>>> rs:main Q:Reg[12532]: segfault at 7f2d00534c5a ip 00007f2d2b95f92a sp >>>> 00007f2d27878418 error 4 in libc-2.12.so[7f2d2b8e0000+18a000] >>>> VMCIUtil: Updating context id from 0x694633da to 0x694633da on event 0. >>> >>> >>> This sounds like a system level issue (something is preventing rsyslog >>> from >>> running for too long) >> >> >> OK. Maybe the issue is related to the TCP message forwarding? >> My OS versions were between red-hat 6.4 and 6.6. > > > no, this looks like a kernel scheduler bug. Since you are running RHEL, I > would suggest that you reach out to RedHat support, at least for the first > "blocked for more than 120 seconds" bug, adn probably for the others as well > >> >>>> Configuration looks like this: >>>> ---------------------------------------------------------------------- >>>> Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1" >>>> KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000") >>>> Module (load="imudp") >>>> Module (load="omudpspoof") >>>> $MaxOpenFiles 9000 >>>> lookup_table(name="lookuptable" file="rsyslog.lookup") >>>> set $!dst = lookup("lookuptable", $fromhost-ip); >>>> $template raw,"%rawmsg%" >>>> $template rel,"%fromhost% %fromhost-ip% %rawmsg%\n" >>>> >>>> ruleset(name="typea"){ >>>> action (type="omudpspoof" target="loghost" port="514" template="raw") >>>> } >>> >>> >>> I think this is invalid. the result is not a RFC compliant syslog >>> message. It's >>> likely that whatever is recieving the logs from this is going to have >>> some >>> problems. >>> >>> >>> http://www.rsyslog.com/doc/v8-stable/configuration/modules/omudpspoof.html >> >> >> What exactly do you think is invalid? >> Actually except the described problems most the time everything works as I >> expect it to do. >> omudpspoof forwards the tcp packages internally to udp listener, so >> spoofing works here fine. > > > oops, I was seeing this as the rel template, not the raw template. you > should be able to leave out the template and rsyslog will format the message > and send it. > >> >> >>> if then >>> stop >>> if then >>> stop >>> >>> will work just as well >> >> Thanks :) >> >>>> >>>> Sometimes (unexpectedly when) I get on a chained rsyslog-server >>>> logevents like this: >>>> Original-Message: host1 1.2.3.4 Apr 21 11:39:43 host1 sshd[11600]: >>>> Accepted publickey for user from 2.3.4.5 port 23869 ssh2 >>>> Message on changed rsyslog: .3.4 Apr 21 11:39:43 host1 sshd[11600]: >>>> Accepted publickey for user from 2.3.4.5 port 23869 ssh2 >>>> >>>> Error-Log on chained system: >>>> Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP >>>> message: invalid octet count 0. [v8.8.0.ad1] >>>> Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP >>>> message: delimiter is not SP but has ASCII value 58. [v8.8.0.ad1] >> >> >>> >>> which template is this using? >> >> The first rsyslog server uses the config below, so template is: $template >> rel,"%fromhost% %fromhost-ip% %rawmsg%\n" >> The recieving rsyslog server uses this template and logs to file: >> $template raw,"%rawmsg%" > > > Ok, if the sending server is using the template rel to send the message to > the second server you have a problem (this is what I thought was happening > with the omspoof config above) > > a valid syslog message is > <###>timestamp hostname syslogtag[pid]: message > > you are sending > > hostname ip <###>timestamp hostname syslogtag[pid]: message > > the receiving message is going to try to 'do the right thing' and correct > for the malformed message, but it's unlikely to be right all the time. > > the fact that you are getting framing errors indicates that the sending > server is doing something very wrong >
I guess there is a numeric hostname. That would trigger octet-counted framing, and that in turn could trigger the error message. Rainer >>>> Do you have any idear / debugging concept? >>>> In my lab, everything seems to be fine, so I see only the option to test >>>> in production, what i definitively don't want to do... >>> >>> >>> check that you have the same version and gnutls version on both sender >>> and >>> receiver. >> >> I do not use TLS at all. >> >>> a good debugging tool is to write the files locally with the same format >>> that >>> you use to send them remotely, that way if something wierd happens, you >>> can look >>> at what was sent. >> >> Yes, I was also thinking about this. Is there any tool/script that I can >> use to "replay" stored messages > > > tcpreplay is a good tool for this. > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
