2015-04-28 18:58 GMT+02:00 David Lang <[email protected]>: > On Tue, 28 Apr 2015, Rainer Gerhards wrote: > >> 2015-04-28 18:45 GMT+02:00 David Lang <[email protected]>: >>> >>> On Tue, 28 Apr 2015, [email protected] wrote: >>> >>>> Hi David, >>>> >>>> Thanks for your response. >>>> >>>>>> I have several distributed virtualized rsyslog servers with the same >>>>>> configuration. >>>>>> On all servers I have an undeterministical dying of rsyslog between >>>>>> once >>>>>> a day and once a week. >>>>>> >>>>>> Messages in kernel ringbuffer (dmesg) are: >>>>>> INFO: task rs:main Q:Reg:2614 blocked for more than 120 seconds. >>>>>> or >>>>>> rs:main Q:Reg D 0000000000000000 0 2614 1 0x00000000 >>>>>> or >>>>>> rs:main Q:Reg[19176]: segfault at 0 ip 00007f9e2c5e492a sp >>>>>> 00007f9e284fd418 error 4 in libc-2.12.so[7f9e2c565000+18a000] >>>>>> or >>>>>> rs:main Q:Reg[12532]: segfault at 7f2d00534c5a ip 00007f2d2b95f92a sp >>>>>> 00007f2d27878418 error 4 in libc-2.12.so[7f2d2b8e0000+18a000] >>>>>> VMCIUtil: Updating context id from 0x694633da to 0x694633da on event >>>>>> 0. >>>>> >>>>> >>>>> >>>>> This sounds like a system level issue (something is preventing rsyslog >>>>> from >>>>> running for too long) >>>> >>>> >>>> >>>> OK. Maybe the issue is related to the TCP message forwarding? >>>> My OS versions were between red-hat 6.4 and 6.6. >>> >>> >>> >>> no, this looks like a kernel scheduler bug. Since you are running RHEL, I >>> would suggest that you reach out to RedHat support, at least for the >>> first >>> "blocked for more than 120 seconds" bug, adn probably for the others as >>> well >>> >>>> >>>>>> Configuration looks like this: >>>>>> ---------------------------------------------------------------------- >>>>>> Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1" >>>>>> KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000") >>>>>> Module (load="imudp") >>>>>> Module (load="omudpspoof") >>>>>> $MaxOpenFiles 9000 >>>>>> lookup_table(name="lookuptable" file="rsyslog.lookup") >>>>>> set $!dst = lookup("lookuptable", $fromhost-ip); >>>>>> $template raw,"%rawmsg%" >>>>>> $template rel,"%fromhost% %fromhost-ip% %rawmsg%\n" >>>>>> >>>>>> ruleset(name="typea"){ >>>>>> action (type="omudpspoof" target="loghost" port="514" template="raw") >>>>>> } >>>>> >>>>> >>>>> >>>>> I think this is invalid. the result is not a RFC compliant syslog >>>>> message. It's >>>>> likely that whatever is recieving the logs from this is going to have >>>>> some >>>>> problems. >>>>> >>>>> >>>>> >>>>> http://www.rsyslog.com/doc/v8-stable/configuration/modules/omudpspoof.html >>>> >>>> >>>> >>>> What exactly do you think is invalid? >>>> Actually except the described problems most the time everything works as >>>> I >>>> expect it to do. >>>> omudpspoof forwards the tcp packages internally to udp listener, so >>>> spoofing works here fine. >>> >>> >>> >>> oops, I was seeing this as the rel template, not the raw template. you >>> should be able to leave out the template and rsyslog will format the >>> message >>> and send it. >>> >>>> >>>> >>>>> if then >>>>> stop >>>>> if then >>>>> stop >>>>> >>>>> will work just as well >>>> >>>> >>>> Thanks :) >>>> >>>>>> >>>>>> Sometimes (unexpectedly when) I get on a chained rsyslog-server >>>>>> logevents like this: >>>>>> Original-Message: host1 1.2.3.4 Apr 21 11:39:43 host1 sshd[11600]: >>>>>> Accepted publickey for user from 2.3.4.5 port 23869 ssh2 >>>>>> Message on changed rsyslog: .3.4 Apr 21 11:39:43 host1 sshd[11600]: >>>>>> Accepted publickey for user from 2.3.4.5 port 23869 ssh2 >>>>>> >>>>>> Error-Log on chained system: >>>>>> Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP >>>>>> message: invalid octet count 0. [v8.8.0.ad1] >>>>>> Apr 23 10:43:11 chained-srv rsyslogd: Framing Error in received TCP >>>>>> message: delimiter is not SP but has ASCII value 58. [v8.8.0.ad1] >>>> >>>> >>>> >>>>> >>>>> which template is this using? >>>> >>>> >>>> The first rsyslog server uses the config below, so template is: >>>> $template >>>> rel,"%fromhost% %fromhost-ip% %rawmsg%\n" >>>> The recieving rsyslog server uses this template and logs to file: >>>> $template raw,"%rawmsg%" >>> >>> >>> >>> Ok, if the sending server is using the template rel to send the message >>> to >>> the second server you have a problem (this is what I thought was >>> happening >>> with the omspoof config above) >>> >>> a valid syslog message is >>> <###>timestamp hostname syslogtag[pid]: message >>> >>> you are sending >>> >>> hostname ip <###>timestamp hostname syslogtag[pid]: message >>> >>> the receiving message is going to try to 'do the right thing' and correct >>> for the malformed message, but it's unlikely to be right all the time. >>> >>> the fact that you are getting framing errors indicates that the sending >>> server is doing something very wrong >>> >> >> I guess there is a numeric hostname. That would trigger octet-counted >> framing, and that in turn could trigger the error message. > > > or just no DNS for the hostname, so that field contains an IP address > instead.
indeed, that's the most probably cause. Rainer > > That makes a lot of sense. > > David Lang > > >> Rainer >>>>>> >>>>>> Do you have any idear / debugging concept? >>>>>> In my lab, everything seems to be fine, so I see only the option to >>>>>> test >>>>>> in production, what i definitively don't want to do... >>>>> >>>>> >>>>> >>>>> check that you have the same version and gnutls version on both sender >>>>> and >>>>> receiver. >>>> >>>> >>>> I do not use TLS at all. >>>> >>>>> a good debugging tool is to write the files locally with the same >>>>> format >>>>> that >>>>> you use to send them remotely, that way if something wierd happens, you >>>>> can look >>>>> at what was sent. >>>> >>>> >>>> Yes, I was also thinking about this. Is there any tool/script that I can >>>> use to "replay" stored messages >>> >>> >>> >>> tcpreplay is a good tool for this. >>> >>> David Lang >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
