Re: [rsyslog] Rsyslog crashing repeatedly
No message suppression, I have $RepeatedMsgReduction off in my configs. I'll try and capture it with debug, it's been happening a few times per hour and the server it's on is not a large volume source so it may be possible. It hasn't happened since 1am today though which is unusual. Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > -Original Message- > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer > Gerhards > Sent: Friday, September 21, 2018 9:55 AM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > sry, pls bear with me a little - been sidestepped by some other > projects. I think we would need to build new packages with debug > symbols. > > Well, one piece of info that may help me if I get to it over the > weekend: do you use repeated message suppression or some other > ratelimiting method? > > If it occurs quickly, could you try go gather a debug log? That would > probably show more precise when it aborts. Interesting are most > probably jast the last 1000 lines or so (you can mail me privately if > you like). > > Rainer > El vie., 21 sept. 2018 a las 15:43, Scot Kreienkamp > () escribió: > > > > Hi Rainer, > > > > Any more info I can provide to help track down the problem? It's hitting > > us on > several servers and driving some of our admins nuts. > > > > > > Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate > > One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > > > -Original Message- > > > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of > Rainer > > > Gerhards > > > Sent: Wednesday, September 19, 2018 2:47 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > > > > > Yes, could be. Let me go back to computer tomorrow morning to write more > > > detail. Valgrind looks promising. Too bad no symbols. Need to check with > > > our packaging guys. Config would be useful. > > > > > > Possible to create a debug log? Last 1000 lines or so probably > > > interesting. > > > > > > Rainer > > > > > > Sent from phone, thus brief. > > > > > > Scot Kreienkamp schrieb am Mi., 19. Sep. > > > 2018, 19:42: > > > > > > > To my (admittedly non-programmer's eyes) this does look like it might be > > > > related. Hopefully Rainer will weigh in. > > > > > > > > > > > > Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate > > > > One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | > > > > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > > > > > -Original Message- > > > > > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of > > > > Peter > > > > > Cullen > > > > > Sent: Wednesday, September 19, 2018 1:32 PM > > > > > To: rsyslog-users > > > > > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > > > > > > > > > I wonder, could this be the same or related to > > > > > https://github.com/rsyslog/rsyslog/issues/3021 ? > > > > > > > > > > On Wed, Sep 19, 2018 at 1:03 PM, Scot Kreienkamp < > > > > > scot.kreienk...@la-z-boy.com> wrote: > > > > > > > > > > > I'm not using anything like that in my config. I have a bunch of if > > > > > > statements that compare values, nothing very complicated. It seems > > > > to be > > > > > > mostly complaining about imfile. > > > > > > > > > > > > > > > > > > Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate > > > > > > One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 > | | > > > > > > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > > > > > > > -Original Message- > > > > > > > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf > > > > Of > > > > > > John > > > > > > > Chivian > > > > > > > Sent: Wednesday, September 19, 2018 12:58 PM > > > > > > > To: rsyslog@lists.adiscon.com > > > > > > > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > > > > > > > > > > > > > I do not know if it relevant to this issue, but I have > > > > > > > experienced an > > > > > > > rsyslog crash that I was able to attribute to... > > > > > > > > > > > > > > set $.mlen = strlen($msg) - 3; > > > > > > > set $.pmsg = substring($msg,2,$.mlen); > > > > > > > > > > > > > > ...when $msg wasn't long enough to support being stripped in that > > > > > manner. > > > > > > > > > > > > > > I would think this should not work, but not crash the application. > > > > > > > > > > > > > > > > > > > > > On 9/19/18 10:52 AM, Scot Kreienkamp wrote: > > > > > > > > OK, ran it under valgrind until it crashed, here's the logfile > > > > output > > > > > > from > > > > > > > valgrind: > > > > > > > > > > > > > > > > > > > > > > > > ==32102== Memcheck, a memory error detector > > > > > > > > ==32102== Copyright (C)
Re: [rsyslog] Rsyslog crashing repeatedly
Hi Rainer, Any more info I can provide to help track down the problem? It's hitting us on several servers and driving some of our admins nuts. Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > -Original Message- > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer > Gerhards > Sent: Wednesday, September 19, 2018 2:47 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > Yes, could be. Let me go back to computer tomorrow morning to write more > detail. Valgrind looks promising. Too bad no symbols. Need to check with > our packaging guys. Config would be useful. > > Possible to create a debug log? Last 1000 lines or so probably interesting. > > Rainer > > Sent from phone, thus brief. > > Scot Kreienkamp schrieb am Mi., 19. Sep. > 2018, 19:42: > > > To my (admittedly non-programmer's eyes) this does look like it might be > > related. Hopefully Rainer will weigh in. > > > > > > Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate > > One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | > > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > > > -Original Message- > > > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of > > Peter > > > Cullen > > > Sent: Wednesday, September 19, 2018 1:32 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > > > > > I wonder, could this be the same or related to > > > https://github.com/rsyslog/rsyslog/issues/3021 ? > > > > > > On Wed, Sep 19, 2018 at 1:03 PM, Scot Kreienkamp < > > > scot.kreienk...@la-z-boy.com> wrote: > > > > > > > I'm not using anything like that in my config. I have a bunch of if > > > > statements that compare values, nothing very complicated. It seems > > to be > > > > mostly complaining about imfile. > > > > > > > > > > > > Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate > > > > One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | > > > > Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com > > > > > -Original Message- > > > > > From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf > > Of > > > > John > > > > > Chivian > > > > > Sent: Wednesday, September 19, 2018 12:58 PM > > > > > To: rsyslog@lists.adiscon.com > > > > > Subject: Re: [rsyslog] Rsyslog crashing repeatedly > > > > > > > > > > I do not know if it relevant to this issue, but I have experienced an > > > > > rsyslog crash that I was able to attribute to... > > > > > > > > > > set $.mlen = strlen($msg) - 3; > > > > > set $.pmsg = substring($msg,2,$.mlen); > > > > > > > > > > ...when $msg wasn't long enough to support being stripped in that > > > manner. > > > > > > > > > > I would think this should not work, but not crash the application. > > > > > > > > > > > > > > > On 9/19/18 10:52 AM, Scot Kreienkamp wrote: > > > > > > OK, ran it under valgrind until it crashed, here's the logfile > > output > > > > from > > > > > valgrind: > > > > > > > > > > > > > > > > > > ==32102== Memcheck, a memory error detector > > > > > > ==32102== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward > > > et > > > > al. > > > > > > ==32102== Using Valgrind-3.8.1 and LibVEX; rerun with -h for > > copyright > > > > info > > > > > > ==32102== Command: /sbin/rsyslogd -f /etc/rsyslog.conf > > > > > > ==32102== Parent PID: 3883 > > > > > > ==32102== > > > > > > ==32107== Warning: invalid file descriptor 65536 in syscall close() > > > > > > ==32102== > > > > > > ==32102== HEAP SUMMARY: > > > > > > ==32102== in use at exit: 199,617 bytes in 2,693 blocks > > > > > > ==32102== total heap usage: 5,889 allocs, 3,196 frees, 479,531 > > bytes > > > > > allocated > > > > > > ==32102== > > > > > > ==32102== LEAK SUMMARY: > > > > > > ==32102==definitely lost: 0 bytes in 0 blocks > > > > > > ==32102==indirectly lost: 0 bytes in 0 blocks > > > > > > ==32102== possibly lost: 0 bytes in 0 blocks > > > > > > ==32102==still reachable: 199,617 bytes in 2,693 blocks > > > > > > ==32102== suppressed: 0 bytes in 0 blocks > > > > > > ==32102== Rerun with --leak-check=full to see details of leaked > > > memory > > > > > > ==32102== > > > > > > ==32102== For counts of detected and suppressed errors, rerun > > with: - > > > v > > > > > > ==32102== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 40 > > > > > from 9) > > > > > > ==32107== Thread 4: > > > > > > ==32107== Invalid read of size 2 > > > > > > ==32107==at 0x168982: ratelimitMsg (in /sbin/rsyslogd) > > > > > > ==32107==by 0x168DFA: ratelimitAddMsg (in /sbin/rsyslogd) > > > > > > ==32107==by 0x6C3B4A5: ??? (in /lib64/rsyslog/imfile.so) > > > > > > ==32107==by 0x6C3C65B: ??? (in /lib64/rsyslog/imfile.so) > > > > > > ==32107==by 0x6C3CB18: ??? (in
Re: [rsyslog] rsyslog data on NFS storage and failover
Hey Andriy! Our central syslog-server is designed in another way: We have a primary/secondary failover via CARP (but you can use VRRP or whatever) which logs everything to their local disk. Then we have another "archive" or "backup"-server which copies the logs to it's SAN-backed storage and removes them from the central syslog servers. This way each centralised syslog-server is self-contained but we still get the features of cheap SAN-storage. Hope this helps. BR, - Simon Simon Lundström Section for Infrastructure IT Services Stockholm University SE-106 91 Stockholm, Sweden www.su.se/english/staff-info/it https://www.su.se/english/about-this-website/privacy-policy On Thu, 2018-09-20 at 17:49:54 +0200, Andriy Tovstik via rsyslog wrote: I try to setup failover rsyslog configuration with primary and failover instances for a lot of services. Both rsyslog instances write files to NFS share to the same location throw omfile module. Looks like solution works fine - when the primary server fails all messages successfully data is successfully stored on NFS through the secondary rsyslog server. But this configuration has a SPOF - NFS server. Is there any ability to avoid or minimize log messages loss in case of NFS goes down? For example - any failover local buffer which will store logs until NFS up. --- Regards, Andrey ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
