Ok - I will set things up and do my best to capture the failure and let you know when I have logs.
Thanks for your help - appreciate it! > On Jun 11, 2019, at 8:53 AM, Rainer Gerhards <[email protected]> wrote: > > Caution - External Email;;; > > ******************************************************************************************************************************************** > El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias > (<[email protected]>) escribió: >> >> Hello Rainer, >> >> Thank you for the information. I set the debug configuration, updated >> rsyslog and restarted. After about 5 minutes, the logs are starting to slow >> and the debug log is well over 12GB in size. >> >> It seems having debug enabled slows general performance considerably and >> given the rate at which the debug log is building, finding relevant, helpful >> data might be a challenge...not to mention getting you the file itself! > > If the 12GiB contain the actual error case, than it is not too bad - I > am used to wading though large debug files. These files usually > compress very well (xz recommended), so the transfer is also not too > bad. It would be best, though, if we can go to a point where the log > really stalls. > > For starters, I would guess that the last minute or so of data could > be interesting. > > Rainer >> >> Tobias >> >> -----Original Message----- >> From: Rainer Gerhards <[email protected]> >> Sent: Tuesday, June 11, 2019 7:33 AM >> To: rsyslog-users <[email protected]> >> Cc: David Lang <[email protected]>; Heaton, Tobias <[email protected]> >> Subject: Re: [rsyslog] 8.1905.0 Logging Stops >> >> Caution - External Email;;; >> >> ******************************************************************************************************************************************** >> Hi Tobias, >> >> this is the "quick guide" to generate a debug log: >> https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html >> - just do the "regular debug" part and let it run until the problem >> manifests. >> >> We also have more elaborate ones, but I guess this one is sufficient >> (let's keep it smooth and simple ;-)). >> >> Rainer >> >> El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog >> (<[email protected]>) escribió: >>> >>> My apologies - I'm a little naïve when it comes to the debugging. >>> >>> What's your recommendation on the best way to capture debug? >>> >>> -----Original Message----- >>> From: David Lang <[email protected]> >>> Sent: Monday, June 10, 2019 3:50 PM >>> To: Heaton, Tobias <[email protected]> >>> Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog >>> <[email protected]> >>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops >>> >>> Caution - External Email;;; >>> >>> ******************************************************************************************************************************************** >>> by the way, looking at your evicted stats, you probably want to tweak your >>> dynafilecachesize on some of your outputs (performance issue, probably not >>> related to this issue) >>> >>> data was still being processed by outputs through the 15:00:17 window. It >>> didn't >>> stop until the 15:00:22 window. >>> >>> If you can get a debug log (send to Rainer) through the time it crashes, >>> that >>> would be helpful >>> >>> Also, can you post your config? >>> >>> David Lang >>> >>>> On Mon, 10 Jun 2019, Heaton, Tobias wrote: >>>> >>>> Date: Mon, 10 Jun 2019 19:37:31 +0000 >>>> From: "Heaton, Tobias" <[email protected]> >>>> To: David Lang <[email protected]> >>>> Cc: "Heaton, Tobias via rsyslog" <[email protected]> >>>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops >>>> >>>> Attached is a snippet of the impstats. Logs stopped writing at 15:00:01. >>>> >>>> -----Original Message----- >>>> From: David Lang <[email protected]> >>>> Sent: Monday, June 10, 2019 3:29 PM >>>> To: Heaton, Tobias <[email protected]> >>>> Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog >>>> <[email protected]> >>>> Subject: RE: [rsyslog] 8.1905.0 Logging Stops >>>> >>>> Caution - External Email;;; >>>> >>>> ******************************************************************************************************************************************** >>>>> On Mon, 10 Jun 2019, Heaton, Tobias wrote: >>>>> >>>>> I configured impstats, updated rsyslog, restarted and awaited the >>>>> failure. Tailing the busiest logs, behavior starts normal. Over the >>>>> course of a few minutes, the logs seems to arrive in chunks...or at least >>>>> appear to write that way to disk. >>>>> >>>>> That behavior continues with logs populating incrementally slower and >>>>> slower until it just stops. The impstats don't show anything of >>>>> consequence - the repeated stats are all normal and the same both prior >>>>> to and after the logs stop writing. >>>>> >>>>> Rsyslog appears still running and impstats continue to populate again >>>>> with no appreciable changes - very strange! >>>> >>>> after it stops, does impstats show new messages arriving (and queues >>>> building)? >>>> or does it show outputs failing and messages being discarded? >>>> or something else? >>>> >>>> a couple cycles of the pstats output after it's failed would be useful >>>> >>>> David Lang >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e= >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e= >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
