Hello Rainer, Thank you for the information. I set the debug configuration, updated rsyslog and restarted. After about 5 minutes, the logs are starting to slow and the debug log is well over 12GB in size.
It seems having debug enabled slows general performance considerably and given the rate at which the debug log is building, finding relevant, helpful data might be a challenge...not to mention getting you the file itself! Tobias -----Original Message----- From: Rainer Gerhards <[email protected]> Sent: Tuesday, June 11, 2019 7:33 AM To: rsyslog-users <[email protected]> Cc: David Lang <[email protected]>; Heaton, Tobias <[email protected]> Subject: Re: [rsyslog] 8.1905.0 Logging Stops Caution - External Email;;; ******************************************************************************************************************************************** Hi Tobias, this is the "quick guide" to generate a debug log: https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html - just do the "regular debug" part and let it run until the problem manifests. We also have more elaborate ones, but I guess this one is sufficient (let's keep it smooth and simple ;-)). Rainer El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog (<[email protected]>) escribió: > > My apologies - I'm a little naïve when it comes to the debugging. > > What's your recommendation on the best way to capture debug? > > -----Original Message----- > From: David Lang <[email protected]> > Sent: Monday, June 10, 2019 3:50 PM > To: Heaton, Tobias <[email protected]> > Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog > <[email protected]> > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > Caution - External Email;;; > > ******************************************************************************************************************************************** > by the way, looking at your evicted stats, you probably want to tweak your > dynafilecachesize on some of your outputs (performance issue, probably not > related to this issue) > > data was still being processed by outputs through the 15:00:17 window. It > didn't > stop until the 15:00:22 window. > > If you can get a debug log (send to Rainer) through the time it crashes, that > would be helpful > > Also, can you post your config? > > David Lang > > On Mon, 10 Jun 2019, Heaton, Tobias wrote: > > > Date: Mon, 10 Jun 2019 19:37:31 +0000 > > From: "Heaton, Tobias" <[email protected]> > > To: David Lang <[email protected]> > > Cc: "Heaton, Tobias via rsyslog" <[email protected]> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01. > > > > -----Original Message----- > > From: David Lang <[email protected]> > > Sent: Monday, June 10, 2019 3:29 PM > > To: Heaton, Tobias <[email protected]> > > Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog > > <[email protected]> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > Caution - External Email;;; > > > > ******************************************************************************************************************************************** > > On Mon, 10 Jun 2019, Heaton, Tobias wrote: > > > >> I configured impstats, updated rsyslog, restarted and awaited the failure. > >> Tailing the busiest logs, behavior starts normal. Over the course of a few > >> minutes, the logs seems to arrive in chunks...or at least appear to write > >> that way to disk. > >> > >> That behavior continues with logs populating incrementally slower and > >> slower until it just stops. The impstats don't show anything of > >> consequence - the repeated stats are all normal and the same both prior to > >> and after the logs stop writing. > >> > >> Rsyslog appears still running and impstats continue to populate again with > >> no appreciable changes - very strange! > > > > after it stops, does impstats show new messages arriving (and queues > > building)? > > or does it show outputs failing and messages being discarded? > > or something else? > > > > a couple cycles of the pstats output after it's failed would be useful > > > > David Lang > > > _______________________________________________ > rsyslog mailing list > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e= > > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e= > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
