El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias (<[email protected]>) escribió: > > Hello Rainer, > > Thank you for the information. I set the debug configuration, updated rsyslog > and restarted. After about 5 minutes, the logs are starting to slow and the > debug log is well over 12GB in size. > > It seems having debug enabled slows general performance considerably and > given the rate at which the debug log is building, finding relevant, helpful > data might be a challenge...not to mention getting you the file itself!
If the 12GiB contain the actual error case, than it is not too bad - I am used to wading though large debug files. These files usually compress very well (xz recommended), so the transfer is also not too bad. It would be best, though, if we can go to a point where the log really stalls. For starters, I would guess that the last minute or so of data could be interesting. Rainer > > Tobias > > -----Original Message----- > From: Rainer Gerhards <[email protected]> > Sent: Tuesday, June 11, 2019 7:33 AM > To: rsyslog-users <[email protected]> > Cc: David Lang <[email protected]>; Heaton, Tobias <[email protected]> > Subject: Re: [rsyslog] 8.1905.0 Logging Stops > > Caution - External Email;;; > > ******************************************************************************************************************************************** > Hi Tobias, > > this is the "quick guide" to generate a debug log: > https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html > - just do the "regular debug" part and let it run until the problem > manifests. > > We also have more elaborate ones, but I guess this one is sufficient > (let's keep it smooth and simple ;-)). > > Rainer > > El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog > (<[email protected]>) escribió: > > > > My apologies - I'm a little naïve when it comes to the debugging. > > > > What's your recommendation on the best way to capture debug? > > > > -----Original Message----- > > From: David Lang <[email protected]> > > Sent: Monday, June 10, 2019 3:50 PM > > To: Heaton, Tobias <[email protected]> > > Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog > > <[email protected]> > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > Caution - External Email;;; > > > > ******************************************************************************************************************************************** > > by the way, looking at your evicted stats, you probably want to tweak your > > dynafilecachesize on some of your outputs (performance issue, probably not > > related to this issue) > > > > data was still being processed by outputs through the 15:00:17 window. It > > didn't > > stop until the 15:00:22 window. > > > > If you can get a debug log (send to Rainer) through the time it crashes, > > that > > would be helpful > > > > Also, can you post your config? > > > > David Lang > > > > On Mon, 10 Jun 2019, Heaton, Tobias wrote: > > > > > Date: Mon, 10 Jun 2019 19:37:31 +0000 > > > From: "Heaton, Tobias" <[email protected]> > > > To: David Lang <[email protected]> > > > Cc: "Heaton, Tobias via rsyslog" <[email protected]> > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > > > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01. > > > > > > -----Original Message----- > > > From: David Lang <[email protected]> > > > Sent: Monday, June 10, 2019 3:29 PM > > > To: Heaton, Tobias <[email protected]> > > > Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog > > > <[email protected]> > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > > > Caution - External Email;;; > > > > > > ******************************************************************************************************************************************** > > > On Mon, 10 Jun 2019, Heaton, Tobias wrote: > > > > > >> I configured impstats, updated rsyslog, restarted and awaited the > > >> failure. Tailing the busiest logs, behavior starts normal. Over the > > >> course of a few minutes, the logs seems to arrive in chunks...or at > > >> least appear to write that way to disk. > > >> > > >> That behavior continues with logs populating incrementally slower and > > >> slower until it just stops. The impstats don't show anything of > > >> consequence - the repeated stats are all normal and the same both prior > > >> to and after the logs stop writing. > > >> > > >> Rsyslog appears still running and impstats continue to populate again > > >> with no appreciable changes - very strange! > > > > > > after it stops, does impstats show new messages arriving (and queues > > > building)? > > > or does it show outputs failing and messages being discarded? > > > or something else? > > > > > > a couple cycles of the pstats output after it's failed would be useful > > > > > > David Lang > > > > > _______________________________________________ > > rsyslog mailing list > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e= > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow > > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e= > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
