El mar., 11 jun. 2019 a las 19:23, Heaton, Tobias (<[email protected]>) escribió: > > Rainer, > > I traced my steps and allowed rsyslog to run for 45 minutes without seeing a > failure. > > The debug puts considerable strain on the system, so much so that the log > timestamps started drifting. I was seeing logs stamped at 13:18 EST at > 13:31EST actual time. > > Furthermore the debug log is nearly 70GB, closing in on the available disk > space. > > Is there anything else I can try to help troubleshoot this problem?
Not really. Maybe the first 12 GiB actually contain info that get's us a bit closer. So if you could share that, it would be useful. We can fine-tune the debug output, but as we have not the slightest idea of what may be the root cause that's no option. Rainer > > Tobias > > -----Original Message----- > From: Rainer Gerhards <[email protected]> > Sent: Tuesday, June 11, 2019 8:53 AM > To: Heaton, Tobias <[email protected]> > Cc: rsyslog-users <[email protected]>; David Lang <[email protected]> > Subject: Re: [rsyslog] 8.1905.0 Logging Stops > > Caution - External Email;;; > > ******************************************************************************************************************************************** > El mar., 11 jun. 2019 a las 14:33, Heaton, Tobias > (<[email protected]>) escribió: > > > > Hello Rainer, > > > > Thank you for the information. I set the debug configuration, updated > > rsyslog and restarted. After about 5 minutes, the logs are starting to slow > > and the debug log is well over 12GB in size. > > > > It seems having debug enabled slows general performance considerably and > > given the rate at which the debug log is building, finding relevant, > > helpful data might be a challenge...not to mention getting you the file > > itself! > > If the 12GiB contain the actual error case, than it is not too bad - I > am used to wading though large debug files. These files usually > compress very well (xz recommended), so the transfer is also not too > bad. It would be best, though, if we can go to a point where the log > really stalls. > > For starters, I would guess that the last minute or so of data could > be interesting. > > Rainer > > > > Tobias > > > > -----Original Message----- > > From: Rainer Gerhards <[email protected]> > > Sent: Tuesday, June 11, 2019 7:33 AM > > To: rsyslog-users <[email protected]> > > Cc: David Lang <[email protected]>; Heaton, Tobias <[email protected]> > > Subject: Re: [rsyslog] 8.1905.0 Logging Stops > > > > Caution - External Email;;; > > > > ******************************************************************************************************************************************** > > Hi Tobias, > > > > this is the "quick guide" to generate a debug log: > > https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html > > - just do the "regular debug" part and let it run until the problem > > manifests. > > > > We also have more elaborate ones, but I guess this one is sufficient > > (let's keep it smooth and simple ;-)). > > > > Rainer > > > > El lun., 10 jun. 2019 a las 21:58, Heaton, Tobias via rsyslog > > (<[email protected]>) escribió: > > > > > > My apologies - I'm a little naïve when it comes to the debugging. > > > > > > What's your recommendation on the best way to capture debug? > > > > > > -----Original Message----- > > > From: David Lang <[email protected]> > > > Sent: Monday, June 10, 2019 3:50 PM > > > To: Heaton, Tobias <[email protected]> > > > Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog > > > <[email protected]> > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > > > Caution - External Email;;; > > > > > > ******************************************************************************************************************************************** > > > by the way, looking at your evicted stats, you probably want to tweak your > > > dynafilecachesize on some of your outputs (performance issue, probably not > > > related to this issue) > > > > > > data was still being processed by outputs through the 15:00:17 window. It > > > didn't > > > stop until the 15:00:22 window. > > > > > > If you can get a debug log (send to Rainer) through the time it crashes, > > > that > > > would be helpful > > > > > > Also, can you post your config? > > > > > > David Lang > > > > > > On Mon, 10 Jun 2019, Heaton, Tobias wrote: > > > > > > > Date: Mon, 10 Jun 2019 19:37:31 +0000 > > > > From: "Heaton, Tobias" <[email protected]> > > > > To: David Lang <[email protected]> > > > > Cc: "Heaton, Tobias via rsyslog" <[email protected]> > > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > > > > > Attached is a snippet of the impstats. Logs stopped writing at 15:00:01. > > > > > > > > -----Original Message----- > > > > From: David Lang <[email protected]> > > > > Sent: Monday, June 10, 2019 3:29 PM > > > > To: Heaton, Tobias <[email protected]> > > > > Cc: David Lang <[email protected]>; Heaton, Tobias via rsyslog > > > > <[email protected]> > > > > Subject: RE: [rsyslog] 8.1905.0 Logging Stops > > > > > > > > Caution - External Email;;; > > > > > > > > ******************************************************************************************************************************************** > > > > On Mon, 10 Jun 2019, Heaton, Tobias wrote: > > > > > > > >> I configured impstats, updated rsyslog, restarted and awaited the > > > >> failure. Tailing the busiest logs, behavior starts normal. Over the > > > >> course of a few minutes, the logs seems to arrive in chunks...or at > > > >> least appear to write that way to disk. > > > >> > > > >> That behavior continues with logs populating incrementally slower and > > > >> slower until it just stops. The impstats don't show anything of > > > >> consequence - the repeated stats are all normal and the same both > > > >> prior to and after the logs stop writing. > > > >> > > > >> Rsyslog appears still running and impstats continue to populate again > > > >> with no appreciable changes - very strange! > > > > > > > > after it stops, does impstats show new messages arriving (and queues > > > > building)? > > > > or does it show outputs failing and messages being discarded? > > > > or something else? > > > > > > > > a couple cycles of the pstats output after it's failed would be useful > > > > > > > > David Lang > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailman_listinfo_rsyslog&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=AIKzntUgGqWkTqo_j2lfE7vckKRK0v9bS913ZhCug6k&e= > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d=DwIFaQ&c=c6MrceVCY5m5A_KAUkrdoA&r=Gev2oD_cwBaLsgGrQ_lnBXqGXR9EUvddO5r7UxzqP5s&m=lhTjg_QDGIeHOeCAcLX5bXSSgEoxAn4zMhoKMn39_Q0&s=Wv4TJP-48wkRkAh4kLGr1DRn9AJcn69V5upqPFFcXkY&e= > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
