I put the full impstats output up on https://pastebin.com/Yr2vvLck, hopefully you can spot something I'm missing. I filtered out the noise (all the lines where everything equals zero), otherwise it's all there. The problem was first noticed at 10:30am so I posted the 10am to 11am ipmstats output.
Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: [email protected] -----Original Message----- From: rsyslog [mailto:[email protected]] On Behalf Of David Lang Sent: Thursday, May 3, 2018 5:41 PM To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] Queue stops writing to disk it's good that you have impstats running, it will let us track this down what you need to look for is to find which of the 13 if statements inside the ruleset are getting blocked and preventing the ruleset from progressing. If those are action() statements you can name them to make them easy to find in the pstats output, otherwise they will just be Action number. David Lang On Thu, 3 May 2018, Scot Kreienkamp wrote: > Date: Thu, 3 May 2018 16:52:36 +0000 > From: Scot Kreienkamp <[email protected]> > Reply-To: rsyslog-users <[email protected]> > To: rsyslog-users <[email protected]> > Subject: [rsyslog] Queue stops writing to disk > > Hi everyone, > > I keep running into a situation where a queue will just stop processing until > rsyslog is restarted, and I can't figure out why. Any help would be > appreciated. > > I've moved some of my incoming messages to its own queue so it doesn't affect > everything in the main queue, but occasionally this queue will just go into > queueing mode and quit writing to disk until rsyslog is restarted. It > happens at seemingly random times, as much as a month apart or as close as > two hours later. I've ran the pstats through the analyzer and it found > nothing. Here's the pstats of that queue from an occurrence this morning > when it quit writing to disk: > > May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=0 enqueued=2067518 full=0 discarded.full=0 > discarded.nf=0 maxqsize=3695 > May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=0 enqueued=2337424 full=0 discarded.full=0 > discarded.nf=0 maxqsize=3695 > May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=0 enqueued=2457977 full=0 discarded.full=0 > discarded.nf=0 maxqsize=3695 > May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=0 enqueued=1333045 full=0 discarded.full=0 > discarded.nf=0 maxqsize=3695 > May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=175505 enqueued=194024 full=0 discarded.full=0 > discarded.nf=0 maxqsize=175505 > May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=176105 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=176105 > May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=176705 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=176705 > May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=177305 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=177305 > May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=177905 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=177905 > May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=178505 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=178505 > May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=179105 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=179105 > May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: > origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 > maxqsize=0 > May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: > origin=core.queue size=179705 enqueued=600 full=0 discarded.full=0 > discarded.nf=0 maxqsize=179705 > > > The truncated config for the queue is: > > ruleset(name="Net-1531" > queue.type="LinkedList" > queue.size="250000" > queue.discardmark="200000" > queue.dequeueBatchSize="2048" > queue.workerThreads="1" > queue.workerThreadMinimumMessages="50000" > queue.filename="Net-1531" > ){ > > 13 if statements that determine where to log the info to > > } > input(type="imudp" port="1531" ruleset="Net-1531") > input(type="imptcp" port="1531" ruleset="Net-1531") > > > Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate > One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * > 7349151444 | * [email protected]<mailto:%7BE-mail%7D> > www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | > facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> > | twitter.com/lazboy<https://twitter.com/lazboy> | > youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy> > > [cid:lzbVertical_hres.jpg] > > > > This message is intended only for the individual or entity to which it is > addressed. It may contain privileged, confidential information which is > exempt from disclosure under applicable laws. If you are not the intended > recipient, you are strictly prohibited from disseminating or distributing > this information (other than to the intended recipient) or copying this > information. If you have received this communication in error, please notify > us immediately by e-mail or by telephone at the above number. Thank you. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
