The only action I see in that config that isn't named is the email one, and that shouldn't be hitting any more. I'm going through the config this morning and trying to put names on anything missing in any file.
Would the stuck action be in that ruleset or could it be in another ruleset? There's about 19 different config files with 200-300 actions in them. It's our central syslog repository. Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | Mobile: 7349151444 | Email: [email protected] -----Original Message----- From: rsyslog [mailto:[email protected]] On Behalf Of David Lang Sent: Friday, May 4, 2018 5:20 PM To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] Queue stops writing to disk Ok, that helps. the one thing I notice is that your e-mail action at the top doesn't have a queue on it, so if your mail server can't keep up, you can fall behind and start queuing. It's also one of the few actions that doesn't have a name on it, so it's hard to find in the logs. (it looks like it and action 283 are part of what you stripped out of the log, they don't show up after 10:30) On Fri, 4 May 2018, Scot Kreienkamp wrote: > Date: Fri, 4 May 2018 18:42:12 +0000 > From: Scot Kreienkamp <[email protected]> > Reply-To: rsyslog-users <[email protected]> > To: rsyslog-users <[email protected]> > Subject: Re: [rsyslog] Queue stops writing to disk > > Thought that part of my config would help too... > > https://pastebin.com/smQnxpDZ > > > > > Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate > One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | | > Mobile: 7349151444 | Email: [email protected] > -----Original Message----- > From: rsyslog [mailto:[email protected]] On Behalf Of David > Lang > Sent: Thursday, May 3, 2018 5:41 PM > To: rsyslog-users <[email protected]> > Subject: Re: [rsyslog] Queue stops writing to disk > > it's good that you have impstats running, it will let us track this down > > what you need to look for is to find which of the 13 if statements inside the > ruleset are getting blocked and preventing the ruleset from progressing. If > those are action() statements you can name them to make them easy to find in > the > pstats output, otherwise they will just be Action number. > > David Lang > > > On Thu, 3 May 2018, Scot > Kreienkamp wrote: > >> Date: Thu, 3 May 2018 16:52:36 +0000 >> From: Scot Kreienkamp <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: [rsyslog] Queue stops writing to disk >> >> Hi everyone, >> >> I keep running into a situation where a queue will just stop processing >> until rsyslog is restarted, and I can't figure out why. Any help would be >> appreciated. >> >> I've moved some of my incoming messages to its own queue so it doesn't >> affect everything in the main queue, but occasionally this queue will just >> go into queueing mode and quit writing to disk until rsyslog is restarted. >> It happens at seemingly random times, as much as a month apart or as close >> as two hours later. I've ran the pstats through the analyzer and it found >> nothing. Here's the pstats of that queue from an occurrence this morning >> when it quit writing to disk: >> >> May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=0 enqueued=2067518 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=3695 >> May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=0 enqueued=2337424 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=3695 >> May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=0 enqueued=2457977 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=3695 >> May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=0 enqueued=1333045 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=3695 >> May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=175505 enqueued=194024 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=175505 >> May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=176105 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=176105 >> May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=176705 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=176705 >> May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=177305 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=177305 >> May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=177905 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=177905 >> May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=178505 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=178505 >> May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=179105 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=179105 >> May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]: >> origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 >> maxqsize=0 >> May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531: >> origin=core.queue size=179705 enqueued=600 full=0 discarded.full=0 >> discarded.nf=0 maxqsize=179705 >> >> >> The truncated config for the queue is: >> >> ruleset(name="Net-1531" >> queue.type="LinkedList" >> queue.size="250000" >> queue.discardmark="200000" >> queue.dequeueBatchSize="2048" >> queue.workerThreads="1" >> queue.workerThreadMinimumMessages="50000" >> queue.filename="Net-1531" >> ){ >> >> 13 if statements that determine where to log the info to >> >> } >> input(type="imudp" port="1531" ruleset="Net-1531") >> input(type="imptcp" port="1531" ruleset="Net-1531") >> >> >> Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate >> One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * >> 7349151444 | * [email protected]<mailto:%7BE-mail%7D> >> www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> | >> facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy> >> | twitter.com/lazboy<https://twitter.com/lazboy> | >> youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy> >> >> [cid:lzbVertical_hres.jpg] >> >> >> >> This message is intended only for the individual or entity to which it is >> addressed. It may contain privileged, confidential information which is >> exempt from disclosure under applicable laws. If you are not the intended >> recipient, you are strictly prohibited from disseminating or distributing >> this information (other than to the intended recipient) or copying this >> information. If you have received this communication in error, please >> notify us immediately by e-mail or by telephone at the above number. Thank >> you. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
