I'm not going to be able to figure things out withotu the full config, and
stripping lines out that you don't think matter actually makes it much harder to
analyze.
the fact that I see action 308, 309, and 311 in the logs tells me you have a
fairly complex config.
It would probably help if you can name the actions inside the net-1531 ruleset
(or at least one of them so we can see what action numbers are near it)
you should be looking for failures or suspended actions
On Fri, 4 May 2018, Scot Kreienkamp wrote:
Date: Fri, 4 May 2018 15:53:43 +0000
From: Scot Kreienkamp <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
I put the full impstats output up on https://pastebin.com/Yr2vvLck, hopefully
you can spot something I'm missing. I filtered out the noise (all the lines
where everything equals zero), otherwise it's all there. The problem was first
noticed at 10:30am so I posted the 10am to 11am ipmstats output.
Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | |
Mobile: 7349151444 | Email: [email protected]
-----Original Message-----
From: rsyslog [mailto:[email protected]] On Behalf Of David Lang
Sent: Thursday, May 3, 2018 5:41 PM
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
it's good that you have impstats running, it will let us track this down
what you need to look for is to find which of the 13 if statements inside the
ruleset are getting blocked and preventing the ruleset from progressing. If
those are action() statements you can name them to make them easy to find in the
pstats output, otherwise they will just be Action number.
David Lang
On Thu, 3 May 2018, Scot
Kreienkamp wrote:
Date: Thu, 3 May 2018 16:52:36 +0000
From: Scot Kreienkamp <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Queue stops writing to disk
Hi everyone,
I keep running into a situation where a queue will just stop processing until
rsyslog is restarted, and I can't figure out why. Any help would be
appreciated.
I've moved some of my incoming messages to its own queue so it doesn't affect
everything in the main queue, but occasionally this queue will just go into
queueing mode and quit writing to disk until rsyslog is restarted. It happens
at seemingly random times, as much as a month apart or as close as two hours
later. I've ran the pstats through the analyzer and it found nothing. Here's
the pstats of that queue from an occurrence this morning when it quit writing
to disk:
May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2067518 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2337424 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2457977 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=1333045 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=175505 enqueued=194024 full=0 discarded.full=0
discarded.nf=0 maxqsize=175505
May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=176105 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=176105
May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=176705 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=176705
May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=177305 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=177305
May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=177905 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=177905
May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=178505 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=178505
May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=179105 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=179105
May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=179705 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=179705
The truncated config for the queue is:
ruleset(name="Net-1531"
queue.type="LinkedList"
queue.size="250000"
queue.discardmark="200000"
queue.dequeueBatchSize="2048"
queue.workerThreads="1"
queue.workerThreadMinimumMessages="50000"
queue.filename="Net-1531"
){
13 if statements that determine where to log the info to
}
input(type="imudp" port="1531" ruleset="Net-1531")
input(type="imptcp" port="1531" ruleset="Net-1531")
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * 7349151444 | *
[email protected]<mailto:%7BE-mail%7D>
www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> |
facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy>
| twitter.com/lazboy<https://twitter.com/lazboy> |
youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy>
[cid:lzbVertical_hres.jpg]
This message is intended only for the individual or entity to which it is
addressed. It may contain privileged, confidential information which is exempt
from disclosure under applicable laws. If you are not the intended recipient,
you are strictly prohibited from disseminating or distributing this information
(other than to the intended recipient) or copying this information. If you
have received this communication in error, please notify us immediately by
e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
