On Mon, 7 May 2018, Scot Kreienkamp wrote:
The only action I see in that config that isn't named is the email one, and
that shouldn't be hitting any more. I'm going through the config this morning
and trying to put names on anything missing in any file.
that helps
Would the stuck action be in that ruleset or could it be in another ruleset?
We are seeing messages going into this ruleset, which has it's own queue. So if
that queue is building up, there needs to be something inside this ruleset that
is blocking or not keeping up.
IIRC, it is showing no messages being processed by any of the actions inside
this ruleset. The difficulty is figuring out why. Is there any chance that
permissions (including SELinux permissions) have gotten broken on any of the
files?
can you check the queue stats, make sure that the size in increasing by the
number of enqueued messages, not by some smaller amount that would indicate that
messages are being processed, just slowly.
you have retries=2, which should make everything keep working (just slowing down
and loosing logs)
David Lang
There's about 19 different config files with 200-300 actions in them. It's
our central syslog repository.
Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | |
Mobile: 7349151444 | Email: [email protected]
-----Original Message-----
From: rsyslog [mailto:[email protected]] On Behalf Of David Lang
Sent: Friday, May 4, 2018 5:20 PM
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
Ok, that helps.
the one thing I notice is that your e-mail action at the top doesn't have a
queue on it, so if your mail server can't keep up, you can fall behind and start
queuing.
It's also one of the few actions that doesn't have a name on it, so it's hard to
find in the logs. (it looks like it and action 283 are part of what you stripped
out of the log, they don't show up after 10:30)
On Fri, 4 May 2018, Scot Kreienkamp wrote:
Date: Fri, 4 May 2018 18:42:12 +0000
From: Scot Kreienkamp <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
Thought that part of my config would help too...
https://pastebin.com/smQnxpDZ
Scot Kreienkamp |Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive| Monroe, Michigan 48162 | Office: 734-384-6403 | |
Mobile: 7349151444 | Email: [email protected]
-----Original Message-----
From: rsyslog [mailto:[email protected]] On Behalf Of David Lang
Sent: Thursday, May 3, 2018 5:41 PM
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Queue stops writing to disk
it's good that you have impstats running, it will let us track this down
what you need to look for is to find which of the 13 if statements inside the
ruleset are getting blocked and preventing the ruleset from progressing. If
those are action() statements you can name them to make them easy to find in the
pstats output, otherwise they will just be Action number.
David Lang
On Thu, 3 May 2018, Scot
Kreienkamp wrote:
Date: Thu, 3 May 2018 16:52:36 +0000
From: Scot Kreienkamp <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Queue stops writing to disk
Hi everyone,
I keep running into a situation where a queue will just stop processing until
rsyslog is restarted, and I can't figure out why. Any help would be
appreciated.
I've moved some of my incoming messages to its own queue so it doesn't affect
everything in the main queue, but occasionally this queue will just go into
queueing mode and quit writing to disk until rsyslog is restarted. It happens
at seemingly random times, as much as a month apart or as close as two hours
later. I've ran the pstats through the analyzer and it found nothing. Here's
the pstats of that queue from an occurrence this morning when it quit writing
to disk:
May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 09:50:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2067518 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:00:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2337424 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:10:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=2457977 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:20:07 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=0 enqueued=1333045 full=0 discarded.full=0
discarded.nf=0 maxqsize=3695
May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=175505 enqueued=194024 full=0 discarded.full=0
discarded.nf=0 maxqsize=175505
May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=176105 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=176105
May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 10:50:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=176705 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=176705
May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:00:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=177305 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=177305
May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:10:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=177905 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=177905
May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:20:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=178505 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=178505
May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:30:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=179105 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=179105
May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531[DA]:
origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0
maxqsize=0
May 3 11:40:08 monvsyslog.na.lzb.hq rsyslogd-pstats: Net-1531:
origin=core.queue size=179705 enqueued=600 full=0 discarded.full=0
discarded.nf=0 maxqsize=179705
The truncated config for the queue is:
ruleset(name="Net-1531"
queue.type="LinkedList"
queue.size="250000"
queue.discardmark="200000"
queue.dequeueBatchSize="2048"
queue.workerThreads="1"
queue.workerThreadMinimumMessages="50000"
queue.filename="Net-1531"
){
13 if statements that determine where to log the info to
}
input(type="imudp" port="1531" ruleset="Net-1531")
input(type="imptcp" port="1531" ruleset="Net-1531")
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | * 734-384-6403 | | * 7349151444 | *
[email protected]<mailto:%7BE-mail%7D>
www<http://www.la-z-boy.com/>.la-z-boy.com<http://www.la-z-boy.com/> |
facebook.<https://www.facebook.com/lazboy>com<https://www.facebook.com/lazboy>/<https://www.facebook.com/lazboy>lazboy<http://facebook.com/lazboy>
| twitter.com/lazboy<https://twitter.com/lazboy> |
youtube.com/<https://www.youtube.com/user/lazboy>lazboy<https://www.youtube.com/user/lazboy>
[cid:lzbVertical_hres.jpg]
This message is intended only for the individual or entity to which it is
addressed. It may contain privileged, confidential information which is exempt
from disclosure under applicable laws. If you are not the intended recipient,
you are strictly prohibited from disseminating or distributing this information
(other than to the intended recipient) or copying this information. If you
have received this communication in error, please notify us immediately by
e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
