so, one cycle of stats
2019-08-01T19:31:30.625929-04:00 lnx21648 rsyslogd-pstats: global:
origin=dynstats
2019-08-01T19:31:30.625943-04:00 lnx21648 rsyslogd-pstats: imuxsock:
origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0
no stats from /dev/log
2019-08-01T19:31:30.625946-04:00 lnx21648 rsyslogd-pstats: action 0:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625947-04:00 lnx21648 rsyslogd-pstats: action 1:
origin=core.action processed=2444154 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625949-04:00 lnx21648 rsyslogd-pstats: action 2:
origin=core.action processed=2444154 failed=0 suspended=0 suspended.duration=0
resumed=0
action # are the outputs (with the new action() syntax you can give them a name,
otherwise you just get a numbe to show where they are in the file)
processed is the number of logs written, if syspended/resumed are not zero, you
have had problems (should never happen when writing to files, but from your
other e-mail, you do have problems sending logs out)
note that message counts default to being running totals, so this is 2444154
since you started rsyslog, not this cycle
2019-08-01T19:31:30.625957-04:00 lnx21648 rsyslogd-pstats: dynafile cache
RemoteClient: origin=omfile requests=3583721 level0=1938992 missed=61922
evicted=61912 maxused=10 closetimeouts=0
you have an output writing to files using a template for the filename, but you
have the cache size set to 10, so
every time it tries to open a new files, it has to close an existing file (this
happened 61922 times in this cycle). This is a _really_ bad thing, you need to
increase the dynafilecachesize to something well over the number of files you
are writing to. change this action to the action() syntax to make it clear
what's happening here.
2019-08-01T19:31:30.625959-04:00 lnx21648 rsyslogd-pstats: action 3:
origin=core.action processed=3584143 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625961-04:00 lnx21648 rsyslogd-pstats: dynafile cache
RemoteClient: origin=omfile requests=0 level0=0 missed=0 evicted=0 maxused=0
closetimeouts=0
2019-08-01T19:31:30.625962-04:00 lnx21648 rsyslogd-pstats: action 4:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625963-04:00 lnx21648 rsyslogd-pstats: action 5:
origin=core.action processed=7 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625964-04:00 lnx21648 rsyslogd-pstats: action 6:
origin=core.action processed=4 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.625967-04:00 lnx21648 rsyslogd-pstats: action 7:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650614-04:00 lnx21648 rsyslogd-pstats: action 8:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650617-04:00 lnx21648 rsyslogd-pstats: action 9:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650620-04:00 lnx21648 rsyslogd-pstats: action 10:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650623-04:00 lnx21648 rsyslogd-pstats: action 11:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650624-04:00 lnx21648 rsyslogd-pstats: action 12:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650629-04:00 lnx21648 rsyslogd-pstats: action 13:
origin=core.action processed=7 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650633-04:00 lnx21648 rsyslogd-pstats: action 14:
origin=core.action processed=4 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650634-04:00 lnx21648 rsyslogd-pstats: action 15:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2019-08-01T19:31:30.650635-04:00 lnx21648 rsyslogd-pstats: action 16:
origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
more actions
2019-08-01T19:31:30.650646-04:00 lnx21648 rsyslogd-pstats: imudp(*:514):
origin=imudp submitted=3683624
you recei ved 3683624 messages via UDP since rsyslog started
2019-08-01T19:31:30.650647-04:00 lnx21648 rsyslogd-pstats: imudp(*:514):
origin=imudp submitted=0
2019-08-01T19:31:30.650648-04:00 lnx21648 rsyslogd-pstats: imtcp(514):
origin=imtcp submitted=0
2019-08-01T19:31:30.650657-04:00 lnx21648 rsyslogd-pstats: resource-usage:
origin=impstats utime=29638410 stime=66783415 maxrss=103028 minflt=237512
majflt=0 inblock=0 oublock=1783768 nvcsw=3306119 nivcsw=18216
this is how much ram you are using and how much cpu you used
2019-08-01T19:31:30.650659-04:00 lnx21648 rsyslogd-pstats: main Q:
origin=core.queue size=99758 enqueued=3683657 full=12831 discarded.full=0
discarded.nf=0 maxqsize=100000
your main queue is currently at 99758 items, you received a total of 3683657 new
mesages, you were full 12831 times, and your max q size is the default 100000
normally you would want to fix this fast, but fix the dynafilecache size above
first and then let's see what happens
2019-08-01T19:31:30.650661-04:00 lnx21648 rsyslogd-pstats: imudp(w0):
origin=imudp called.recvmmsg=3397708 called.recvmsg=0 msgs.received=3683624
more stats about udp messages that you recevied
David Lang
On Fri, 2 Aug 2019, Singh, Radesh wrote:
Date: Fri, 2 Aug 2019 09:18:35 +0000
From: "Singh, Radesh" <[email protected]>
To: David Lang <[email protected]>
Cc: "Singh, Radesh via rsyslog" <[email protected]>
Subject: RE: [E]Re: [rsyslog] Missing messages
I enabled stats, by adding the following:
module(load="impstats"
interval="600"
severity="7")
# to actually gather the data:
syslog.=debug /dumps/rsyslogd/stats
I've got a bunch of data, but not really sure what to look for.
As you see I'm sampling in 10 minute intervals, and I took severity from an
example provided... not sure if that was the right move, pls let me know if I
should choose some other value.
I've provided the full output of the stats as an attachment.
Thanks again!
Radesh
-----Original Message-----
From: David Lang <[email protected]>
Sent: Thursday, August 01, 2019 8:48 PM
To: Singh, Radesh <[email protected]>
Cc: David Lang <[email protected]>; Singh, Radesh via rsyslog
<[email protected]>
Subject: RE: [E]Re: [rsyslog] Missing messages
these buffers are small compared to the rest of your system (I think you bumped
them to 8M If I remember the units correctly), which is peanuts on your system
so enable impstats, look for rsyslog error messages in your log output, and
watch these buffer errors to see if they really go away.
Then with a little more data, we can look to tune the rsyslog.conf a bit.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
