David, thank you. Apologies, I had cut out the dynafile for testing. Here is some additional config file:
template(name="EduroamACSFile" type="string"
string="/var/log/collection/eduroam_acs.log")
template(name="EduroamACSFormat" type="string" string="%timestamp%
%hostname% %syslogtag%%msg:::escape-cc%\n")
ruleset(name="ruleset1" queue.type="LinkedList" queue.size="750000"
queue.workerthreads="4") {
# at very top of ruleset - IPs obscured for privacy
if $fromhost-ip == 'x.x.2.75'
or $fromhost-ip == 'x.x.2.85'
then {
/tmp/debug.log;RSYSLOG_DebugFormat
action(type="omfile" dynaFile="EduroamACSFile"
template="EduroamACSFormat" name="writeEduroamACSFile")
stop
}
#... other rules below
} # end ruleset
I do have impstats enabled for every 5 minutes. Here's output specific to
this file for a recent sample for this file, plus the general rule set
queues:
Oct 5 14:06:47 its-syslog-up1 rsyslogd-pstats: dynafile cache
EduroamACSFile: origin=omfile requests=31123 level0=30781 missed=342
evicted=0 maxused=1 closetimeouts=51
Oct 5 14:06:47 its-syslog-up1 rsyslogd-pstats: writeEduroamACSFile:
origin=core.action processed=31123 failed=0 suspended=0 suspended.duration=0
resumed=0
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: resource-usage:
origin=impstats utime=217851481196 stime=87433080781 maxrss=628664
minflt=43999703 majflt=216 inblock=3033769 oublock=-1119976592
nvcsw=379912551 nivcsw=20999555
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: LinuxRuleSet:
origin=core.queue size=6 enqueued=3304935974 full=0 discarded.full=0
discarded.nf=0 maxqsize=350911
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: WindowsRuleSet:
origin=core.queue size=0 enqueued=648043087 full=0 discarded.full=0
discarded.nf=0 maxqsize=156862
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: MailRuleSet:
origin=core.queue size=0 enqueued=307960420 full=0 discarded.full=0
discarded.nf=0 maxqsize=4243
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: NetworkDeviceRuleSet:
origin=core.queue size=0 enqueued=11008320810 full=0 discarded.full=0
discarded.nf=0 maxqsize=456471
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: main Q: origin=core.queue
size=120 enqueued=1496835 full=0 discarded.full=0 discarded.nf=0
maxqsize=1738
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: imudp(w1): origin=imudp
called.recvmmsg=8282193695 called.recvmsg=0 msgs.received=2833007396
Oct 5 14:11:47 its-syslog-up1 rsyslogd-pstats: imudp(w0): origin=imudp
called.recvmmsg=14154695256 called.recvmsg=0 msgs.received=8909255564
To estimate volume of logs this is receiving, I took a recent impstats
output for enqueued for all rule sets and added them, then subtracted the
values from an hour ago, divided by 3600 to get msgs/second and got 29521.
This is peak time of day for us so I would expect this to be the high point,
dropping to less than half that in the middle of the night. Messages are
still randomly lost in the middle of the night. If there's a better way to
estimate volume being processed by the machine, let me know!
Based on that volume, what would good settings be for threads for imudp and
ruleset? The rulesets are bound to both tcp and udp as we have some devices
that can't send over tcp.
Thanks for the help,
Dan
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Monday, October 5, 2015 1:11 PM
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] RHEL7.1 / rsyslog 8.x random message loss
the config you show doesn't match your explination (there's no use of
dynafile output in the config you are showing for example)
one thing that is odd, and usually wrong is that you have lots of threads
configured (2 for imudp, 4 for your ruleset), this is usually wrong and
causes excessive load.
can you share more of your config file?
Also, it would be useful to enable impstats, it will show what logs rsyslog
is getting what backlogs it's seeing on it's internal queues, etc.
what volume of logs is this box handling? I see you have the max log size
set very high (128K), but how many logs/sec?
David Lang
_______________________________________________
rsyslog mailing list
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailma
n_listinfo_rsyslog&d=BQICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=
0S5vJ8-FmQy6Qk5D6_T4U9EYbkCcMc4ijDuyUem89Lk&m=eCaEIfjQf1jiYfjxMW0Y4qyIDBWIx2
mzax7G74cNUeE&s=6VLuRizR9ocNzCNlf6GMFKcgSMuxnixtIByGlNRxxJk&e=
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professi
onal-2Dservices_&d=BQICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=0S
5vJ8-FmQy6Qk5D6_T4U9EYbkCcMc4ijDuyUem89Lk&m=eCaEIfjQf1jiYfjxMW0Y4qyIDBWIx2mz
ax7G74cNUeE&s=euTrOfcbgKu-79Oe3ij2d2QE9YlVaSqyJ2JFgtY-WqM&e=
What's up with rsyslog? Follow
https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d
=BQICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=0S5vJ8-FmQy6Qk5D6_T4
U9EYbkCcMc4ijDuyUem89Lk&m=eCaEIfjQf1jiYfjxMW0Y4qyIDBWIx2mzax7G74cNUeE&s=vGkT
Sgy6gaI5azRJJ4Y7CzltTOyyIwRGCEeZbZQqB9Y&e=
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
