I'm surprised it didn't help more than that. what do the impstats look
like.
Impstats looks good - only two dynafiles that still have evictions, which
I'll tune a little more now. See bottom of message for a full recent output.
at high traffic volumes increasing the batch size can also be helpful, but if
the queue is usually empty it probably won't make much difference.
We have several %fromhost% templates for servers. I noticed that there is a
rush of DNS requests when rsyslog first starts up but then it quiets down
due to the internal caching. I also have a local DNS cache running to point
to - not currently set - but when I did have the system pointed there, I
still had missing messages.
well, you've fixed a lot of things, the dynafile thrashing was a major problem.
DNS is probably less of an issue, and only an issue for UDP not TCP (UDP needs
to do a lookup for each packet, TCP only for each connection)
I personally recommend using fromhost-ip instead of fromhost. If you want to use
a name, use the hostname that the server put in the log message, if you need to
ignore that and see where it last came from, the IP is faster and has fewer
failure modes.
with your config, only traffic arriving on port 10517 will get written to the
testingruleset, internal syslog messages won't show up.
now, there are things that can cause you to see messages via tcpdump and not see
them in rsyslog.
The most common is iptables rules
But since you are writing to /tmp you could be running into permission problems
with SELinux/AppArmor (depending on your distro). try putting the file under
/var/log and see if it appears. It should appear when the first log message
arrives.
by the way, you don't need to do *.*, just list the actions directly, so the
following works just fine (permissions allowing)
ruleset(name="TestingRuleSet") {
/tmp/debug.log;RSYSLOG_DebugFormat
action(type="omfile" dynaFile="EduroamACSFile" template="EduroamACSFormat"
name="writeEduroamACSFile")
stop
} # end ruleset
in impstats, the /tmp/debug.log line will be action ## while the line after it
will have a name, check what it says about messages that are processed/etc
Can you deliver messages to port 10517 via TCP? there is an odd case where a
missing route can cause UDP messages to not be delivered to the app.
Oct 6 16:27:45 its-syslog-up1 rsyslogd-pstats: imudp(*:10517): origin=imudp
submitted=1221
Oct 6 16:27:45 its-syslog-up1 rsyslogd-pstats: imudp(*:10517): origin=imudp
submitted=0
yep, that says that messages are arriving
with impstats, do you have it configured to reset counters each time or not?
there is a little inaccuracy that creeps in when resetting the counters, and the
impstats analysis tool that Rainer pointed you at assumes the counters aren't
being reset.
We really need at least two sets of stats to be able to see what's happening.
now, looking at the impstats
I see one smoking gun:
main Q: origin=core.queue size=149 enqueued=492522977 full=51 discarded.full=0
discarded.nf=0 maxqsize=100000
maxqsize is such a nice round number that I'll bet it's the max you have
configured. That means that at some point rsyslog filled up the queue completely
and would have been unable to process any more messages. That means that UDP
messages that arrived during that time would have been dropped.
this also says it was full 51 times, so that is at least 51 messages lost.
on a high volume server, you probably want to bump the main queue size up a bit
larger as well.
another issue is that several actions show that they have been suspended and not
resumed, that will cause logs to be missing from those outputs (action2, 30, 43,
67). one advantage of the new action() format is the ability to name things.
Even if you don't name everything, naming several of them will help figure out
what the rest are.
as you said, there are still some dynafile cache settings that probably need to
be increased. They can slow everything down and cause the queue to fill (but
without knowing if the stats are reset, I don't know if the numbers are since
startup or since the last report) Since the busiest outputs are also the ones
with the highest miss rates, you may end up with a more dramatic I/O savings
once you fix them.
I don't understand why you have multiple lines for the same thing (imudp and
writeEduroamACSFile for example), unless it's one line per thread, in which case
these are showing that you really don't need multiple threads for the inputs,
you aren't using them :-)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
