On Thu, 17 Mar 2016, David Lang wrote:
I have been running 8.17 from the repo combined with a copy of liblognorm
2.0.0 that I compiled during the 8.16 timeframe. Since the upgrade to 8.17
I've been getting a few coredumps. Since I enabled async writing to
dynamically generated files, rsyslog is using 1500+ threads (up from ~17
threads prior to this change) and has been running a 32G box OOM repeatedly.
Figuring that there is a reasonable chance that these problems are due to the
mixing of versions, I compiled from today's git tree and deployed that to a
server that's receiving a flood of logs from queues that are flushing to it.
When I deployed the new version, the throughput dropped noticably (~30% drop
from handling ~300K messages/min to handling 200K messages/min)
This is with no config changes, just changing the binary packages.
This is with a rather complex ruleset (11 queues, 75+ actions, at least 4
mmnormalize calls, one with a 1500 line ruleset)
Since pushing the new version to this machine, no coredumps and no OOM (not
definitive given that it's only been about 4 hours, but highly suggestive)
note that cpu utilization also dropped with the new version, from ~550% to
~450%, so my guess is that the performance problem is some locking change.
it didn't hit OOM, but it was steadily eating more RAM while under constant load
(with the queue sizes remaning stable)
I'm turning off async writes and will report on if this stabilizes the memory
utilization.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
