On Wed, 4 Nov 2015, Rainer Gerhards wrote:
2015-11-04 17:24 GMT+01:00 Joe Blow <[email protected]>:
Thanks for the input Rainer! It definitely helps, and I love hearing some
of this from the horse's mouth. Let me start this post by saying i'm
extremely grateful for all the help that rsyslog has provided me throughout
my career. The support on this mailing list is arguably better than any of
the paid vendors i've used for logging/SIEM.
As much as I'd love to just give up on this, I'm far too confident in the
rsyslog tool to admit defeat. Rsyslog is a beast, but a beast with many
knobs :). I'm interested in potentially using the failover option, but the
DA queue configuration ease might keep me using that for the time being.
What about getting creative and moving the files to another rsyslog
instance (on the same box) that doesn't have any input modules? Here's my
thoughts:
Stop rsyslog.
Move rsyslog DA files and .qi file to another directory which a secondary
instance of rsyslog knows about (but has no input modules running).
Start rsyslog with input modules to get the realtime data flowing back in,
with an empty DA queue.
Turn on the second rsyslog instance which only knows about the backlog
files, and has no input modules.
My thought is that this would give at least 1 dedicated worker (per queue)
1 full core of resources to chug through the backlog, and only the
backlog. Is my logic sound?
not sure. Let's find the bottleneck. Is it i/o or CPU? What hard facts
tell you which one it is (you already commented partly on i/o, this
the more solid questions).
IMO, the disk queue should primarily be i/o intense, and not put a lot
of stress on the cpu. If so, the logic wouldn't work.
I expect that it's going to be a lot of cpu spent in system calls rather than
I/O. Unless syncing/checkpointing is turned on, most of the I/O (especially in
delivering messages from the queues) is not going to actually hit the disk.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
