On Wed, 23 Oct 2013, Robert McIntyre wrote:
So, I've had decent luck with Pavel's suggestion (field($timegenerated,':',3),
and it rotates around nicely based on the second.
I'm trying a slightly different approach, though, to try to get sub-second
rotation.
While I am in no way saying that you shouldn't continue trying to solve the
problem. I will ask if you really need the sub-second rotation in practice?
if you can rotate between outputs, and give each output it's own action queue,
with that queue having space for one second's worth of logs, then load balancing
across 3 outputs will give you 3x the throughput at the cost of delaying logs up
to ~3 seconds (if everything is on the edge of overload)
if it takes N seconds for one output to process 1 second's worth of logs, and
you spread the traffic across M outputs (where M > N to give you a little
headroom just in case), then your most delayed log will be delayed up N seconds
going to rotation every 1/10 second will change the delay to be N/10
This will cause your logs to arrive at the destination in a different order than
they were in the first place, but any load balancing scheme gives this
potential.
The batch mode processing in rsyslog where each worker thread grabs up to N
messages from the main queue and works on them while another worker thread grabs
the next up to N messages results in the same sort of thing (in this case N
defaults to 128 or 256 for the main queue, while a full second's worth of logs
will almost certinly be a much larger number :-)
When doing load balancing on the network layer, I try to set the rebind interval
large enough that it only rebalances once a second (or a handful of times per
second) and when I put logs into something that doesn't take a stream well (like
Splunk), I write the logs to disk and move them to the destination once a
minute, and even that ends up not being really significant in practice.
Yes, at lower traffic levels the balancing is choppy with different workers
going visibly idle for a bit, but finer grain balancing still has the workers
going idle, just for shorter periods that you are not seeing due to the course
granularity of your measurements :-)
if you balance per second across 10 outputs and you have 8 of them idle looking
at top with a refresh interval of 1, then changed to balancing every 1/10 second
you would still have 8 of them idle if you were able to set top to give you a
refresh interval of 1/10
the real question to think about is what is the maximum overall delay that you
can live with.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
