Thanks for your suggestion... Definitely looking at a hardware/software RR load balancer. In the meantime, also looking for other solutions using existing technologies. Also looking at shifting to TCP and using rsyslog's queuing. That would be my preferred approach for several reasons (reliability, etc.).
Thanks! Robert -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Wednesday, October 23, 2013 4:56 PM To: rsyslog-users Subject: Re: [rsyslog] Another approach to action load balancing so are you sending the logs to these destinations via UDP syslog? If so, you can put a round-robin load balancer between your syslog relay and the destination, set rebindinterval to 1000 (or 10,000) and the load will be spread evenly. The thing I would be worried about from your description is that these systems may loose messages that arrive too fast, even if only 1000 arrive in too short a period (as an example) several years ago I had a AIX server that if it was sent too many TCP connections would ack the connection to the sender, and then forget about it, causing the firewalls in between sender and receiver (who did not loose track of the connection) to overload and crash. If the firewall had not crashed, the senders would eventually crash (and in any case, made no progress to service customers). So I've seen cases where things react really badly to high volumes of connections in a short time. David Lang On Wed, 23 Oct 2013, Robert McIntyre wrote: > Great questions, David. In a nutshell, it's all about load-balancing > the destinations. Say (for example) that my downstream recipients > have a cap of 20K messages per second, I'm trying to respect those. > There's some question (I'm following up on) whether if I were to dump > 60K into one for a second, then switch to the next one for a second, > and so forth, such that the > *average* rate was < 20K mps that would be OK. I'm assuming that it > won't be, so I need to continue to work towards getting a more true distribution. > > In an ideal world, I'd be able to use TCP to forward to the > destinations, and just let standard TCP throttling take care of it. > But, I'm stuck with UDP for now, so can't do that. A true round-robin > would be the gold standard, but absent a "$messagecount" property, or > global variables (or a full-fledged load balancing feature, which I > offhand don't think rsyslog needs, rather, it needs to allow people to > implement their own via the above methods), I'm left pursuing these avenues. > > As for latency, I'm actually OK with several minutes of latency. Not > concerned about message order, either. > > Thanks!!! > Robert > > >> Date: Wed, 23 Oct 2013 16:01:11 -0700 >> From: [email protected] >> To: [email protected] >> Subject: Re: [rsyslog] Another approach to action load balancing >> >> On Wed, 23 Oct 2013, Robert McIntyre wrote: >> >>> So, I've had decent luck with Pavel's suggestion >>> (field($timegenerated,':',3), and it rotates around nicely based on the second. >>> >>> I'm trying a slightly different approach, though, to try to get >>> sub-second rotation. >> >> While I am in no way saying that you shouldn't continue trying to >> solve the problem. I will ask if you really need the sub-second rotation in practice? >> >> if you can rotate between outputs, and give each output it's own >> action queue, with that queue having space for one second's worth of >> logs, then load balancing across 3 outputs will give you 3x the >> throughput at the cost of delaying logs up to ~3 seconds (if >> everything is on the edge of overload) >> >> if it takes N seconds for one output to process 1 second's worth of >> logs, and you spread the traffic across M outputs (where M > N to >> give you a little headroom just in case), then your most delayed log >> will be delayed up N seconds >> >> going to rotation every 1/10 second will change the delay to be N/10 >> >> This will cause your logs to arrive at the destination in a different >> order than they were in the first place, but any load balancing >> scheme gives this potential. >> >> The batch mode processing in rsyslog where each worker thread grabs >> up to N messages from the main queue and works on them while another >> worker thread grabs the next up to N messages results in the same >> sort of thing (in this case N defaults to 128 or 256 for the main >> queue, while a full second's worth of logs will almost certinly be a >> much larger number :-) >> >> >> When doing load balancing on the network layer, I try to set the >> rebind interval large enough that it only rebalances once a second >> (or a handful of times per >> second) and when I put logs into something that doesn't take a stream >> well (like Splunk), I write the logs to disk and move them to the >> destination once a minute, and even that ends up not being really significant in practice. >> >> >> >> Yes, at lower traffic levels the balancing is choppy with different >> workers going visibly idle for a bit, but finer grain balancing still >> has the workers going idle, just for shorter periods that you are not >> seeing due to the course granularity of your measurements :-) >> >> if you balance per second across 10 outputs and you have 8 of them >> idle looking at top with a refresh interval of 1, then changed to >> balancing every 1/10 second you would still have 8 of them idle if >> you were able to set top to give you a refresh interval of 1/10 >> >> >> the real question to think about is what is the maximum overall delay >> that you can live with. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
