On 10/23/2013 03:42 PM, David Lang wrote:
On Wed, 23 Oct 2013, Erik Steffl wrote:
On 10/23/2013 12:56 PM, Pavel Levshin wrote:
IMHO, for high-load TCP, rebindinterval is counter-productive, because
you need to establish a new connection each time. But yes, this
parameter exists.
why would it be counter productive? It should not be done too
frequently, that would be counter productive. But done at the right
interval the cost of re-establishing TCP connection is neglible and
makes it possible to do load balancing.
so you are saying the RebindInterval is just missing in docs?
I think that the RELp module just does not have rebindInterval, you can
try it and see if it works, but I suspect that it won't.
it seems like it should be working but it segfaults when rebinding
should occur (in relpCltHintBurstEnd in librelp.so.0), e.g. when I set
it to 5 it segfaults every 5 messages
Is there anybody successfully using RebindInterval with omrelp? Which
rsyslog version? I am using 7.5.5-0adiscon2 (from adiscon repo
http://ubuntu.adiscon.com/v7-devel quantal/)
Load balancing is very dependent on nature of your load, on your
bottlenecks, and so on. Why do you want to balance TCP connections?
If nothing else it provides high availability, spefically we are
using Amazon elastic load balancer (Amazon makes sure it's up and
running) with autoscaling group behind it (again, Amazon makes sure
that the bad machines are terminated, that at least N machines are up
and running (we specify N) etc.
well, with RELP, you actually don't need rebindinterval for HA, when the
ELB or an instance dies, traffic will pause temporarily, timeout, and
establish a new connection.
yes, I get that, tested it and it works great.
this particular setup also gives me high availability from the other
side, sort of. Amazon makes sure that load balancer and the machines
behind it are up, creates new machine if there is something wrong with a
given machine (or creates more machines if needed to handle the load).
from this perspective the actual load balancing does not make any
difference (i.e. it helps us even if we just put one machine behind load
balancer)
rebindinterval will not change this. rebindinterval just makes it so
that when everything is working well, it will disconnect after N
messages and reconnect.
Other than that we are doing load balancing for the obvious reason,
there are tons of machines that send tons of logs, the collector
machine should not become a bottleneck so we want to have more than
one machine. Load balancer is a way to do that. Other solution would
be to have bunch of machines that all those log senders would
magically know about which is not exactly a solution (more like
nightmare).
what sort of traffic volume are you dealing with? a single system can
handle a pretty high traffic volume, so unless you are sending the
traffic to a slow destination of have a _very_ high traffic volume
(several hundred thousands of messages per second) a single box should
be able to handle the full load.
I don't think we need it right now but we're just rolling it out,
still in mostly experimental mode. At the moment we are somewhere in
range of 1000 messages per second but it will go at least few times up.
The messages are relatively big (not huge but slightly bigger than one
liners one usually sees in /var/log/syslog, we record lot of details
about requests).
erik
David Lang
erik
--
Pavel Levshin
23.10.2013 23:26, Erik Steffl:
What are recommendations for use of rsyslog with loadbalancers? We
have setup where number of machines run rsyslog and forward syslog
messages to central location using RELP protocol (using omrelp module).
The central location is a load balancer with cluster of machines
behind it (that run rsyslog and use imrelp).
I assume this is not an uncommon scenario so hoping that there is a
way to do this in a way that does load balancing.
The problem is that TCP connection stays open for a long time so there
is effectively no load balancing done. Looking for a solution I found
http://www.rsyslog.com/load-balancing-for-rsyslog/ which mentions
RebindInterval which is documented in omfwd docs
http://www.rsyslog.com/doc/omfwd.html
However I don't see RebindInterval in omrelp docs
http://www.rsyslog.com/doc/omrelp.html and when I try to use it
rsyslog segfaults (in relpCltHintBurstEnd in librelp.so.0, also sent
email specifically about that).
Should RebindInterval work for omrelp? Or is there some other way to
re-open TCP connection so that load balancing works?
Thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
