On Thu, 24 Oct 2013, Pavel Levshin wrote:
24.10.2013 3:06, Erik Steffl:
this particular setup also gives me high availability from the other
side, sort of. Amazon makes sure that load balancer and the machines behind
it are up, creates new machine if there is something wrong with a given
machine (or creates more machines if needed to handle the load).
Oops, I think this could justify RebindInterval for TCP, for some cases. I
mean, to be able to "create more machines if needed to handle the load"
automatically, you need to reestablish connections from time to time.
But it still has some implications, like connection delay and TCP slow-start.
You mileage may vary.
Yep, there is a delay at the start of the connection, which is why you want to
rebind as infrequently as possible.
but the overhead of doing this is pretty minor if you only do it once a second
or so.
In any case, what I like to do is to have a 4-tier system
app servers send via UDP over the local subnet to:
1st tier relay servers (Edge Aggregators), which:
cleanup messages, add metadata, etc and send the results via TCP or RELP to:
2nd tier relay servers (Core Aggregators), which do nothing but relay the
messages with no format changes out via UDP to farms of analysis machines.
This final relay is where you need to do the rebind to load balance, because
even fairly modest relay servers can handle very high log volumes if all they
are doing is gathering and forwarding the messages.
with UDP you don't have the slow-start and handshaking overhead that you do with
rebinding TCP connections, but there's still a little OS overhead to make a new
connection.
now, I'm not sure which is _really_ better performance when you set the
rebindinterval to something in the 10k-100k message range between UDP and TCP
with compression turned on. Compression on the TCP connection is a fairly new
feature, it may save enough bandwidth to make it faster overall than UDP even
with slow start, etc.
if you have a requirement to encrypt the data over the wire, you have no choice
but to use TCP or RELP (relp does not currently support compression)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
