On Tue, 31 Jan 2012, Radu Gheorghe wrote:
Thanks a lot, David. These solutions seem much better than DNS round robin.
@Rainer: I'm having trouble with setting up more exact performance
tests, but the system is supposed to scale to something like 50K
messages per second. And I just assumed that one sever won't handle
the load, especially since these machines are slow.
But you have a good point, because the external program inserts these
logs in ElasticSearch. And I have ElasticSearch on the same server
right now, which I don't have to. Using a dedicated server for Rsyslog
only might not cut it in the long run, but it would probably work well
for a while. More than enough for me to set up a cluster :D
the best thing that you could do for your performance is to commission the
writng of an output module that would let rsyslog insert the messages into
ElasticSearch instead of doing it with an external program.
At first glance this may seem like a trivial change, but the killer
feature that you can take advantage of with an output module is the
ability to handle multiple log messages as a single transaction.
I'm not familiar with ElasticSearch, but it's common for databases to be
able to handle inserts of 100 or even 1000 records as a single transaction
at exactly the same transaction/sec rate as inserting a single record per
transaction (or at a very slight reduction in insert rate). I've seen good
database setups where 10,000 inserts as a single transaction was only 1/2
the transaction rate of one insert per transaction (a 5,000x speedup)
Adiscon does this sort of work (contact Rainer directly if you want a
quote)
but 50K logs/sec is not likely to end up with rsyslog as the bottlneck.
You should setup a test environment and stress test things to see how high
you can push the message rate before you can't keep up. There are a number
of variables that can end up being the bottleneck and you want to find
these in testing, not in production :-)
The fist thing is that you want to be running a very recent rsyslog (5.8.x
or 6.x), the speedups in rsyslog since 4.x (which is in RHEL5 I believe)
are very significant. 6.3.x introduces a DNS cache that can be a drastic
speedup if you need DNS lookups (if not, you can start rsyslog with -x to
disable them on earlier versions)
you also need to define 'slow hardware', one person's slow hardware is
another person's mid-range server :-)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
