2012/1/31 <[email protected]>: > the best thing that you could do for your performance is to commission the > writng of an output module that would let rsyslog insert the messages into > ElasticSearch instead of doing it with an external program. > > At first glance this may seem like a trivial change, but the killer feature > that you can take advantage of with an output module is the ability to > handle multiple log messages as a single transaction. > > I'm not familiar with ElasticSearch, but it's common for databases to be > able to handle inserts of 100 or even 1000 records as a single transaction > at exactly the same transaction/sec rate as inserting a single record per > transaction (or at a very slight reduction in insert rate). I've seen good > database setups where 10,000 inserts as a single transaction was only 1/2 > the transaction rate of one insert per transaction (a 5,000x speedup) >
I know there is an ES plugin available in the development version but I couldn't get it to work: http://kb.monitorware.com/can-install-elasticsearch-output-module-t11309.html My script does bulk inserts already (I'm inserting each second). So there shouldn't be a significant performance gain by using an rsyslog plugin. Although I would prefer using plugins anyway. > Adiscon does this sort of work (contact Rainer directly if you want a quote) > > but 50K logs/sec is not likely to end up with rsyslog as the bottlneck. You > should setup a test environment and stress test things to see how high you > can push the message rate before you can't keep up. There are a number of > variables that can end up being the bottleneck and you want to find these in > testing, not in production :-) Yes, I will do some proper testing and consider solutions afterwards. Sorry for not doing my homework properly in the first place :( > > The fist thing is that you want to be running a very recent rsyslog (5.8.x > or 6.x), the speedups in rsyslog since 4.x (which is in RHEL5 I believe) are > very significant. 6.3.x introduces a DNS cache that can be a drastic speedup > if you need DNS lookups (if not, you can start rsyslog with -x to disable > them on earlier versions) > > you also need to define 'slow hardware', one person's slow hardware is > another person's mid-range server :-) I guess defining 'slow hardware' must come after proper testing... So I won't go there for now :) _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
