On 01/03/13 14:51, Radu Gheorghe wrote:
Hello Ben,
I'm not sure another rsyslog on the Logstash side will help if the
bottleneck is on indexing to Elasticsearch.
AFAIK logstash has an internal buffer of 20 or so entries, and when that's
full (because the output is not fast enough) it blocks the input. At this
point, you need to queue on whatever outputs to Logstash. Whether that's
done by rsyslog on the client or on the Logstash side, I don't think it
matters.
I would try a performance test using elasticsearch_http with bulks. After
you find a good bulk size for your setup, 10 to 100x gain in performance
won't be a surprise.
If that doesn't help, I see two options:
- scale Logstash on multiple instances/servers
- use omelasticsearch. I've used it quite a lot, and will be using it a
whole lot more. No significant issues so far - if ES goes down, it will
queue. As for performance, I've indexed 20K logs/s on a laptop.
Best regards,
Radu
I've switched to elasticsearch_http in the meantime which should give me
much higher throughput. I've set a batch size of 50 which should be fine
until I've got all 6 web servers online. At that point I might need to
push it up to 100 to make it so there's maximum 1 HTTP request to
ElasticSearch per second.
I really like Rsyslog's queue management though.
At the moment my logs are just going over the network using tcp syslog
(omfwd). Are there any other transport formats (JSON?) that are
supported by Rsyslog that can be read by logstash?
I still worry about syslog message size limits, although I actually see
very few of these as the longest log messages are cut off by being over
Apache's 8192 byte request limit or by /sbin/logger
Cheers, Ben
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
