Hi Ben,
2013/3/2 Ben Bradley <[email protected]>
> On 01/03/13 14:51, Radu Gheorghe wrote:
>
>> Hello Ben,
>>
>> I'm not sure another rsyslog on the Logstash side will help if the
>> bottleneck is on indexing to Elasticsearch.
>>
>> AFAIK logstash has an internal buffer of 20 or so entries, and when that's
>> full (because the output is not fast enough) it blocks the input. At this
>> point, you need to queue on whatever outputs to Logstash. Whether that's
>> done by rsyslog on the client or on the Logstash side, I don't think it
>> matters.
>>
>> I would try a performance test using elasticsearch_http with bulks. After
>> you find a good bulk size for your setup, 10 to 100x gain in performance
>> won't be a surprise.
>>
>> If that doesn't help, I see two options:
>> - scale Logstash on multiple instances/servers
>> - use omelasticsearch. I've used it quite a lot, and will be using it a
>> whole lot more. No significant issues so far - if ES goes down, it will
>> queue. As for performance, I've indexed 20K logs/s on a laptop.
>>
>> Best regards,
>> Radu
>>
>
> I've switched to elasticsearch_http in the meantime which should give me
> much higher throughput. I've set a batch size of 50 which should be fine
> until I've got all 6 web servers online. At that point I might need to push
> it up to 100 to make it so there's maximum 1 HTTP request to ElasticSearch
> per second.
>
> I really like Rsyslog's queue management though.
>
> At the moment my logs are just going over the network using tcp syslog
> (omfwd). Are there any other transport formats (JSON?) that are supported
> by Rsyslog that can be read by logstash?
>
In terms of actual transport, with rsyslog you can choose between TCP, UDP
and RELP. Logstash has an input plugin for RELP, and the one for "syslog"
listens on both TCP and UDP.
As for the formats, you can make it output in various ways via templates.
Including JSON, if you want, with something like:
$template
testTemplate,"{\"message\":\"%msg:::json%\",\"host\":\"%HOSTNAME:::json%\",\"severity\":\"%syslogseverity%\",\"date\":\"%timereported:1:19:date-rfc3339%.%timereported:1:3:date-subseconds%\",\"tag\":\"%syslogtag:::json%\"}"
As for how Logstash would parse that, I'm not sure. Maybe you can ask the
guys on the Logstash mailing list for details, but it seems like the JSON
filter might do it:
http://logstash.net/docs/1.1.9/filters/json
>
> I still worry about syslog message size limits, although I actually see
> very few of these as the longest log messages are cut off by being over
> Apache's 8192 byte request limit or by /sbin/logger
>
With rsyslog you can change that with MaxMessageSize (defaults to 2k). Take
a look here for more details:
http://www.rsyslog.com/doc/rsyslog_conf_global.html
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
