Re: [rsyslog] Rsyslog queue in front of Logstash

2013-03-02 Thread Ben Bradley
On 01/03/13 14:51, Radu Gheorghe wrote: Hello Ben, I'm not sure another rsyslog on the Logstash side will help if the bottleneck is on indexing to Elasticsearch. AFAIK logstash has an internal buffer of 20 or so entries, and when that's full (because the output is not fast enough) it blocks

Re: [rsyslog] Rsyslog queue in front of Logstash

2013-03-02 Thread Radu Gheorghe
Hi Ben, 2013/3/2 Ben Bradley bbradle...@gmail.com On 01/03/13 14:51, Radu Gheorghe wrote: Hello Ben, I'm not sure another rsyslog on the Logstash side will help if the bottleneck is on indexing to Elasticsearch. AFAIK logstash has an internal buffer of 20 or so entries, and when that's

Re: [rsyslog] Rsyslog queue in front of Logstash

2013-03-02 Thread David Lang
On Sat, 2 Mar 2013, Ben Bradley wrote: At the moment my logs are just going over the network using tcp syslog (omfwd). Are there any other transport formats (JSON?) that are supported by Rsyslog that can be read by logstash? look at the lumberjack option in logstash, rsyslog supports JSON

[rsyslog] Rsyslog queue in front of Logstash

2013-03-01 Thread Ben Bradley
Hi everyone So I've got Rsyslog happily transmitting log messages over the network to Logstash. I have disk assisted queueing on the rsyslog log clients. Sometimes I don't think Logstash can keep up on the other end, it blocks because it can't get data into ElasticSearch fast enough. I've not

Re: [rsyslog] Rsyslog queue in front of Logstash

2013-03-01 Thread Ben Bradley
On Fri, 1 Mar 2013 10:20:28 + Ben Bradley bbradle...@gmail.com wrote: Hi everyone So I've got Rsyslog happily transmitting log messages over the network to Logstash. I have disk assisted queueing on the rsyslog log clients. Sometimes I don't think Logstash can keep up on the other end,

Re: [rsyslog] Rsyslog queue in front of Logstash

2013-03-01 Thread Radu Gheorghe
Hello Ben, I'm not sure another rsyslog on the Logstash side will help if the bottleneck is on indexing to Elasticsearch. AFAIK logstash has an internal buffer of 20 or so entries, and when that's full (because the output is not fast enough) it blocks the input. At this point, you need to queue

Re: [rsyslog] Rsyslog queue in front of Logstash

2013-03-01 Thread David Lang
On Fri, 1 Mar 2013, Ben Bradley wrote: Hi everyone So I've got Rsyslog happily transmitting log messages over the network to Logstash. I have disk assisted queueing on the rsyslog log clients. Sometimes I don't think Logstash can keep up on the other end, it blocks because it can't get data