Thanks, Ciprian. I ran rsyslogd -dn and I can see that imfile is reading
changes from cassandra.log, but it's not sending them to elasticsearch
until I restart. Thoughts?
Rsyslog trace before restart doesn't have any omelasticsearch logs:
...
5623.332950664:imfile.c : DDDD: imfile: in_processEvent (wd=2) event
Mask='0x00000002'
5623.332954364:imfile.c : DDDD: imfile: wd 2 got file 0x7ffe24002190,
dir -1
5623.332965540:imfile.c : strm 0x7ffe240057c0: file 7 read 0 bytes
5623.332978458:imfile.c : stream checking for file change on
'/var/log/cassandra/cassandra.log', inode 264465/264465
5623.332982339:imfile.c : DDDDD: readLine returns[-2026]: '(null)'
[*ppCStr 0x7ffe24026850]
Rsyslog after restart which causes logs to be sent to elasticsearch:
...
6033.733447868:action 2 queue:Reg/w0: omelasticsearch: result doAction:
-2121 (bulkmode 1)
6033.733450384:action 2 queue:Reg/w0: omelasticsearch: endTransaction init
6033.733482066:action 2 queue:Reg/w0: omelasticsearch: endTransaction,
batch: '{"index":{"_index": "logstash-2015.12.12","_type":"cassandra"}}
{ "@timestamp": "2015-12-12T21:33:53.484399+00:00", "host": "m0051948",
"severity": "notice", "facility": "local2", "syslogtag": "cassandra",
"filename": "cassandra.log", "message": "Enqueuing flush of
Memtable-local@518770933(84\/840 serialized\/live bytes, 4 ops)",
"log_time": "21:26:55,179", "log_level": "INFO" }
{"index":{"_index": "logstash-2015.12.12","_type":"cassandra"}}
Thanks,
Alec
On Sat, Dec 12, 2015 at 12:32 PM, Ciprian Hacman <
[email protected]> wrote:
> Hi Alec,
>
> Actually, Rsyslog doesn't wait for the batch size to fill. Once it gets a
> log, it starts preparing a batch and sends it as soon as it's ready to send
> it. We use queue.dequeueslowdown to slow it down to get more logs in a
> batch.
>
> Do you log suspensions?
>
> global (
> action.reportSuspension="on"
> action.reportSuspensionContinuation="on"
> )
>
>
> You might also want to run Rsyslog manually using "rsyslogd -dn". It will
> be very verbose, but you may get an idea of what happens.
>
> Regards,
> Ciprian
> ---
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
> On Sat, Dec 12, 2015 at 8:35 PM, Alec Swan <[email protected]> wrote:
>
> > I am sending logs from rsyslog 8.14.0 to ElasticSearch 1.6. I configured
> > omelasticsearch module as shown below. In cases where log traffic is low
> > the logs don't get sent to elasticsearch until I restart rsyslog. I am
> > assuming that's because omelasticsearch module keeps batching the logs
> > until dequeuebatchsize is reached. This is a problem for applications
> that
> > generate startup logs and after that infrequently generate big bursts of
> > logs, e.g. during some maintenance process.
> >
> > Is my theory about dequeuebatchsize correct? If so, is there a way to
> > define a timeout within which logs should be sent to elasticsearch even
> if
> > dequeuebatchsize has not been reached?
> >
> > Thanks
> >
> > action(
> > type = "omelasticsearch"
> > template = "es-payload"
> > dynSearchIndex = "on"
> > searchIndex = "logstash-index"
> > searchType = "<%= logfile['tag'] %>"
> > server = "<%= elasticsearch_config['host'] %>"
> > serverport = "<%= elasticsearch_config['port'] %>"
> > errorFile = "<%= @log_dir %>/ES-error.log"
> > bulkmode = "on"
> > action.resumeretrycount="5" # retry if ES is unreachable (-1
> for
> > infinite retries)
> > action.resumeInterval="60"
> > queue.dequeuebatchsize="1000" # ES bulk size
> > queue.type="linkedlist"
> > queue.size="100000"
> > queue.workerthreads="5"
> > queue.spoolDirectory="<%= @spool_dir %>"
> > queue.filename="omelasticsearch-queue"
> > queue.maxfilesize="100m"
> > queue.maxdiskspace="1g"
> > queue.highwatermark="50000" # when to start spilling to disk
> > queue.lowwatermark="20000" # when to stop spilling to disk
> > queue.saveonshutdown="on"
> > )
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
