It shouldn't matter, but what is the rate of log delivery? is there any chance
that it is waiting to deliver a full batch?
I would consider putting this in a ruleset and moving the queue to the ruleset.
I would then have the ruleset contain two items
1. the output to ES
2. a write to a debug log locally (not necessarily the full messages, timestamp
would be enough)
you can then see if the local file in growing while things are not yet showing
up in ES to see if the issue is on the sending side or on the receiving side.
David Lang
On Tue, 24 May 2016, Alec Swan wrote:
Date: Tue, 24 May 2016 22:17:22 -0600
From: Alec Swan <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Logs are delayed being pushed to Elasticsearch
Hello,
I recently upgraded to rsyslog 8.18 and was happy to see that on disk
queues no longer cause rsyslog to get in a bad state. However, now I see
very long delays (several hours) of logs being pushed to Elasticsearch. It
seems that somehow the logs are being buffered on the client for several
hours because eventually they do show up in Elasticsearch. I don't see any
errors in /var/log/rsyslog/ES-error.log (see config below) or
/var/log/messages.
I enabled impstats but didn't see any errors related to omelasticsearch.
What else can I do to troubleshoot this?
Here is my omelasticsearch config:
action(
type = "omelasticsearch"
template = "es-payload"
dynSearchIndex = "on"
searchIndex = "logstash-index"
searchType = "syslog"
server = "127.0.0.1"
serverport = "9200"
uid = "xxx"
pwd = "yyy"
errorFile = "/var/log/rsyslog/ES-error.log"
bulkmode = "on"
action.resumeretrycount="-1" # retry if ES is unreachable (-1 for
infinite retries)
action.resumeInterval="60"
queue.dequeuebatchsize="1000" # ES bulk size
queue.type="linkedlist"
queue.size="100000"
queue.workerthreads="5"
queue.timeoutworkerthreadshutdown="2000"
queue.spoolDirectory="/var/spool/rsyslog"
queue.filename="omelasticsearch-queue"
queue.maxfilesize="100m"
queue.maxdiskspace="1g"
queue.highwatermark="80000" # when to start spilling to disk
queue.lowwatermark="20000" # when to stop spilling to disk
queue.saveonshutdown="on"
)
Thanks,
Alec
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
