David, The rate of delivery is about 1 log per second. If you are referring to queue.dequeuebatchsize="1000" batch size, then I would expect the logs to be batched for 15-20 minutes. However, I am observing delays of multiple hours.
When I restart rsyslog all buffered logs get sent to elasticsearch. I had logging to log file enabled before and could see all logs being written to log files correctly. I enabled it again and will keep an eye on it, but I am sure the problem is pushing to ES. I currently have a host which hasn't sent logs for about 12 hours. The following are the last logs I received from that node. Anything I can do to troubleshoot while the host is in a bad state? May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880 host:myhost hostip:10.0.0.1 severity:debug facility:syslog syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog message:omelasticsearch-myapp-launcher.log: origin=core.action processed=333 failed=0 suspended=0 suspended.duration=0 resumed=0 _id:AVTnrAgpFO4BDB55DTh2 _type:syslog _index:logstash-syslog-myapp-2016.05.25 _score: May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880 host:myhost hostip:10.0.0.1 severity:debug facility:syslog syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog message:action 4: origin=core.action processed=336 failed=0 suspended=0 suspended.duration=0 resumed=0 _id:AVTnrAgpFO4BDB55DTh3 _type:syslog _index:logstash-syslog-myapp-2016.05.25 _score: May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880 host:myhost hostip:10.0.0.1 severity:debug facility:syslog syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog message:omelasticsearch-syslog queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 _id:AVTnrAlsFO4BDB55DTiD _type:syslog _index:logstash-syslog-myapp-2016.05.25 _score: May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880 host:myhost hostip:10.0.0.1 severity:debug facility:syslog syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog message:omelasticsearch-myapp-launcher.log queue[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 _id:AVTnrAlsFO4BDB55DTiF _type:syslog _index:logstash-syslog-myapp-2016.05.25 _score: Thanks, Alec On Tue, May 24, 2016 at 10:35 PM, David Lang <[email protected]> wrote: > It shouldn't matter, but what is the rate of log delivery? is there any > chance that it is waiting to deliver a full batch? > > I would consider putting this in a ruleset and moving the queue to the > ruleset. I would then have the ruleset contain two items > > 1. the output to ES > 2. a write to a debug log locally (not necessarily the full messages, > timestamp would be enough) > > you can then see if the local file in growing while things are not yet > showing up in ES to see if the issue is on the sending side or on the > receiving side. > > David Lang > > On Tue, 24 May 2016, Alec Swan wrote: > > Date: Tue, 24 May 2016 22:17:22 -0600 >> From: Alec Swan <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: [rsyslog] Logs are delayed being pushed to Elasticsearch >> >> >> Hello, >> >> I recently upgraded to rsyslog 8.18 and was happy to see that on disk >> queues no longer cause rsyslog to get in a bad state. However, now I see >> very long delays (several hours) of logs being pushed to Elasticsearch. It >> seems that somehow the logs are being buffered on the client for several >> hours because eventually they do show up in Elasticsearch. I don't see any >> errors in /var/log/rsyslog/ES-error.log (see config below) or >> /var/log/messages. >> >> I enabled impstats but didn't see any errors related to omelasticsearch. >> What else can I do to troubleshoot this? >> >> Here is my omelasticsearch config: >> >> action( >> type = "omelasticsearch" >> template = "es-payload" >> dynSearchIndex = "on" >> searchIndex = "logstash-index" >> searchType = "syslog" >> server = "127.0.0.1" >> serverport = "9200" >> uid = "xxx" >> pwd = "yyy" >> errorFile = "/var/log/rsyslog/ES-error.log" >> bulkmode = "on" >> action.resumeretrycount="-1" # retry if ES is unreachable (-1 for >> infinite retries) >> action.resumeInterval="60" >> queue.dequeuebatchsize="1000" # ES bulk size >> queue.type="linkedlist" >> queue.size="100000" >> queue.workerthreads="5" >> queue.timeoutworkerthreadshutdown="2000" >> queue.spoolDirectory="/var/spool/rsyslog" >> queue.filename="omelasticsearch-queue" >> queue.maxfilesize="100m" >> queue.maxdiskspace="1g" >> queue.highwatermark="80000" # when to start spilling to disk >> queue.lowwatermark="20000" # when to stop spilling to disk >> queue.saveonshutdown="on" >> ) >> >> Thanks, >> >> Alec >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
