David,
After enabling the omfile action to write to a log file before sending to
elasticsearch (see below) I confirmed that logs stop being written to
/var/log/rsyslog/output.log and to ES at the same time even though the log
file being monitored /var/log/file-to-monitor.log keeps getting new logs.
So, the problem seems to be related to imfile input and not to output
plugins or queue settings.
Maybe it's related to how I am monitoring the log file?
*module(load = "imfile")*
*input(*
* type = "imfile"*
* File = "/var/log/file-to-monitor.log"*
* Tag = "myapp"*
* startmsg.regex = "^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"*
* Facility = "local2"*
* Ruleset = "myruleset"*
* )*
*ruleset(name = "myruleset") {*
* action(type = "mmnormalize" rulebase =
"/etc/rsyslog.d/rules/myrule.rb")*
* action(type = "omfile" template="es-payload"
file="/var/log/rsyslog/output.log" FileCreateMode="0644")*
* action(type = "omelasticsearch" queue.xxx)*
*}*
Thanks,
Alec
On Wed, May 25, 2016 at 11:13 AM, Alec Swan <[email protected]> wrote:
> David,
>
> The rate of delivery is about 1 log per second. If you are referring to
> queue.dequeuebatchsize="1000"
> batch size, then I would expect the logs to be batched for 15-20 minutes.
> However, I am observing delays of multiple hours.
>
> When I restart rsyslog all buffered logs get sent to elasticsearch. I had
> logging to log file enabled before and could see all logs being written to
> log files correctly. I enabled it again and will keep an eye on it, but I
> am sure the problem is pushing to ES.
>
> I currently have a host which hasn't sent logs for about 12 hours. The
> following are the last logs I received from that node. Anything I can do to
> troubleshoot while the host is in a bad state?
>
> May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880
> host:myhost hostip:10.0.0.1 severity:debug facility:syslog
> syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog
> message:omelasticsearch-myapp-launcher.log: origin=core.action
> processed=333 failed=0 suspended=0 suspended.duration=0 resumed=0
> _id:AVTnrAgpFO4BDB55DTh2 _type:syslog
> _index:logstash-syslog-myapp-2016.05.25 _score:
> May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880
> host:myhost hostip:10.0.0.1 severity:debug facility:syslog
> syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog
> message:action 4: origin=core.action processed=336 failed=0 suspended=0
> suspended.duration=0 resumed=0 _id:AVTnrAgpFO4BDB55DTh3 _type:syslog
> _index:logstash-syslog-myapp-2016.05.25 _score:
> May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880
> host:myhost hostip:10.0.0.1 severity:debug facility:syslog
> syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog
> message:omelasticsearch-syslog queue[DA]: origin=core.queue size=0
> enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
> _id:AVTnrAlsFO4BDB55DTiD _type:syslog
> _index:logstash-syslog-myapp-2016.05.25 _score:
> May 25th 2016, 05:28:05.880 @timestamp:May 25th 2016, 05:28:05.880
> host:myhost hostip:10.0.0.1 severity:debug facility:syslog
> syslogtag:rsyslogd-pstats: programname:rsyslogd-pstats logtag:syslog
> message:omelasticsearch-myapp-launcher.log queue[DA]: origin=core.queue
> size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
> _id:AVTnrAlsFO4BDB55DTiF _type:syslog
> _index:logstash-syslog-myapp-2016.05.25 _score:
>
> Thanks,
>
> Alec
>
> On Tue, May 24, 2016 at 10:35 PM, David Lang <[email protected]> wrote:
>
>> It shouldn't matter, but what is the rate of log delivery? is there any
>> chance that it is waiting to deliver a full batch?
>>
>> I would consider putting this in a ruleset and moving the queue to the
>> ruleset. I would then have the ruleset contain two items
>>
>> 1. the output to ES
>> 2. a write to a debug log locally (not necessarily the full messages,
>> timestamp would be enough)
>>
>> you can then see if the local file in growing while things are not yet
>> showing up in ES to see if the issue is on the sending side or on the
>> receiving side.
>>
>> David Lang
>>
>> On Tue, 24 May 2016, Alec Swan wrote:
>>
>> Date: Tue, 24 May 2016 22:17:22 -0600
>>> From: Alec Swan <[email protected]>
>>> Reply-To: rsyslog-users <[email protected]>
>>> To: rsyslog-users <[email protected]>
>>> Subject: [rsyslog] Logs are delayed being pushed to Elasticsearch
>>>
>>>
>>> Hello,
>>>
>>> I recently upgraded to rsyslog 8.18 and was happy to see that on disk
>>> queues no longer cause rsyslog to get in a bad state. However, now I see
>>> very long delays (several hours) of logs being pushed to Elasticsearch.
>>> It
>>> seems that somehow the logs are being buffered on the client for several
>>> hours because eventually they do show up in Elasticsearch. I don't see
>>> any
>>> errors in /var/log/rsyslog/ES-error.log (see config below) or
>>> /var/log/messages.
>>>
>>> I enabled impstats but didn't see any errors related to omelasticsearch.
>>> What else can I do to troubleshoot this?
>>>
>>> Here is my omelasticsearch config:
>>>
>>> action(
>>> type = "omelasticsearch"
>>> template = "es-payload"
>>> dynSearchIndex = "on"
>>> searchIndex = "logstash-index"
>>> searchType = "syslog"
>>> server = "127.0.0.1"
>>> serverport = "9200"
>>> uid = "xxx"
>>> pwd = "yyy"
>>> errorFile = "/var/log/rsyslog/ES-error.log"
>>> bulkmode = "on"
>>> action.resumeretrycount="-1" # retry if ES is unreachable (-1
>>> for
>>> infinite retries)
>>> action.resumeInterval="60"
>>> queue.dequeuebatchsize="1000" # ES bulk size
>>> queue.type="linkedlist"
>>> queue.size="100000"
>>> queue.workerthreads="5"
>>> queue.timeoutworkerthreadshutdown="2000"
>>> queue.spoolDirectory="/var/spool/rsyslog"
>>> queue.filename="omelasticsearch-queue"
>>> queue.maxfilesize="100m"
>>> queue.maxdiskspace="1g"
>>> queue.highwatermark="80000" # when to start spilling to disk
>>> queue.lowwatermark="20000" # when to stop spilling to disk
>>> queue.saveonshutdown="on"
>>> )
>>>
>>> Thanks,
>>>
>>> Alec
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
