Hi,
Lately we stared noticing issues with out Rsyslog receiver which, about
once a week, starts using a lot of CPU and memory. At the same time, we
also see some logs duplicated 1000+ times.
We are also using Rsyslog in our infrastructure to forward logs from all
servers to Elasticsearch. When this thing happens, it happens on multiple
systems at the same time and, because of this, we could not easily debug
things. The number 1 priority was to make things work.
We hoped that some setting would help, but they didn't:
queue.dequeueslowdown="5000000"
action.resumeRetryCount="3"
action.resumeInterval="60"
While doing some unrelated things, I managed to reproduce this quite
easily, as described in https://github.com/rsyslog/rsyslog/issues/500.
Debug log and sample conf are also attached there.
The main reason, seems to be that actionTryCommit doesn't handle well
unknown return codes from output module transactions and this can put the
action in an (almost) infinite loop.
The problem is even worse. At each passing through the loop the log entry
is duplicated and, if at some point the output module succeeds, you get the
same log entries 1000+ times.
Depending on how output modules are written, this could affect any output
module that uses transactions.
The issue text from github has more info and a possible solution, but would
like some feedback.
Thanks,
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
