Hi Rsyslog users,

We have been periodically experiencing an issue with our rsyslog setup
where some RELP relay nodes appear to fill up their queue and stop
processing any messages.

Our log flow essentially is made up of a number of "clients" that send
messages over RELP to one or more "relay" layers which finally send to a
number of rsyslog processes which index messages in elasticsearch, for

client -> relay -> relay (x2) -> indexer (x2) -> elasticsearch

Each relay sends to at least 2 rsyslog servers balancing messages
between them using config like:

if $$uptime % 2 == 0 then {
else if $$uptime % 1 == 0 then {

During peak times we are pushing about 25000 messages per second on each
of the most busy relays and indexers (limited by the indexing
operation). The "relays" do not write queue to disk.

The problem has always been that one or more "relays" simply stops
forwarding, inspection of the process shows memory usage higher than
others as the queue is full. Normally, restarting the rsyslog process
clears the queue and resumes normal processing.

This looks like a bug, and perhaps gets triggered by some badly formed
or encoded incoming message or something (noting this is also a largely
Japanese environment), but I was curious if anyone here has experienced
similar or knows where to look or any suggestions how to get useful
information to report about this.

I appreciate any help you can give, thanks,


*This correspondence (including any attachments) is for the intended 
recipient(s) only. It may contain confidential or privileged information or 
both. No confidentiality or privilege is waived or lost by any 
mis-transmission. If you receive this correspondence by mistake, please 
contact the sender immediately, delete this correspondence (and all 
attachments) and destroy any hard copies. You must not use, disclose, copy, 
distribute or rely on any part of this correspondence (including any 
attachments) if you are not the intended 
rsyslog mailing list
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 

Reply via email to