Hi Rsyslog users,
We have been periodically experiencing an issue with our rsyslog setup
where some RELP relay nodes appear to fill up their queue and stop
processing any messages.
Our log flow essentially is made up of a number of "clients" that send
messages over RELP to one or more "relay" layers which finally send to a
number of rsyslog processes which index messages in elasticsearch, for
example:
client -> relay -> relay (x2) -> indexer (x2) -> elasticsearch
Each relay sends to at least 2 rsyslog servers balancing messages
between them using config like:
if $$uptime % 2 == 0 then {
RELP A
}
else if $$uptime % 1 == 0 then {
RELP B
}
During peak times we are pushing about 25000 messages per second on each
of the most busy relays and indexers (limited by the indexing
operation). The "relays" do not write queue to disk.
The problem has always been that one or more "relays" simply stops
forwarding, inspection of the process shows memory usage higher than
others as the queue is full. Normally, restarting the rsyslog process
clears the queue and resumes normal processing.
This looks like a bug, and perhaps gets triggered by some badly formed
or encoded incoming message or something (noting this is also a largely
Japanese environment), but I was curious if anyone here has experienced
similar or knows where to look or any suggestions how to get useful
information to report about this.
I appreciate any help you can give, thanks,
-Arik
--
*This correspondence (including any attachments) is for the intended
recipient(s) only. It may contain confidential or privileged information or
both. No confidentiality or privilege is waived or lost by any
mis-transmission. If you receive this correspondence by mistake, please
contact the sender immediately, delete this correspondence (and all
attachments) and destroy any hard copies. You must not use, disclose, copy,
distribute or rely on any part of this correspondence (including any
attachments) if you are not the intended
recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。*
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
