João, Two quick ideas:
1) Enable disk-assisted queueing on the first rsyslog's output action 2) Make sure you have pstats enabled on both rsyslog instances (and that those pstats logs go somewhere you can find) then you can review the pstats to see what the message queue sizes are (on both rsyslogs) and whether you are filling your buffers, discarding logs, etc. -- Dave Caplinger | NTT Security > On Mar 7, 2019, at 9:21 AM, João Pereira <[email protected]> wrote: > > Hi all, > > We are facing an issue with rsyslog and we cannot find what is happening > behind. > > We're using rsyslog to receive logs from one of our providers, the problem > is that the provider stops sending logs (during aprox 10m) when it detects > the receiver is down meaning that every time we restart rsyslog server we > loose logs for ~10m. > > As we cannot control what the provider does, we came up with the idea of > having two rsyslog services on our machines. The first would only receive > the logs sent by our provider and forward them to the other rsyslog > service, the latests being responsible for parsing the logs and send it to > elasticsearch. This would allow us to change the configuration on the > second service (which are changes mostly on parsing rules) without having > to restart the forwarding service that contacts with our provider. > > That way we would be able to fool our provider because the forwarding > service would always be available, this sounded good on paper but when we > put it in production we realised that when we restart the second service > the first hangs (stops working for a while) and the failure is detected by > our provider which stops sending logs. > > Is there any way to improve this setup ? Can we make the forwarding service > to not hang ? Why rsyslog has this behaviour ? > > Thanks in advance > > -- > > João Pereira > > <https://www.marfeel.com> > > <https://www.marfeel.com/> > [image: Inline images 4] > <https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-google-certified-publishing-partner> > [image: Inline images 3] > <https://atenea.marfeel.com/atn/marfeel-business/what-it-means-to-be-a-facebook-instant-articles-partner> > > > Avda. Josep Tarradellas 20-30, 6th Floor > > 08029 Barcelona, Spain > > ES: (34) 93 178 59 50 > <%2834%29%2093%20178%2059%2050%20%C2%A0ext.%20107> > US: (1) 917-341-2540 <%281%29%20917-341-2540%20ext.%20107> > <facebook2.png><google-3.png>_______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. Confidentiality Notice: The content of this communication, along with any attachments, is covered by federal and state law governing electronic communications and may contain confidential and legally privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of the information contained herein is strictly prohibited. If you have received this communication in error, please immediately contact us by telephone at 402.361.3000 or e-mail [email protected]. Copyright 2000-2018 NTT Security (US) Inc., a wholly-owned subsidiary of NTT Group. All rights reserved. NTT Security is a trademark of NTT Security GMBH. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
