Well, as David said: if you configure rsyslog to not discard any messages, you need to think about the consequences. These are obviously if no discard is permitted, the output destination cannot be reached, and the available buffer space has been used up, then no more input messages can be accepted, what usually results in a slowdown of the senders and ultimately in a system hang. This is nothing to blame rsyslog about: there are actually some folks that need this type of processing (due to compliance rules). You can easily avoid the problem by configuring rsyslog to discard messages when it needs to do so. Or, as said here, you can write to disk (but keep in mind that disk is also a limited resource).
For additional information, I would also suggest to have a look at section 4.5 of this paper: http://download.rsyslog.com/design.pdf Note: the paper is still unfinished, but it contains a lot of useful information. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Olivier Fauchon > Sent: Monday, March 07, 2011 10:24 AM > To: rsyslog-users > Subject: Re: [rsyslog] Apache/Rsyslog stop working after several hours > > Hi ... > > I just found the solution: > > When Rsyslog on webserver could not communicate with my central rsyslog > server with RELP protocol, all the logs passed to /dev/log are spooled in > memory, until link is back. > > When memory limit is reached (10000 Entry), rsyslog throttles the input > (/dev/log was really slow). > This is the reason Apache goes mad and blocks (it used /dev/log, in my case) > > The workaround is to allow rsyslog to spool its mainqueue on disk . > Then, if rsyslog can't forward logs to rsyslog central server, it spools all the > input logs on disk. > > When connectivity to central server is back, it process all the data. > > Here is the important part of the configuration: > > $MainMsgQueueFileName mainqueue # <<<<<<< allow to spool on disk if > required > $MainMsgQueueType LinkedList # dynamic size of buffer > $MainMsgQueueSaveOnShutdown on > > > Tell me if that helped ! > > Olivier > > > ----- Mail original ----- > De: "Todd Michael Bushnell" <[email protected]> > À: "rsyslog-users" <[email protected]> > Envoyé: Lundi 7 Mars 2011 07:43:32 > Objet: [rsyslog] Apache/Rsyslog stop working after several hours > > Been planning an rsyslog deployment for about a month. Everything > performed as expected in my limited use dev environment, but when I > deployed rsyslog today to my production environment multiple systems > yielded similar disastrous results: > > After a few hours Apache jumped up to 250+ processes (max=256, > normal=~50) and then started hanging. At this time, rsyslog also stopped > logging altogether. As soon as I killed rsyslog and started sysklog, httpd > processes dropped to 50 and everything went back to normal. > > I'm not sure if this is a case where rsyslog froze and it's state resulted in > Apache's inability to close processes or if there is a problem with Apache and > Rsyslog when a decent volume of traffic is passed through. I'm happy to > provide additional information if someone could give me some clues as to > where to start looking. At this point we're reverting until I can diagnose this > issue and assure my team that I've fixed the problem for good. > > Version: rsyslog-3.22.1-3.el5_5.1 > System: Linux ******* 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST > 2008 x86_64 x86_64 x86_64 GNU/Linux > > > > Todd Michael Bushnell > [email protected] > > > > _______________________________________________ rsyslog mailing > list http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > > -- > Olivier Fauchon > Digitick > Administrateur système et réseau > Tel: 06.85.53.26.41 > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
