Hi ... I just found the solution:
When Rsyslog on webserver could not communicate with my central rsyslog server with RELP protocol, all the logs passed to /dev/log are spooled in memory, until link is back. When memory limit is reached (10000 Entry), rsyslog throttles the input (/dev/log was really slow). This is the reason Apache goes mad and blocks (it used /dev/log, in my case) The workaround is to allow rsyslog to spool its mainqueue on disk . Then, if rsyslog can't forward logs to rsyslog central server, it spools all the input logs on disk. When connectivity to central server is back, it process all the data. Here is the important part of the configuration: $MainMsgQueueFileName mainqueue # <<<<<<< allow to spool on disk if required $MainMsgQueueType LinkedList # dynamic size of buffer $MainMsgQueueSaveOnShutdown on Tell me if that helped ! Olivier ----- Mail original ----- De: "Todd Michael Bushnell" <[email protected]> À: "rsyslog-users" <[email protected]> Envoyé: Lundi 7 Mars 2011 07:43:32 Objet: [rsyslog] Apache/Rsyslog stop working after several hours Been planning an rsyslog deployment for about a month. Everything performed as expected in my limited use dev environment, but when I deployed rsyslog today to my production environment multiple systems yielded similar disastrous results: After a few hours Apache jumped up to 250+ processes (max=256, normal=~50) and then started hanging. At this time, rsyslog also stopped logging altogether. As soon as I killed rsyslog and started sysklog, httpd processes dropped to 50 and everything went back to normal. I'm not sure if this is a case where rsyslog froze and it's state resulted in Apache's inability to close processes or if there is a problem with Apache and Rsyslog when a decent volume of traffic is passed through. I'm happy to provide additional information if someone could give me some clues as to where to start looking. At this point we're reverting until I can diagnose this issue and assure my team that I've fixed the problem for good. Version: rsyslog-3.22.1-3.el5_5.1 System: Linux ******* 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Todd Michael Bushnell [email protected] _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com -- Olivier Fauchon Digitick Administrateur système et réseau Tel: 06.85.53.26.41 _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
