I am attempting to configure rsyslog to queue messages while the destination server is down but I am finding it is only queuing approx 1 message per second and we receive a whole lot more than that.
We have many hosts that send messages to a 'proxy' if you will at a location which then in turn forwards it to our central log server for archival. We are going to have an outage on the central log server soon so I am trying to replace syslog-ng with rsyslog so the intermediates can queue messages while the central server is down. I am using 3.22.1 packaged with RHEL 5 which I understand is old but I will not be able to use a newer version. I have followed the setup at http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html but am not getting the expected results. Here's my config. # Globals $MaxMessageSize 8k $MainMsgQueueType LinkedList # Use traditional timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # Provides kernel logging support (previously done by rklogd) $ModLoad imklog # Provides support for local system logging (e.g. via logger command) $ModLoad imuxsock $ModLoad imtcp.so $InputTCPMaxSessions 1024 $InputTCPServerRun 5142 $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueType LinkedList # run asynchronously $ActionQueueFileName remotqeque # unique name prefix for spool files $ActionResumeRetryCount -1 # infinite retries if host is down $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueMaxDiskSpace 2g # 1gb space limit (use as much as possible) $ActionQueueDequeueSlowdown 0 *.* @@redacted:51422 To get logs to rsyslog I have configured syslog-ng to deliver to both the central syslog server and to rsyslog on the intermediate. Rsyslog is sending to redacted:51422 which is not actually listening (since I don't really want to double my messages to the central server) which simulates the central server being offline. In practice I am seeing in the debug rsyslog process about one log entry per second which it is sending to my disk assisted queue. Syslog-ng however is receiving thousands of messages per minute. If I get rid of the action queue and just log to a file rsyslog keeps up with syslog-ng no problem. Here's a snippet from the debug log. 2402.830836385:main queue:Reg/w0: action 1 queue: enqueueMsg: cond timeout, dropping message! 2402.830862669:main queue:Reg/w0: wtpAdviseMaxWorkers signals busy 2402.830868074:main queue:Reg/w0: action 1 queue: EnqueueMsg advised worker start 2402.830879658:main queue:Reg/w0: main queue: entering rate limiter 2402.830887901:main queue:Reg/w0: main queue: entry deleted, state 0, size now 7385 entries 2402.830897655:main queue:Reg/w0: Called action, logging to builtin-fwd 2402.830903570:main queue:Reg/w0: action 1 queue: enqueueMsg: LightDelay mark reached for light delayable message - blocking a bit. 2403.059393630:imuxsock.c: Message from UNIX socket: #6 2403.059418695:imuxsock.c: logmsg: flags 4, from 'hostnameredacted', message content redacted 2403.059424142:imuxsock.c: Message has legacy syslog format. 2403.059434733:imuxsock.c: main queue: entry added, size now 7386 entries 2403.059441696:imuxsock.c: wtpAdviseMaxWorkers signals busy 2403.059446737:imuxsock.c: main queue: EnqueueMsg advised worker start 2403.059453751:imuxsock.c: --------imuxsock calling select, active file descriptors (max 6): 6 2403.087373110:imuxsock.c: Message from UNIX socket: #6 2403.087416859:imuxsock.c: logmsg: flags 4, from 'hostnameredacted', message content redacted 2403.087422226:imuxsock.c: Message has legacy syslog format. 2403.087432941:imuxsock.c: main queue: entry added, size now 7387 entries 2403.087439847:imuxsock.c: wtpAdviseMaxWorkers signals busy 2403.087444749:imuxsock.c: main queue: EnqueueMsg advised worker start 2403.087451626:imuxsock.c: --------imuxsock calling select, active file descriptors (max 6): 6 2403.088771548:imuxsock.c: Message from UNIX socket: #6 2403.088785375:imuxsock.c: logmsg: flags 4, from 'hostnameredacted', message content redacted 2403.088790500:imuxsock.c: Message has legacy syslog format. 2403.088797998:imuxsock.c: main queue: entry added, size now 7388 entries 2403.088804299:imuxsock.c: wtpAdviseMaxWorkers signals busy 2403.088809079:imuxsock.c: main queue: EnqueueMsg advised worker start 2403.088826047:imuxsock.c: --------imuxsock calling select, active file descriptors (max 6): 6 2403.113614170:imuxsock.c: Message from UNIX socket: #6 2403.113659621:imuxsock.c: logmsg: flags 4, from 'hostnameredacted', message content redacted 2403.113675071:imuxsock.c: main queue: entry added, size now 7389 entries 2403.113681913:imuxsock.c: wtpAdviseMaxWorkers signals busy 2403.113687091:imuxsock.c: main queue: EnqueueMsg advised worker start 2403.113693974:imuxsock.c: --------imuxsock calling select, active file descriptors (max 6): 6 2403.709635421:imtcp.c: main queue: entry added, size now 7390 entries 2403.709660157:imtcp.c: wtpAdviseMaxWorkers signals busy 2403.709665565:imtcp.c: main queue: EnqueueMsg advised worker start 2403.709683772:imtcp.c: logmsg: flags 0, from 'hostnameredacted', message content redacted 2403.709688841:imtcp.c: Message has legacy syslog format. 2403.709697186:imtcp.c: main queue: enqueueMsg: LightDelay mark reached for light delayable message - blocking a bit. 2403.832590271:main queue:Reg/w0: action 1 queue: enqueueMsg: queue FULL - waiting to drain. 2404.712385305:imtcp.c: main queue: entry added, size now 7391 entries 2404.712414709:imtcp.c: wtpAdviseMaxWorkers signals busy Lines containing "cond timeout, dropping message!" and "queue FULL - waiting to drain." have me concerned. But it doesn't look like all of my logs are making it to the main queue anyway. Is imtcp dropping messages and the debug level is not high enough to see it maybe? Please help! How do I get rsyslog to queue all the messages bound for the simulated down central log host? Regards, -Alan _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
