I think I saw the problem in the config as exposed by the debug log. If I saw correctly, you use
$ActionResumeRetryCount -1 Some filter ~ Some filter :ommysql: The retry count is of type auto-reset. So the unlimited retries apply to the first action (the discard) and the sql writer uses the default value, which is to give up fairly quickly. It is important that you group these statements right in front of the actual action you want them to apply to -- actually, they *are* part of that action. I know its cumbersome, and things have and will continue to improve greatly in v6. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Steve Chupack > Sent: Monday, September 12, 2011 4:39 PM > To: [email protected] > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > Thanks for taking the time to respond, I really appreciate it. > > OK, I think I can better describe this now. You are correct, rsyslog is not > seeing a need to queue. My debug log is filled with the following entries, > which clearly indicate it sees no need to queue or spool. So again, I think I am > missing something very basic here... > > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries > > I used the very basic config from your article on buffering. > > $MainMsgQueueSize 500 > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > $MainMsgQueueFileName mainq # set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > Mysql is definitely not running, as show by the following: > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) > rsyslogd: db error (2002): Can't connect to local MySQL server through socket > '/var/lib/mysql/mysql.sock' (2) > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg > db error (2002): Can't connect to local MySQL server through socket > '/var/lib/mysql/mysql.sock' (2) > 8220.441040000:41537940: Called action, logging to ommysql > 8220.573450000:41537940: Called action, logging to ommysql > 8220.574801000:41537940: Called action, logging to ommysql > 8220.602406000:41537940: Called action, logging to ommysql > 8220.604330000:41537940: Called action, logging to ommysql > > > > On Mon, 12 Sep 2011 15:18:32 +0200 > Rainer Gerhards <[email protected]> wrote: > > > > > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Steve Chupack > > > Sent: Monday, September 12, 2011 3:02 PM > > > To: [email protected] > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had > > > mysql turned off for 20 minutes... And my message flow rate is up > > > around 1 per second. So I would have expected some queuing to > > > happen. Is there anything else I should look at or try? > > > > I think the unusual extremely low queue sizes results in some > > watermarks set to 0 and inhibiting DA queue startup. Set the main > > queue size to, say, 500 or 1000 and try again. Note that you can use > > logger or the tcpflood tool to injects lots of message in a short period. > > > > Rainer > > > > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > > Rainer Gerhards <[email protected]> wrote: > > > > > > > Queue files are only created if necessary. The debug log does not > > > contain any > > > > such situation. > > > > > > > > Rainer > > > > > > > > > -----Original Message----- > > > > > From: [email protected] [mailto:rsyslog- > > > > > [email protected]] On Behalf Of Steve Chupack > > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > > To: [email protected] > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > Ooops... previous message's attachment was the config, not the > > > debug > > > > > output. Debug attached. > > > > > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack > > > > > <[email protected]> wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > > loganalyzer > > > > > as a front end. Very cool. > > > > > > > > > > > > However, I can't seem to get on-disk queuing working, which > > > > > > would > > > be > > > > > nice if mysql goes down or a table is locked because I'm purging > > > > > records during a nightly maintenance cron. > > > > > > > > > > > > I never see any queue files created. I dropped the size of the > > > memory > > > > > queue to something ridiculously small, but still no queue files > > > even > > > > > after 20 minutes. > > > > > > > > > > > > I followed the instructions given here: > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > > with a > > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > > Details > > > > > below... And thanks to anyone who might be able to point out > > > > > what > > > I'm > > > > > doing wrong. > > > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules > > > > > > ;- > > > ) > > > > > > > > > > > > # if you experience problems, check # > > > > > > http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > > > # rsyslog v3: load input modules # If you do not load inputs, > > > > > > nothing happens! > > > > > > # You may need to set the module load path if modules are not > > > found. > > > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > $ModLoad imuxsock # provides support for local system logging > > > (e.g. > > > > > via logger command) > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > > $ModLoad ommysql > > > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > > unavailable > > > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > > (spool) > > > > > files > > > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > > mode > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert > > > > > > failure > > > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > > > then ~ > > > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > > values > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > > %iut%, > > > > > '%syslogtag%')",SQL > > > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > > & ~ > > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches > > > > > > (Message, > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > '%HOSTNAME%', > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > 'core01' > > > > > or $hostname contains_i 'core02' then > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > > #if ($fromhost-ip contains '10.128.0' or > > > > > > $source=='vt1hs1-dc216- > > > > > core01' or $source=='vt1hs1-dc216-core02') then > > > > > /var/log/switches > > > > > > & ~ > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > 'core01' > > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > > Facility, > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > > & ~ > > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > '%HOSTNAME%', > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > '10.128.24.44' > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > or > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > '10.128.24.84') > > > and > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > > 'warn' or > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' > > > or > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > > 'panic') > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > > & ~ > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > '10.128.24.44' > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > > > or > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > '10.128.24.84') > > > and > > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > > > # Log all kernel messages to the console. > > > > > > # Logging much else clutters up the screen. > > > > > > #kern.* > > > /dev/console > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > > # Don't log private authentication messages! > > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > > /var/log/messages > > > > > > > > > > > > # The authpriv file has restricted access. > > > > > > authpriv.* > > > > > /var/log/secure > > > > > > > > > > > > # Log all the mail messages in one place. > > > > > > mail.* - > > > > > /var/log/maillog > > > > > > > > > > > > > > > > > > # Log cron stuff > > > > > > cron.* - > > > > > /var/log/cron > > > > > > > > > > > > # Everybody gets emergency messages > > > > > > *.emerg * > > > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > > uucp,news.crit - > > > > > /var/log/spooler > > > > > > > > > > > > # Save boot messages also to boot.log > > > > > > local7.* > > > > > /var/log/boot.log > > > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An > > > > > > on-disk queue is created for this action. If the remote host > > > is > > > > > > # down, messages are spooled to disk and sent when it is up > > > again. > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > > files > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > > possible) > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > > shutdown > > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > > optional > > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > # > > > > > > TCP Syslog Server: > > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > > support > > > > > it) > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # > > > > > > start up TCP listener at port 514 > > > > > > > > > > > > # UDP Syslog Server: > > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > > 514 > > > > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > > V: 877.327.8422 x 1242 > > > > > > [email protected] | www.dealer.com > > > > > > > > > > > > > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
