sorry, i incorrectly spelled your name the first time around on this. ;-) On Mon, 12 Sep 2011 16:06:27 -0400 Steve Chupack <[email protected]> wrote:
> Rainer, > > Thank you! I just tested and spool files are created and grow when mysql is > off, then disappear when mysql comes back. The debug log also confirmed this > behavior. > > Again, your help is hugely appreciated. > > Steve > > > > On Mon, 12 Sep 2011 19:00:28 +0200 > Rainer Gerhards <[email protected]> wrote: > > > I think I saw the problem in the config as exposed by the debug log. If I > > saw > > correctly, you use > > > > $ActionResumeRetryCount -1 > > Some filter ~ > > Some filter :ommysql: > > > > The retry count is of type auto-reset. So the unlimited retries apply to the > > first action (the discard) and the sql writer uses the default value, which > > is to give up fairly quickly. It is important that you group these > > statements > > right in front of the actual action you want them to apply to -- actually, > > they *are* part of that action. I know its cumbersome, and things have and > > will continue to improve greatly in v6. > > > > Rainer > > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Steve Chupack > > > Sent: Monday, September 12, 2011 4:39 PM > > > To: [email protected] > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > Thanks for taking the time to respond, I really appreciate it. > > > > > > OK, I think I can better describe this now. You are correct, rsyslog is > > > not > > > seeing a need to queue. My debug log is filled with the following entries, > > > which clearly indicate it sees no need to queue or spool. So again, I > > > think > > I am > > > missing something very basic here... > > > > > > > > > 7878.780426000:455e1940: main Q: entry added, size now 1 entries > > > 7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy > > > 7878.780566000:455e1940: main Q: EnqueueMsg advised worker start > > > 7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 > > > entries > > > > > > I used the very basic config from your article on buffering. > > > > > > $MainMsgQueueSize 500 > > > $WorkDirectory /var/log/rsyslogq # default location for work (spool) files > > > $MainMsgQueueFileName mainq # set file name, also enables disk mode > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > > > > Mysql is definitely not running, as show by the following: > > > > > > 8220.440850000:41537940: Called LogError, msg: db error (2002): Can't > > > connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' > > (2) > > > rsyslogd: db error (2002): Can't connect to local MySQL server through > > socket > > > '/var/lib/mysql/mysql.sock' (2) > > > 8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg > > > db error (2002): Can't connect to local MySQL server through socket > > > '/var/lib/mysql/mysql.sock' (2) > > > 8220.441040000:41537940: Called action, logging to ommysql > > > 8220.573450000:41537940: Called action, logging to ommysql > > > 8220.574801000:41537940: Called action, logging to ommysql > > > 8220.602406000:41537940: Called action, logging to ommysql > > > 8220.604330000:41537940: Called action, logging to ommysql > > > > > > > > > > > > On Mon, 12 Sep 2011 15:18:32 +0200 > > > Rainer Gerhards <[email protected]> wrote: > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: [email protected] [mailto:rsyslog- > > > > > [email protected]] On Behalf Of Steve Chupack > > > > > Sent: Monday, September 12, 2011 3:02 PM > > > > > To: [email protected] > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had > > > > > mysql turned off for 20 minutes... And my message flow rate is up > > > > > around 1 per second. So I would have expected some queuing to > > > > > happen. Is there anything else I should look at or try? > > > > > > > > I think the unusual extremely low queue sizes results in some > > > > watermarks set to 0 and inhibiting DA queue startup. Set the main > > > > queue size to, say, 500 or 1000 and try again. Note that you can use > > > > logger or the tcpflood tool to injects lots of message in a short > > > > period. > > > > > > > > Rainer > > > > > > > > > > > > > > On Mon, 12 Sep 2011 11:13:28 +0200 > > > > > Rainer Gerhards <[email protected]> wrote: > > > > > > > > > > > Queue files are only created if necessary. The debug log does not > > > > > contain any > > > > > > such situation. > > > > > > > > > > > > Rainer > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: [email protected] [mailto:rsyslog- > > > > > > > [email protected]] On Behalf Of Steve Chupack > > > > > > > Sent: Sunday, September 11, 2011 11:11 PM > > > > > > > To: [email protected] > > > > > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > > > > > > > > > Ooops... previous message's attachment was the config, not the > > > > > debug > > > > > > > output. Debug attached. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 Steve Chupack > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I have rsyslogd up and running and logging to mysql with > > > > > loganalyzer > > > > > > > as a front end. Very cool. > > > > > > > > > > > > > > > > However, I can't seem to get on-disk queuing working, which > > > > > > > > would > > > > > be > > > > > > > nice if mysql goes down or a table is locked because I'm purging > > > > > > > records during a nightly maintenance cron. > > > > > > > > > > > > > > > > I never see any queue files created. I dropped the size of the > > > > > memory > > > > > > > queue to something ridiculously small, but still no queue files > > > > > even > > > > > > > after 20 minutes. > > > > > > > > > > > > > > > > I followed the instructions given here: > > > > > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > > > > > with a > > > > > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > > > > > > > > > I'm quite sure I'm just completely missing something obvious. > > > > > Details > > > > > > > below... And thanks to anyone who might be able to point out > > > > > > > what > > > > > I'm > > > > > > > doing wrong. > > > > > > > > > > > > > > > > -Steve > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules > > > > > > > > ;- > > > > > ) > > > > > > > > > > > > > > > > # if you experience problems, check # > > > > > > > > http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > > > > > > > > > # rsyslog v3: load input modules # If you do not load inputs, > > > > > > > > nothing happens! > > > > > > > > # You may need to set the module load path if modules are not > > > > > found. > > > > > > > > > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > > > > > $ModLoad imuxsock # provides support for local system logging > > > > > (e.g. > > > > > > > via logger command) > > > > > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > > > > > $ModLoad ommysql > > > > > > > > > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > > > > > unavailable > > > > > > > > > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > > > > > (spool) > > > > > > > files > > > > > > > > > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > > > > > $ActionQueueFileName dbq # set file name, also enables disk > > > > > mode > > > > > > > > $ActionResumeRetryCount -1 # infinite retries on insert > > > > > > > > failure > > > > > > > > > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains > > > > > > > > 'snmpd' > > > > > > > then ~ > > > > > > > > > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > > > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > > > > > values > > > > > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > > > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > > > > > %iut%, > > > > > > > '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > > > > > & ~ > > > > > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches > > > > > > > > (Message, > > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > > '%HOSTNAME%', > > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > > 'core01' > > > > > > > or $hostname contains_i 'core02' then > > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > > > > > #if ($fromhost-ip contains '10.128.0' or > > > > > > > > $source=='vt1hs1-dc216- > > > > > > > core01' or $source=='vt1hs1-dc216-core02') then > > > > > > > /var/log/switches > > > > > > > > & ~ > > > > > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > > > > > 'core01' > > > > > > > or $hostname contains_i 'core02' then ~ > > > > > > > > > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > > > > > Facility, > > > > > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > > > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > > > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > > > > > & ~ > > > > > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > > > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > > > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > > > > > '%HOSTNAME%', > > > > > > > %syslogpriority%, '%timereported:::date-mysql%', > > > > > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > > '10.128.24.44' > > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == > > > > > > > '10.128.24.52' > > > > > or > > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > > '10.128.24.84') > > > > > and > > > > > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > > > > > 'warn' or > > > > > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' > > > > > > > or $syslogseverity-text == 'crit' or $syslogseverity-text == > > 'alert' > > > > > or > > > > > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > > > > > 'panic') > > > > > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > > > > > & ~ > > > > > > > > > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > > > > > '10.128.24.44' > > > > > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == > > > > > > > '10.128.24.52' > > > > > or > > > > > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' > > > > > > > or $fromhost-ip == '10.128.24.80' or $fromhost-ip == > > > > > > > '10.128.24.84') > > > > > and > > > > > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > > > > > > > > > # Log all kernel messages to the console. > > > > > > > > # Logging much else clutters up the screen. > > > > > > > > #kern.* > > > > > /dev/console > > > > > > > > > > > > > > > > # Log anything (except mail) of level info or higher. > > > > > > > > # Don't log private authentication messages! > > > > > > > > *.info;mail.none;authpriv.none;cron.none - > > > > > > > /var/log/messages > > > > > > > > > > > > > > > > # The authpriv file has restricted access. > > > > > > > > authpriv.* > > > > > > > /var/log/secure > > > > > > > > > > > > > > > > # Log all the mail messages in one place. > > > > > > > > mail.* - > > > > > > > /var/log/maillog > > > > > > > > > > > > > > > > > > > > > > > > # Log cron stuff > > > > > > > > cron.* - > > > > > > > /var/log/cron > > > > > > > > > > > > > > > > # Everybody gets emergency messages > > > > > > > > *.emerg * > > > > > > > > > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > > > > > uucp,news.crit - > > > > > > > /var/log/spooler > > > > > > > > > > > > > > > > # Save boot messages also to boot.log > > > > > > > > local7.* > > > > > > > /var/log/boot.log > > > > > > > > > > > > > > > > # Remote Logging (we use TCP for reliable delivery) # An > > > > > > > > on-disk queue is created for this action. If the remote host > > > > > is > > > > > > > > # down, messages are spooled to disk and sent when it is up > > > > > again. > > > > > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > > > > > files > > > > > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > > > > > possible) > > > > > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > > > > > shutdown > > > > > > > > #$ActionQueueType LinkedList # run asynchronously > > > > > > > > #$ActionResumeRetryCount -1 # infinite retries if host is > > > > > > > > down > > > > > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > > > > > optional > > > > > > > > #*.* @@remote-host:514 > > > > > > > > > > > > > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > # > > > > > > > > TCP Syslog Server: > > > > > > > > # provides TCP syslog reception and GSS-API (if compiled to > > > > > support > > > > > > > it) > > > > > > > > $ModLoad imtcp.so # load module $InputTCPServerRun 1470 # > > > > > > > > start up TCP listener at port 514 > > > > > > > > > > > > > > > > # UDP Syslog Server: > > > > > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > > > > > 514 > > > > > > > > > > > > > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > > > > > V: 877.327.8422 x 1242 > > > > > > > > [email protected] | www.dealer.com > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > rsyslog mailing list > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > > http://www.rsyslog.com > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
