> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Steve Chupack > Sent: Monday, September 12, 2011 3:02 PM > To: [email protected] > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql > turned off for 20 minutes... And my message flow rate is up around 1 > per second. So I would have expected some queuing to happen. Is there > anything else I should look at or try?
I think the unusual extremely low queue sizes results in some watermarks set to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500 or 1000 and try again. Note that you can use logger or the tcpflood tool to injects lots of message in a short period. Rainer > > On Mon, 12 Sep 2011 11:13:28 +0200 > Rainer Gerhards <[email protected]> wrote: > > > Queue files are only created if necessary. The debug log does not > contain any > > such situation. > > > > Rainer > > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Steve Chupack > > > Sent: Sunday, September 11, 2011 11:11 PM > > > To: [email protected] > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing > > > > > > Ooops... previous message's attachment was the config, not the > debug > > > output. Debug attached. > > > > > > > > > > > > On Sun, 11 Sep 2011 16:50:57 -0400 > > > Steve Chupack <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > I have rsyslogd up and running and logging to mysql with > loganalyzer > > > as a front end. Very cool. > > > > > > > > However, I can't seem to get on-disk queuing working, which would > be > > > nice if mysql goes down or a table is locked because I'm purging > > > records during a nightly maintenance cron. > > > > > > > > I never see any queue files created. I dropped the size of the > memory > > > queue to something ridiculously small, but still no queue files > even > > > after 20 minutes. > > > > > > > > I followed the instructions given here: > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along > with a > > > couple of other tutorials on rsyslogd and queuing. > > > > > > > > I'm quite sure I'm just completely missing something obvious. > Details > > > below... And thanks to anyone who might be able to point out what > I'm > > > doing wrong. > > > > > > > > -Steve > > > > > > > > > > > > > > > > MY CONFIG, please, no flames re the horrendously complex rules ;- > ) > > > > > > > > # if you experience problems, check > > > > # http://www.rsyslog.com/troubleshoot for assistance > > > > > > > > # rsyslog v3: load input modules > > > > # If you do not load inputs, nothing happens! > > > > # You may need to set the module load path if modules are not > found. > > > > > > > > $ModLoad immark # provides --MARK-- message capability > > > > $ModLoad imuxsock # provides support for local system logging > (e.g. > > > via logger command) > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd) > > > > $ModLoad ommysql > > > > > > > > # SC 2010.11.11: configure disk caching in case mysql is > unavailable > > > > > > > > $MainMsgQueueSize 5 > > > > > > > > $WorkDirectory /var/log/rsyslogq # default location for work > (spool) > > > files > > > > > > > > $ActionQueueType LinkedList # use asynchronous processing > > > > $ActionQueueFileName dbq # set file name, also enables disk > mode > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure > > > > > > > > # send snmpd INFO messages to the dustbin > > > > > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd' > > > then ~ > > > > > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost, > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) > values > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', > %iut%, > > > '%syslogtag%')",SQL > > > > > > > > if ($fromhost-ip contains '10.128.255') then > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw > > > > & ~ > > > > #if ($fromhost-ip contains '10.128.255') then ~ > > > > > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message, > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > '%HOSTNAME%', > > > %syslogpriority%, '%timereported:::date-mysql%', > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i > 'core01' > > > or $hostname contains_i 'core02' then > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216- > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches > > > > & ~ > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i > 'core01' > > > or $hostname contains_i 'core02' then ~ > > > > > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message, > Facility, > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', > > > %syslogpriority%, '%timereported:::date-mysql%', > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > if $fromhost-ip contains '10.128.244' then > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi > > > > & ~ > > > > #if $fromhost-ip contains '10.128.244' then ~ > > > > > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message, > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, > '%HOSTNAME%', > > > %syslogpriority%, '%timereported:::date-mysql%', > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > '10.128.24.44' > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > or > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > and > > > ($syslogseverity-text == 'warning' or $syslogseverity-text == > 'warn' or > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert' > or > > > $syslogseverity-text == 'emergency' or $syslogseverity-text == > 'panic') > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere > > > > & ~ > > > > > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip == > '10.128.24.44' > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52' > or > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84') > and > > > ($syslogseverity-text == 'notice') then ~ > > > > > > > > > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44 > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44 > > > > > > > > # Log all kernel messages to the console. > > > > # Logging much else clutters up the screen. > > > > #kern.* > /dev/console > > > > > > > > # Log anything (except mail) of level info or higher. > > > > # Don't log private authentication messages! > > > > *.info;mail.none;authpriv.none;cron.none - > > > /var/log/messages > > > > > > > > # The authpriv file has restricted access. > > > > authpriv.* > > > /var/log/secure > > > > > > > > # Log all the mail messages in one place. > > > > mail.* - > > > /var/log/maillog > > > > > > > > > > > > # Log cron stuff > > > > cron.* - > > > /var/log/cron > > > > > > > > # Everybody gets emergency messages > > > > *.emerg * > > > > > > > > # Save news errors of level crit and higher in a special file. > > > > uucp,news.crit - > > > /var/log/spooler > > > > > > > > # Save boot messages also to boot.log > > > > local7.* > > > /var/log/boot.log > > > > > > > > # Remote Logging (we use TCP for reliable delivery) > > > > # An on-disk queue is created for this action. If the remote host > is > > > > # down, messages are spooled to disk and sent when it is up > again. > > > > #$WorkDirectory /rsyslog/spool # where to place spool files > > > > #$ActionQueueFileName uniqName # unique name prefix for spool > files > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as > > > possible) > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on > shutdown > > > > #$ActionQueueType LinkedList # run asynchronously > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port > optional > > > > #*.* @@remote-host:514 > > > > > > > > > > > > # ######### Receiving Messages from Remote Hosts ########## > > > > # TCP Syslog Server: > > > > # provides TCP syslog reception and GSS-API (if compiled to > support > > > it) > > > > $ModLoad imtcp.so # load module > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514 > > > > > > > > # UDP Syslog Server: > > > > $ModLoad imudp.so # provides UDP syslog reception > > > > $UDPServerRun 514 # start a UDP syslog server at standard port > 514 > > > > > > > > > > > > SOME DEBUG OUTPUT: > > > > > > > > (see attached) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Steve Chupack | IT Systems Administrator > > > > V: 877.327.8422 x 1242 > > > > [email protected] | www.dealer.com > > > > > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
