Thanks for taking the time to respond, I really appreciate it.
OK, I think I can better describe this now. You are correct, rsyslog is not
seeing a need to queue. My debug log is filled with the following entries,
which clearly indicate it sees no need to queue or spool. So again, I think I
am missing something very basic here...
7878.780426000:455e1940: main Q: entry added, size now 1 entries
7878.780439000:455e1940: wtpAdviseMaxWorkers signals busy
7878.780566000:455e1940: main Q: EnqueueMsg advised worker start
7878.780585000:423dc940: main Q: entry deleted, state 0, size now 0 entries
I used the very basic config from your article on buffering.
$MainMsgQueueSize 500
$WorkDirectory /var/log/rsyslogq # default location for work (spool) files
$MainMsgQueueFileName mainq # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
Mysql is definitely not running, as show by the following:
8220.440850000:41537940: Called LogError, msg: db error (2002): Can't connect
to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
rsyslogd: db error (2002): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)
8220.440865000:41537940: logmsg: flags 1, from 'vt1hs1-netservices01', msg db
error (2002): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)
8220.441040000:41537940: Called action, logging to ommysql
8220.573450000:41537940: Called action, logging to ommysql
8220.574801000:41537940: Called action, logging to ommysql
8220.602406000:41537940: Called action, logging to ommysql
8220.604330000:41537940: Called action, logging to ommysql
On Mon, 12 Sep 2011 15:18:32 +0200
Rainer Gerhards <[email protected]> wrote:
>
>
> > -----Original Message-----
> > From: [email protected] [mailto:rsyslog-
> > [email protected]] On Behalf Of Steve Chupack
> > Sent: Monday, September 12, 2011 3:02 PM
> > To: [email protected]
> > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> >
> > Hmmmm... the thing is, my $MainMsgQueueSize is only 5, and I had mysql
> > turned off for 20 minutes... And my message flow rate is up around 1
> > per second. So I would have expected some queuing to happen. Is there
> > anything else I should look at or try?
>
> I think the unusual extremely low queue sizes results in some watermarks set
> to 0 and inhibiting DA queue startup. Set the main queue size to, say, 500
> or 1000 and try again. Note that you can use logger or the tcpflood tool to
> injects lots of message in a short period.
>
> Rainer
>
> >
> > On Mon, 12 Sep 2011 11:13:28 +0200
> > Rainer Gerhards <[email protected]> wrote:
> >
> > > Queue files are only created if necessary. The debug log does not
> > contain any
> > > such situation.
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: [email protected] [mailto:rsyslog-
> > > > [email protected]] On Behalf Of Steve Chupack
> > > > Sent: Sunday, September 11, 2011 11:11 PM
> > > > To: [email protected]
> > > > Subject: Re: [rsyslog] Unable to implement on-disk queuing
> > > >
> > > > Ooops... previous message's attachment was the config, not the
> > debug
> > > > output. Debug attached.
> > > >
> > > >
> > > >
> > > > On Sun, 11 Sep 2011 16:50:57 -0400
> > > > Steve Chupack <[email protected]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have rsyslogd up and running and logging to mysql with
> > loganalyzer
> > > > as a front end. Very cool.
> > > > >
> > > > > However, I can't seem to get on-disk queuing working, which would
> > be
> > > > nice if mysql goes down or a table is locked because I'm purging
> > > > records during a nightly maintenance cron.
> > > > >
> > > > > I never see any queue files created. I dropped the size of the
> > memory
> > > > queue to something ridiculously small, but still no queue files
> > even
> > > > after 20 minutes.
> > > > >
> > > > > I followed the instructions given here:
> > > > http://www.rsyslog.com/doc/rsyslog_high_database_rate.html along
> > with a
> > > > couple of other tutorials on rsyslogd and queuing.
> > > > >
> > > > > I'm quite sure I'm just completely missing something obvious.
> > Details
> > > > below... And thanks to anyone who might be able to point out what
> > I'm
> > > > doing wrong.
> > > > >
> > > > > -Steve
> > > > >
> > > > >
> > > > >
> > > > > MY CONFIG, please, no flames re the horrendously complex rules ;-
> > )
> > > > >
> > > > > # if you experience problems, check
> > > > > # http://www.rsyslog.com/troubleshoot for assistance
> > > > >
> > > > > # rsyslog v3: load input modules
> > > > > # If you do not load inputs, nothing happens!
> > > > > # You may need to set the module load path if modules are not
> > found.
> > > > >
> > > > > $ModLoad immark # provides --MARK-- message capability
> > > > > $ModLoad imuxsock # provides support for local system logging
> > (e.g.
> > > > via logger command)
> > > > > $ModLoad imklog # kernel logging (formerly provided by rklogd)
> > > > > $ModLoad ommysql
> > > > >
> > > > > # SC 2010.11.11: configure disk caching in case mysql is
> > unavailable
> > > > >
> > > > > $MainMsgQueueSize 5
> > > > >
> > > > > $WorkDirectory /var/log/rsyslogq # default location for work
> > (spool)
> > > > files
> > > > >
> > > > > $ActionQueueType LinkedList # use asynchronous processing
> > > > > $ActionQueueFileName dbq # set file name, also enables disk
> > mode
> > > > > $ActionResumeRetryCount -1 # infinite retries on insert failure
> > > > >
> > > > > # send snmpd INFO messages to the dustbin
> > > > >
> > > > > if $syslogseverity-text == 'info' and $syslogtag contains 'snmpd'
> > > > then ~
> > > > >
> > > > > $template vtfw,"insert into vtfw (Message, Facility, FromHost,
> > > > Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)
> > values
> > > > ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
> > > > '%timereported:::date-mysql%', '%timegenerated:::date-mysql%',
> > %iut%,
> > > > '%syslogtag%')",SQL
> > > > >
> > > > > if ($fromhost-ip contains '10.128.255') then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vtfw
> > > > > & ~
> > > > > #if ($fromhost-ip contains '10.128.255') then ~
> > > > >
> > > > > $template vt1hs1_switches,"insert into vt1hs1_switches (Message,
> > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > 'core01'
> > > > or $hostname contains_i 'core02' then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_switches
> > > > > #if ($fromhost-ip contains '10.128.0' or $source=='vt1hs1-dc216-
> > > > core01' or $source=='vt1hs1-dc216-core02') then /var/log/switches
> > > > > & ~
> > > > > #if $fromhost-ip contains '10.128.0' or $hostname contains_i
> > 'core01'
> > > > or $hostname contains_i 'core02' then ~
> > > > >
> > > > > $template vt1hs1_wifi,"insert into vt1hs1_wifi (Message,
> > Facility,
> > > > FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID,
> > > > SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if $fromhost-ip contains '10.128.244' then
> > > > :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_wifi
> > > > > & ~
> > > > > #if $fromhost-ip contains '10.128.244' then ~
> > > > >
> > > > > $template vt1hs1_vsphere,"insert into vt1hs1_vsphere (Message,
> > > > Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
> > > > InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%,
> > '%HOSTNAME%',
> > > > %syslogpriority%, '%timereported:::date-mysql%',
> > > > '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL
> > > > >
> > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > '10.128.24.44'
> > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > or
> > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> > and
> > > > ($syslogseverity-text == 'warning' or $syslogseverity-text ==
> > 'warn' or
> > > > $syslogseverity-text == 'err' or $syslogseverity-text == 'error' or
> > > > $syslogseverity-text == 'crit' or $syslogseverity-text == 'alert'
> > or
> > > > $syslogseverity-text == 'emergency' or $syslogseverity-text ==
> > 'panic')
> > > > then :ommysql:localhost,Syslog,root,mysqldb44;vt1hs1_vsphere
> > > > > & ~
> > > > >
> > > > > if ($fromhost-ip == '10.128.24.40' or $fromhost-ip ==
> > '10.128.24.44'
> > > > or $fromhost-ip == '10.128.24.48' or $fromhost-ip == '10.128.24.52'
> > or
> > > > $fromhost-ip == '10.128.24.72' or $fromhost-ip == '10.128.24.76' or
> > > > $fromhost-ip == '10.128.24.80' or $fromhost-ip == '10.128.24.84')
> > and
> > > > ($syslogseverity-text == 'notice') then ~
> > > > >
> > > > >
> > > > > #*.* :ommysql:localhost,Syslog,root,mysqldb44
> > > > > *.notice :ommysql:localhost,Syslog,root,mysqldb44
> > > > >
> > > > > # Log all kernel messages to the console.
> > > > > # Logging much else clutters up the screen.
> > > > > #kern.*
> > /dev/console
> > > > >
> > > > > # Log anything (except mail) of level info or higher.
> > > > > # Don't log private authentication messages!
> > > > > *.info;mail.none;authpriv.none;cron.none -
> > > > /var/log/messages
> > > > >
> > > > > # The authpriv file has restricted access.
> > > > > authpriv.*
> > > > /var/log/secure
> > > > >
> > > > > # Log all the mail messages in one place.
> > > > > mail.* -
> > > > /var/log/maillog
> > > > >
> > > > >
> > > > > # Log cron stuff
> > > > > cron.* -
> > > > /var/log/cron
> > > > >
> > > > > # Everybody gets emergency messages
> > > > > *.emerg *
> > > > >
> > > > > # Save news errors of level crit and higher in a special file.
> > > > > uucp,news.crit -
> > > > /var/log/spooler
> > > > >
> > > > > # Save boot messages also to boot.log
> > > > > local7.*
> > > > /var/log/boot.log
> > > > >
> > > > > # Remote Logging (we use TCP for reliable delivery)
> > > > > # An on-disk queue is created for this action. If the remote host
> > is
> > > > > # down, messages are spooled to disk and sent when it is up
> > again.
> > > > > #$WorkDirectory /rsyslog/spool # where to place spool files
> > > > > #$ActionQueueFileName uniqName # unique name prefix for spool
> > files
> > > > > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as
> > > > possible)
> > > > > #$ActionQueueSaveOnShutdown on # save messages to disk on
> > shutdown
> > > > > #$ActionQueueType LinkedList # run asynchronously
> > > > > #$ActionResumeRetryCount -1 # infinite retries if host is down
> > > > > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port
> > optional
> > > > > #*.* @@remote-host:514
> > > > >
> > > > >
> > > > > # ######### Receiving Messages from Remote Hosts ##########
> > > > > # TCP Syslog Server:
> > > > > # provides TCP syslog reception and GSS-API (if compiled to
> > support
> > > > it)
> > > > > $ModLoad imtcp.so # load module
> > > > > $InputTCPServerRun 1470 # start up TCP listener at port 514
> > > > >
> > > > > # UDP Syslog Server:
> > > > > $ModLoad imudp.so # provides UDP syslog reception
> > > > > $UDPServerRun 514 # start a UDP syslog server at standard port
> > 514
> > > > >
> > > > >
> > > > > SOME DEBUG OUTPUT:
> > > > >
> > > > > (see attached)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Steve Chupack | IT Systems Administrator
> > > > > V: 877.327.8422 x 1242
> > > > > [email protected] | www.dealer.com
> > > > >
> > > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
