Ok. I have it logging as you suggested and I also have it running with ‘-d -n’. Should I expect for rsyslogd debugging output to stop at the point where I see this behavior or will the debugging output continue on with no discernible change?
On Nov 6, 2013, at 12:11 PM, David Lang <[email protected]> wrote: > What is probably happening here is that you have a log message that cannot be > written out, and that is causing the output to stop. > > The fact that you have to delete the queue files matches this. > > looking at your ruleset, my guess is that you are getting a hostname arriving > that creates an invalid filename in some way, and the OS is refusing to allow > rsyslog to write the file. > > I would suggest creating a new logfile that logs fromhost-ip and hostname > (nothing else), put this at the beginning of the config and when a stall > happens look at the last entry in that file. If you can get a debug output > when it's stuck, it probably will tell you what it's trying to do at that > point. > > what version are you running? > > David Lang > > On Wed, 6 Nov 2013 18:01:57 +0000, Leggett, Torrance I. wrote: >> I have a central logging server that accepts messages and writes them >> out - <year>/<month>/<host>. However, recently I’ve had a problem >> where the server stops writing out virtually all such messages and the >> main message queue fills and starts writing to the disk asssisted >> queue. Once it starts doing this, the DA queue only grows until it >> hits the max size or the disk fills. Restarting doesn’t seem to help >> unless I remove all the files in the DA spool directory. Below is the >> server portion of the config. If you need more configs or debugging, >> just let me know what. I’m relatively new to debugging rsyslog issues. >> >> # cat 99-server.conf >> # Switch to server ruleset >> $RuleSet server >> >> $MainMsgQueueFileName mainqueue # unique name prefix for spool files >> $MainMsgQueueType LinkedList # main queue should be a >> dynamic list in memory >> $MainMsgQueueSize 100000 # increase the queue size to >> handle the message traffic >> $MainMsgQueueHighWatermark 80000 # increase the high water >> mark to write messages to disk >> $MainMsgQueueLowWatermark 20000 # increase the low water mark >> to stop writing to disk >> $MainMsgQueueMaxDiskSpace 1g # 1gb disk space limit >> $MainMsgQueueSaveOnShutdown off # save messages to disk on shutdown >> $MainMsgQueueWorkerThreads 5 # spawn up to 5 threads for >> queue processing >> $MaxMessageSize 8k # handle larger messages if needed >> $RepeatedMsgReduction off # log all messages as they come >> >> # Load UDP module >> $ModLoad imudp >> $InputUDPServerBindRuleset server >> $UDPServerRun 514 >> >> # Load TCP module >> $ModLoad imtcp >> $InputTCPServerBindRuleset server >> $InputTCPServerRun 514 >> >> # Load RELP module >> $ModLoad imrelp >> $InputRELPServerBindRuleset server >> $InputRELPServerRun 20514 >> >> # Send logs to logstash for indexing >> *.* @@127.0.0.1:5544;RSYSLOG_TraditionalForwardFormat >> >> # Templates >> $Template auditFormat,"%MSG%\n" >> $Template radiusFormat,"%MSG%\n" >> $Template tsmFormat,"%MSG%\n" >> $Template >> dynAuditLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/audit.log" >> $Template dynAuthLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/secure" >> $Template dynCronLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/cron.log" >> $Template >> dynDaemonLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/daemon.log" >> $Template dynDebug,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/debug" >> $Template >> dynHttpAccess,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_access.log" >> $Template >> dynHttpError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_error.log" >> $Template dynKernLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/kern.log" >> $Template dynMailLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/mail.log" >> $Template >> dynPuppetAgent,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-agent.log" >> $Template >> dynPuppetMaster,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-master.log" >> $Template >> dynRadiusLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/radius.log" >> $Template dynSyslog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/messages" >> $Template >> dynTsmInfo,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmcmd.log" >> $Template >> dynTsmError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmerror.log" >> $Template dynUserLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/user.log" >> >> # First capture auditd messages from remotes >> # >> if $programname == 'auditd' and $syslogfacility-text == 'local6' then >> ?dynAuditLog;auditFormat >> >> # Next capture RADIUS messages from remotes >> # >> if $programname == 'radiusd' and $syslogfacility-text == 'local6' >> then ?dynRadiusLog;radiusFormat >> >> # Next handle any apache logs and remove them from the stream >> # >> if $programname == 'httpd' and $syslogfacility-text == 'local6' then { >> ?dynHttpAccess >> stop >> } >> if $programname == 'httpd' and $syslogfacility-text == 'local7' then { >> ?dynHttpError >> stop >> } >> >> # Next handle any nginx logs and remove them from the stream >> # >> if $programname == 'nginx' and $syslogfacility-text == 'local6' then { >> ?dynHttpAccess >> stop >> } >> if $programname == 'nginx' and $syslogfacility-text == 'local7' then { >> ?dynHttpError >> stop >> } >> >> # Next handle any puppet logs and remove them from the stream >> # >> if $programname == 'puppet-agent' then { >> ?dynPuppetAgent >> stop >> } >> if $programname == 'puppet-master' then { >> ?dynPuppetMaster >> stop >> } >> >> # Next handle any TSM logs and remove them from the stream >> # >> if $programname == 'dsmc' and $syslogfacility-text == 'local3' and >> $syslogseverity-text == 'info' then ?dynTsmInfo;tsmFormat >> if $programname == 'dsmserv' and $syslogfacility-text == 'local3' >> and $syslogseverity-text == 'err' then ?dynTsmError;tsmFormat >> >> # Rules >> auth,authpriv.* ?dynAuthLog >> *.*;\ >> mail.none;\ >> cron.none -?dynSyslog >> cron.* ?dynCronLog >> daemon.* -?dynDaemonLog >> kern.* -?dynKernLog >> mail.* -?dynMailLog >> user.* -?dynUserLog >> >> # Switch back to default ruleset >> $RuleSet RSYSLOG_DefaultRuleset > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT.
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
