I would appreciate more concrete information on if my rsyslog version contains a known issue, as you suggest. I searched for a couple hours Wednesday, found a some similar issues, but ultimately didn't feel the issues I found were identical.
The logs have NOT been sent to my remote logging server which is (probably) the reason the spool directory has not been cleaned. Newer logs are being forwarded correctly to the logging server but they are largely complaints about being out of space. :) Fixing it in the quickest way possible is not my goal. If it were I would have removed all the spool files already. It's just a development machine. I want to make sure this doesn't happen to me in production where deleting log entries would be unacceptable. -Sean -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Thursday, March 21, 2013 3:51 PM To: rsyslog-users Subject: Re: [rsyslog] Having an issue with spool file cleanup after remote syslog server comes back online. Ahh, I did not understand that. I seem to remember reading a thread where this was not happening. It may be that it was fixed in a version much newer than the 5.8 that you are running. If you are sure that all the logs have been sent, you should be able to just rm all the old spool files to get you operational again. David Lang On Thu, 21 Mar 2013, EXT-Edge, Sean wrote: > David, > Perhaps I wasn't clear in my initial email. I know what inodes are and I > know where the majority of them are in use, my rsyslog spool directory. > There are 891185 files (only 971040 total inodes available for that FS) in my > rsyslog spool directory. I only included this information in case it might > be useful in troubleshooting why rsyslog is not clearing these files now that > my remote logging server is back online. > > -Sean > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of David Lang > Sent: Thursday, March 21, 2013 3:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] Having an issue with spool file cleanup after remote > syslog server comes back online. > > Lots of Inodes in use means lots of files. > > There is a possibility that files have been deleted, but some process still > has them open (preventing them from being really deleted), but a restart of > rsyslog means that it doesn't have them open any longer. > > You just need to look at your filesystem and see where you have so many files. > > ls |while read dir; echo -n "$dir "; find $dir |wc -l ; done > > then cd to the directory with the most files in it and repeat until you rind > what's useing so many inodes. > > David Lang > > > On Thu, 21 Mar 2013, EXT-Edge, Sean wrote: > >> Date: Thu, 21 Mar 2013 06:52:35 -0700 >> From: "EXT-Edge, Sean" <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: "[email protected]" <[email protected]> >> Subject: [rsyslog] Having an issue with spool file cleanup after remote >> syslog >> server comes back online. >> >> The spool files for my DA syslog forwarder queue have consumed all the >> inodes on my system and aren't being cleaned up by rsyslog: >> >> [root@devhost ~]# df -i . >> Filesystem Inodes IUsed IFree IUse% Mounted on >> /dev/mapper/myvg-rootvol >> 971040 971040 0 100% / >> [root@devhost ~]# cd /var/spool/rsyslog_spool [root@devhost >> rsyslog_spool]# ls -l | wc -l >> 891185 >> >> Here's my queue index file: >> >> [root@devhost rsyslog_spool]# cat aqfwd.qi >> <OPB:1:qqueue:1: >> +iQueueSize:2:2:34: >> +tVars.disk.sizeOnDisk:2:5:22629: >> +tVars.disk.bytesRead:2:3:499: >>> End >> . >> <Obj:1:strm:1: >> +iCurrFNum:2:2:64: >> +pszFName:1:5:aqfwd: >> +iMaxFiles:2:8:10000000: >> +bDeleteOnClose:2:1:0: >> +sType:2:1:1: >> +tOperationsMode:2:1:2: >> +tOpenMode:2:3:384: >> +iCurrOffs:2:1:0: >>> End >> . >> <Obj:1:strm:1: >> +iCurrFNum:2:2:27: >> +pszFName:1:5:aqfwd: >> +iMaxFiles:2:8:10000000: >> +bDeleteOnClose:2:1:1: >> +sType:2:1:1: >> +tOperationsMode:2:1:1: >> +tOpenMode:2:3:384: >> +iCurrOffs:2:3:499: >>> End >> . >> [root@devhost rsyslog_spool]# >> >> Rsyslog has open file descriptors to these files: >> >> [root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep rsyslog_spool >> rsyslogd 1086 root 4w REG 253,0 541 522935 >> /var/spool/rsyslog_spool/aqfwd.00000064 >> rsyslogd 1086 root 5r REG 253,0 499 529874 >> /var/spool/rsyslog_spool/aqfwd.00000027 >> [root@devhost rsyslog_spool]# >> >> [root@devhost rsyslog_spool]# ls -l >> /var/spool/rsyslog_spool/aqfwd.00000064 >> /var/spool/rsyslog_spool/aqfwd.00000027 >> -rw-------. 1 root root 499 Feb 22 19:12 >> /var/spool/rsyslog_spool/aqfwd.00000027 >> -rw-------. 1 root root 541 Mar 11 17:26 >> /var/spool/rsyslog_spool/aqfwd.00000064 >> [root@devhost rsyslog_spool]# >> >> >> Here's the action queue config: >> >> # Forward messages to central log server. >> $ActionQueueType LinkedList >> $ActionQueueFileName aqfwd >> $ActionQueueSize 150000 >> $ActionQueueLowWaterMark 30000 # 20% of QueueSize >> $ActionQueueHighWaterMark 120000 # 80% of QueueSize >> $ActionQueueSaveOnShutdown on >> $ActionQueueDiscardSeverity 7 # Dont discard messages. >> $ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want to >> lose messages :) >> $ActionResumeRetryCount -1 # Keep trying >> *.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line that >> actually does forwarding >> >> >> I thought maybe clearing up a few inodes (and a service restart) might give >> rsyslog enough breathing room to start cleaning up these files but that >> hasn't proven to be the case. >> >> This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64. >> >> If there's any additional information needed please let me know. I still >> relatively new to rsyslog and I didn't write the config above so I can't >> answer any "why" questions about it. >> >> Thanks in advance for any help! >> >> -Sean >> >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This > is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our > control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
