I don't recall seeing a previous response from you, Rainer. I saw the thread with the Perl tool you are talking about, but I thought that was used to rebuild a QI file. I have a QI file, which I pasted in the original email. The files it lists exist, have log data, and are currently held open by rsyslogd.
-Sean -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Friday, March 22, 2013 9:31 AM To: rsyslog-users Subject: Re: [rsyslog] Having an issue with spool file cleanup after remote syslog server comes back online. > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of EXT-Edge, Sean > Sent: Friday, March 22, 2013 2:20 PM > To: rsyslog-users > Subject: Re: [rsyslog] Having an issue with spool file cleanup after > remote syslog server comes back online. > > I would appreciate more concrete information on if my rsyslog version > contains a known issue, as you suggest. I searched for a couple hours > Wednesday, found a some similar issues, but ultimately didn't feel the > issues I found were identical. Well, the ChangeLog would probably be a really good place to look at ;) > > The logs have NOT been sent to my remote logging server which is > (probably) the reason the spool directory has not been cleaned. Newer > logs are being forwarded correctly to the logging server but they are > largely complaints about being out of space. :) As I wrote, the queue files will never drain, because the .qi file does not list them as active. There is the user-contributed recover_qi.pl file in the tools directory which most probably can fix the situation. HTH Rainer > > Fixing it in the quickest way possible is not my goal. If it were I > would have removed all the spool files already. It's just a > development machine. I want to make sure this doesn't happen to me in > production where deleting log entries would be unacceptable. > > -Sean > > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of David Lang > Sent: Thursday, March 21, 2013 3:51 PM > To: rsyslog-users > Subject: Re: [rsyslog] Having an issue with spool file cleanup after > remote syslog server comes back online. > > Ahh, I did not understand that. > > I seem to remember reading a thread where this was not happening. It > may be that it was fixed in a version much newer than the 5.8 that you > are running. > > If you are sure that all the logs have been sent, you should be able > to just rm all the old spool files to get you operational again. > > David Lang > > On Thu, 21 Mar 2013, EXT-Edge, Sean wrote: > > > David, > > Perhaps I wasn't clear in my initial email. I know what inodes are > > and I > know where the majority of them are in use, my rsyslog spool directory. > There are 891185 files (only 971040 total inodes available for that > FS) in my rsyslog spool directory. I only included this information > in case it might be useful in troubleshooting why rsyslog is not > clearing these files now that my remote logging server is back online. > > > > -Sean > > > > -----Original Message----- > > From: [email protected] > > [mailto:[email protected]] On Behalf Of David Lang > > Sent: Thursday, March 21, 2013 3:33 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Having an issue with spool file cleanup after > > remote > syslog server comes back online. > > > > Lots of Inodes in use means lots of files. > > > > There is a possibility that files have been deleted, but some > > process still has > them open (preventing them from being really deleted), but a restart > of rsyslog means that it doesn't have them open any longer. > > > > You just need to look at your filesystem and see where you have so > > many > files. > > > > ls |while read dir; echo -n "$dir "; find $dir |wc -l ; done > > > > then cd to the directory with the most files in it and repeat until > > you rind > what's useing so many inodes. > > > > David Lang > > > > > > On Thu, 21 Mar 2013, EXT-Edge, Sean wrote: > > > >> Date: Thu, 21 Mar 2013 06:52:35 -0700 > >> From: "EXT-Edge, Sean" <[email protected]> > >> Reply-To: rsyslog-users <[email protected]> > >> To: "[email protected]" <[email protected]> > >> Subject: [rsyslog] Having an issue with spool file cleanup after > >> remote > syslog > >> server comes back online. > >> > >> The spool files for my DA syslog forwarder queue have consumed all > >> the > inodes on my system and aren't being cleaned up by rsyslog: > >> > >> [root@devhost ~]# df -i . > >> Filesystem Inodes IUsed IFree IUse% Mounted on > >> /dev/mapper/myvg-rootvol > >> 971040 971040 0 100% / > >> [root@devhost ~]# cd /var/spool/rsyslog_spool [root@devhost > >> rsyslog_spool]# ls -l | wc -l > >> 891185 > >> > >> Here's my queue index file: > >> > >> [root@devhost rsyslog_spool]# cat aqfwd.qi > >> <OPB:1:qqueue:1: > >> +iQueueSize:2:2:34: > >> +tVars.disk.sizeOnDisk:2:5:22629: > >> +tVars.disk.bytesRead:2:3:499: > >>> End > >> . > >> <Obj:1:strm:1: > >> +iCurrFNum:2:2:64: > >> +pszFName:1:5:aqfwd: > >> +iMaxFiles:2:8:10000000: > >> +bDeleteOnClose:2:1:0: > >> +sType:2:1:1: > >> +tOperationsMode:2:1:2: > >> +tOpenMode:2:3:384: > >> +iCurrOffs:2:1:0: > >>> End > >> . > >> <Obj:1:strm:1: > >> +iCurrFNum:2:2:27: > >> +pszFName:1:5:aqfwd: > >> +iMaxFiles:2:8:10000000: > >> +bDeleteOnClose:2:1:1: > >> +sType:2:1:1: > >> +tOperationsMode:2:1:1: > >> +tOpenMode:2:3:384: > >> +iCurrOffs:2:3:499: > >>> End > >> . > >> [root@devhost rsyslog_spool]# > >> > >> Rsyslog has open file descriptors to these files: > >> > >> [root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep > rsyslog_spool > >> rsyslogd 1086 root 4w REG 253,0 541 522935 > /var/spool/rsyslog_spool/aqfwd.00000064 > >> rsyslogd 1086 root 5r REG 253,0 499 529874 > /var/spool/rsyslog_spool/aqfwd.00000027 > >> [root@devhost rsyslog_spool]# > >> > >> [root@devhost rsyslog_spool]# ls -l > >> /var/spool/rsyslog_spool/aqfwd.00000064 > >> /var/spool/rsyslog_spool/aqfwd.00000027 > >> -rw-------. 1 root root 499 Feb 22 19:12 > >> /var/spool/rsyslog_spool/aqfwd.00000027 > >> -rw-------. 1 root root 541 Mar 11 17:26 > >> /var/spool/rsyslog_spool/aqfwd.00000064 > >> [root@devhost rsyslog_spool]# > >> > >> > >> Here's the action queue config: > >> > >> # Forward messages to central log server. > >> $ActionQueueType LinkedList > >> $ActionQueueFileName aqfwd > >> $ActionQueueSize 150000 > >> $ActionQueueLowWaterMark 30000 # 20% of QueueSize > >> $ActionQueueHighWaterMark 120000 # 80% of QueueSize > >> $ActionQueueSaveOnShutdown on > >> $ActionQueueDiscardSeverity 7 # Dont discard messages. > >> $ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want > >> to > lose messages :) > >> $ActionResumeRetryCount -1 # Keep trying > >> *.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line > >> that actually does forwarding > >> > >> > >> I thought maybe clearing up a few inodes (and a service restart) > >> might > give rsyslog enough breathing room to start cleaning up these files > but that hasn't proven to be the case. > >> > >> This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64. > >> > >> If there's any additional information needed please let me know. I > >> still > relatively new to rsyslog and I didn't write the config above so I > can't answer any "why" questions about it. > >> > >> Thanks in advance for any help! > >> > >> -Sean > >> > >> > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
