David, Perhaps I wasn't clear in my initial email. I know what inodes are and I know where the majority of them are in use, my rsyslog spool directory. There are 891185 files (only 971040 total inodes available for that FS) in my rsyslog spool directory. I only included this information in case it might be useful in troubleshooting why rsyslog is not clearing these files now that my remote logging server is back online.
-Sean -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Thursday, March 21, 2013 3:33 PM To: rsyslog-users Subject: Re: [rsyslog] Having an issue with spool file cleanup after remote syslog server comes back online. Lots of Inodes in use means lots of files. There is a possibility that files have been deleted, but some process still has them open (preventing them from being really deleted), but a restart of rsyslog means that it doesn't have them open any longer. You just need to look at your filesystem and see where you have so many files. ls |while read dir; echo -n "$dir "; find $dir |wc -l ; done then cd to the directory with the most files in it and repeat until you rind what's useing so many inodes. David Lang On Thu, 21 Mar 2013, EXT-Edge, Sean wrote: > Date: Thu, 21 Mar 2013 06:52:35 -0700 > From: "EXT-Edge, Sean" <[email protected]> > Reply-To: rsyslog-users <[email protected]> > To: "[email protected]" <[email protected]> > Subject: [rsyslog] Having an issue with spool file cleanup after remote syslog > server comes back online. > > The spool files for my DA syslog forwarder queue have consumed all the inodes > on my system and aren't being cleaned up by rsyslog: > > [root@devhost ~]# df -i . > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/mapper/myvg-rootvol > 971040 971040 0 100% / > [root@devhost ~]# cd /var/spool/rsyslog_spool [root@devhost > rsyslog_spool]# ls -l | wc -l > 891185 > > Here's my queue index file: > > [root@devhost rsyslog_spool]# cat aqfwd.qi > <OPB:1:qqueue:1: > +iQueueSize:2:2:34: > +tVars.disk.sizeOnDisk:2:5:22629: > +tVars.disk.bytesRead:2:3:499: >> End > . > <Obj:1:strm:1: > +iCurrFNum:2:2:64: > +pszFName:1:5:aqfwd: > +iMaxFiles:2:8:10000000: > +bDeleteOnClose:2:1:0: > +sType:2:1:1: > +tOperationsMode:2:1:2: > +tOpenMode:2:3:384: > +iCurrOffs:2:1:0: >> End > . > <Obj:1:strm:1: > +iCurrFNum:2:2:27: > +pszFName:1:5:aqfwd: > +iMaxFiles:2:8:10000000: > +bDeleteOnClose:2:1:1: > +sType:2:1:1: > +tOperationsMode:2:1:1: > +tOpenMode:2:3:384: > +iCurrOffs:2:3:499: >> End > . > [root@devhost rsyslog_spool]# > > Rsyslog has open file descriptors to these files: > > [root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep rsyslog_spool > rsyslogd 1086 root 4w REG 253,0 541 522935 > /var/spool/rsyslog_spool/aqfwd.00000064 > rsyslogd 1086 root 5r REG 253,0 499 529874 > /var/spool/rsyslog_spool/aqfwd.00000027 > [root@devhost rsyslog_spool]# > > [root@devhost rsyslog_spool]# ls -l > /var/spool/rsyslog_spool/aqfwd.00000064 > /var/spool/rsyslog_spool/aqfwd.00000027 > -rw-------. 1 root root 499 Feb 22 19:12 > /var/spool/rsyslog_spool/aqfwd.00000027 > -rw-------. 1 root root 541 Mar 11 17:26 > /var/spool/rsyslog_spool/aqfwd.00000064 > [root@devhost rsyslog_spool]# > > > Here's the action queue config: > > # Forward messages to central log server. > $ActionQueueType LinkedList > $ActionQueueFileName aqfwd > $ActionQueueSize 150000 > $ActionQueueLowWaterMark 30000 # 20% of QueueSize > $ActionQueueHighWaterMark 120000 # 80% of QueueSize > $ActionQueueSaveOnShutdown on > $ActionQueueDiscardSeverity 7 # Dont discard messages. > $ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want to > lose messages :) > $ActionResumeRetryCount -1 # Keep trying > *.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line that > actually does forwarding > > > I thought maybe clearing up a few inodes (and a service restart) might give > rsyslog enough breathing room to start cleaning up these files but that > hasn't proven to be the case. > > This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64. > > If there's any additional information needed please let me know. I still > relatively new to rsyslog and I didn't write the config above so I can't > answer any "why" questions about it. > > Thanks in advance for any help! > > -Sean > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
