Ahh, I did not understand that.
I seem to remember reading a thread where this was not happening. It may
be that it was fixed in a version much newer than the 5.8 that you are running.
If you are sure that all the logs have been sent, you should be able to just rm
all the old spool files to get you operational again.
David Lang
On Thu, 21 Mar 2013, EXT-Edge, Sean wrote:
David,
Perhaps I wasn't clear in my initial email. I know what inodes are and I know
where the majority of them are in use, my rsyslog spool directory. There are
891185 files (only 971040 total inodes available for that FS) in my rsyslog
spool directory. I only included this information in case it might be useful
in troubleshooting why rsyslog is not clearing these files now that my remote
logging server is back online.
-Sean
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Thursday, March 21, 2013 3:33 PM
To: rsyslog-users
Subject: Re: [rsyslog] Having an issue with spool file cleanup after remote
syslog server comes back online.
Lots of Inodes in use means lots of files.
There is a possibility that files have been deleted, but some process still has
them open (preventing them from being really deleted), but a restart of rsyslog
means that it doesn't have them open any longer.
You just need to look at your filesystem and see where you have so many files.
ls |while read dir; echo -n "$dir "; find $dir |wc -l ; done
then cd to the directory with the most files in it and repeat until you rind
what's useing so many inodes.
David Lang
On Thu, 21 Mar 2013, EXT-Edge, Sean wrote:
Date: Thu, 21 Mar 2013 06:52:35 -0700
From: "EXT-Edge, Sean" <sean.e...@boeing.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
Subject: [rsyslog] Having an issue with spool file cleanup after remote syslog
server comes back online.
The spool files for my DA syslog forwarder queue have consumed all the inodes
on my system and aren't being cleaned up by rsyslog:
[root@devhost ~]# df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/myvg-rootvol
971040 971040 0 100% /
[root@devhost ~]# cd /var/spool/rsyslog_spool [root@devhost
rsyslog_spool]# ls -l | wc -l
891185
Here's my queue index file:
[root@devhost rsyslog_spool]# cat aqfwd.qi
<OPB:1:qqueue:1:
+iQueueSize:2:2:34:
+tVars.disk.sizeOnDisk:2:5:22629:
+tVars.disk.bytesRead:2:3:499:
End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:64:
+pszFName:1:5:aqfwd:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:0:
+sType:2:1:1:
+tOperationsMode:2:1:2:
+tOpenMode:2:3:384:
+iCurrOffs:2:1:0:
End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:27:
+pszFName:1:5:aqfwd:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:1:
+sType:2:1:1:
+tOperationsMode:2:1:1:
+tOpenMode:2:3:384:
+iCurrOffs:2:3:499:
End
.
[root@devhost rsyslog_spool]#
Rsyslog has open file descriptors to these files:
[root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep rsyslog_spool
rsyslogd 1086 root 4w REG 253,0 541 522935
/var/spool/rsyslog_spool/aqfwd.00000064
rsyslogd 1086 root 5r REG 253,0 499 529874
/var/spool/rsyslog_spool/aqfwd.00000027
[root@devhost rsyslog_spool]#
[root@devhost rsyslog_spool]# ls -l
/var/spool/rsyslog_spool/aqfwd.00000064
/var/spool/rsyslog_spool/aqfwd.00000027
-rw-------. 1 root root 499 Feb 22 19:12
/var/spool/rsyslog_spool/aqfwd.00000027
-rw-------. 1 root root 541 Mar 11 17:26
/var/spool/rsyslog_spool/aqfwd.00000064
[root@devhost rsyslog_spool]#
Here's the action queue config:
# Forward messages to central log server.
$ActionQueueType LinkedList
$ActionQueueFileName aqfwd
$ActionQueueSize 150000
$ActionQueueLowWaterMark 30000 # 20% of QueueSize
$ActionQueueHighWaterMark 120000 # 80% of QueueSize
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7 # Dont discard messages.
$ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want to
lose messages :)
$ActionResumeRetryCount -1 # Keep trying
*.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line that
actually does forwarding
I thought maybe clearing up a few inodes (and a service restart) might give
rsyslog enough breathing room to start cleaning up these files but that hasn't
proven to be the case.
This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64.
If there's any additional information needed please let me know. I still relatively new
to rsyslog and I didn't write the config above so I can't answer any "why"
questions about it.
Thanks in advance for any help!
-Sean
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
