The spool files for my DA syslog forwarder queue have consumed all the inodes
on my system and aren't being cleaned up by rsyslog:
[root@devhost ~]# df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/myvg-rootvol
971040 971040 0 100% /
[root@devhost ~]# cd /var/spool/rsyslog_spool
[root@devhost rsyslog_spool]# ls -l | wc -l
891185
Here's my queue index file:
[root@devhost rsyslog_spool]# cat aqfwd.qi
<OPB:1:qqueue:1:
+iQueueSize:2:2:34:
+tVars.disk.sizeOnDisk:2:5:22629:
+tVars.disk.bytesRead:2:3:499:
>End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:64:
+pszFName:1:5:aqfwd:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:0:
+sType:2:1:1:
+tOperationsMode:2:1:2:
+tOpenMode:2:3:384:
+iCurrOffs:2:1:0:
>End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:27:
+pszFName:1:5:aqfwd:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:1:
+sType:2:1:1:
+tOperationsMode:2:1:1:
+tOpenMode:2:3:384:
+iCurrOffs:2:3:499:
>End
.
[root@devhost rsyslog_spool]#
Rsyslog has open file descriptors to these files:
[root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep rsyslog_spool
rsyslogd 1086 root 4w REG 253,0 541 522935
/var/spool/rsyslog_spool/aqfwd.00000064
rsyslogd 1086 root 5r REG 253,0 499 529874
/var/spool/rsyslog_spool/aqfwd.00000027
[root@devhost rsyslog_spool]#
[root@devhost rsyslog_spool]# ls -l /var/spool/rsyslog_spool/aqfwd.00000064
/var/spool/rsyslog_spool/aqfwd.00000027
-rw-------. 1 root root 499 Feb 22 19:12 /var/spool/rsyslog_spool/aqfwd.00000027
-rw-------. 1 root root 541 Mar 11 17:26 /var/spool/rsyslog_spool/aqfwd.00000064
[root@devhost rsyslog_spool]#
Here's the action queue config:
# Forward messages to central log server.
$ActionQueueType LinkedList
$ActionQueueFileName aqfwd
$ActionQueueSize 150000
$ActionQueueLowWaterMark 30000 # 20% of QueueSize
$ActionQueueHighWaterMark 120000 # 80% of QueueSize
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7 # Dont discard messages.
$ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want to
lose messages :)
$ActionResumeRetryCount -1 # Keep trying
*.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line that actually
does forwarding
I thought maybe clearing up a few inodes (and a service restart) might give
rsyslog enough breathing room to start cleaning up these files but that hasn't
proven to be the case.
This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64.
If there's any additional information needed please let me know. I still
relatively new to rsyslog and I didn't write the config above so I can't answer
any "why" questions about it.
Thanks in advance for any help!
-Sean
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
