If you manually do a kill -HUP 11311 do these files get released?
David Lang
On Tue, 25 Jun 2013, Xuri Nagarin wrote:
More fun info:
$ sudo lsof -s | awk '$5 == "REG"' | sort -n -r -k 7,7 | head -n 50 | grep
deleted
rsyslogd 11331 root 20w REG 9,2 117175430149
8847368 /var/log/joe/CISCO/app7/logfile.1 (deleted)
rsyslogd 11331 root 9w REG 9,2 53533745904
8839187 /var/log/joe/Microsoft/app1/logfile.1 (deleted)
rsyslogd 11331 root 10w REG 9,2 16268001760
8839237 /var/log/joe/Microsoft/app2/logfile.1 (deleted)
rsyslogd 11331 root 18w REG 9,2 1831944964
8847369 /var/log/joe/Microsoft/app3/logfile.1 (deleted)
On Tue, Jun 25, 2013 at 12:10 PM, Xuri Nagarin <[email protected]> wrote:
Happening again:
$ sudo du -sh *
9.8M bin
26M boot
220K dev
6.9M etc
3.5G home
129M lib
22M lib64
20K lost+found
8.0K media
8.0K mnt
6.4G opt
du: cannot access `proc/2713/task/9576': No such file or directory
du: cannot access `proc/9490/task/9490/fd/4': No such file or directory
du: cannot access `proc/9490/task/9490/fdinfo/4': No such file or directory
du: cannot access `proc/9490/fd/4': No such file or directory
du: cannot access `proc/9490/fdinfo/4': No such file or directory
0 proc
116K root
13M sbin
4.0K selinux
4.0K srv
0 sys
296K tmp
1.3G usr
39G var
$ sudo df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/md2 913G 212G 655G 25% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/md0 97M 31M 62M 34% /boot
I am seeing the same baheviour on a second system.
On Tue, Jun 25, 2013 at 12:04 PM, Xuri Nagarin <[email protected]> wrote:
My suspicion was something to do with dynamic file creation but looking
at the code, both static and dynamic file creation use the same function of
file "open":
From tools/omfile.c
------------xxxxxxxxxxxxxxxxx---------------
fd = open((char*) newFileName,
O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC,
556 pData->fCreateMode);
------------xxxxxxxxxxxxxxxxx---------------
"O_APPEND" should take care of file being rotated while rsyslog is trying
to write to it.
On Tue, Jun 25, 2013 at 11:47 AM, Xuri Nagarin <[email protected]> wrote:
Not a typo or error, in RHEL the rsyslogd start up script in init.d has
an explicit variable "PIDFILE" set as:
PIDFILE=/var/run/syslogd.pid
:)
I did not write the script, using whatever was bundled in the RPM I
grabbed from Adiscon.
To confirm:
$ ps ax | grep rsyslog
11331 ? Sl 61:18 /sbin/rsyslogd -i /var/run/syslogd.pid -x
On Tue, Jun 25, 2013 at 11:41 AM, Soham Chakraborty <
[email protected]> wrote:
Hi,
In the logrotate config, change it to /var/run/rsyslog.pid and test. On
a
quick skim, it looks like a typo.
Soham
On Tue, Jun 25, 2013 at 11:07 PM, David Lang <[email protected]> wrote:
well, one thing that looks wrong is that logrotate is looking for
/var/run/syslog.pid, but on my systems the pid is in
/var/log/rsyslog.pid
are you sure that rsyslog is actually getting the HUP?
can you try sending it a HUP manually and see if it closes the files?
David Lang
On Tue, 25 Jun 2013, Xuri Nagarin wrote:
Date: Tue, 25 Jun 2013 11:34:08 -0700
From: Xuri Nagarin <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] HUP-ing rsyslog does not free up disk space
Hi David,
The master conf file is simple with few additions that are
self-explanatory. The other conf in rsyslog.d, I have added comments
inline
in the conf that explain the config logic.
/etc/rsyslog.conf
==============================**==============================**
===========
module(load="impstats" interval="600" severity="7"
log.syslog="off" /* need to turn log stream logging off! */
log.file="/var/log/rsyslog-**stats.log")
$ModLoad imuxsock.so # Unix sockets
$ModLoad imklog.so # Kernel logger
$MainMsgQueueSize 1000000
$OMFileIOBufferSize 512k
$MaxMessageSize 8k
$MainMsgQueueWorkerThreads 64
$umask 0000
$FileOwner joe
$FileGroup joe
$DirOwner joe
$DirGroup joe
$DirCreateMode 0755
$FileCreateMode 0644
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$RuleSet local
*.info;mail.none;authpriv.**none;cron.none
/var/log/messages
authpriv.*
/var/log/secure
mail.*
-/var/log/maillog
cron.* /var/log/cron
*.emerg *
uucp,news.crit
/var/log/spooler
local7.*
/var/log/boot.log
$DefaultRuleset local
==============================**==============================**
===========
/etc/rsyslog.d/cef.conf
==============================**==============================**
===========
# Senders are four Arcsight Logger devices that send logs in CEF
format
#template for writing CEF formatted logs
template(name="cefdynfile" type="string"
string="/var/log/joe/%$!**vendor%/%$!product%/logfile")
#template for writing logs from non-CEF sources
template(name="noncefdynfile" type="string"
string="/var/log/joe/noncef/%**hostname%/%programname%/**logfile")
ruleset(name="tcpcef") {
#CEF uses the pipe delimiter, fields 2 and 3 are product vendor and
product
type respectively
set $!vendor = field($msg, 124, 2);
set $!product = field($msg, 124, 3);
# Rules to write CEF formatted logs to disk and send logs by app
type to
their flume destinations that are listening on the same box
if $syslogtag=="CEF:" then { action (type="omfile" ASyncWriting="on"
IOBufferSize="8192K" FileOwner="joe" FileGroup="joe" DirOwner="joe"
DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644"
DynaFile="cefdynfile") }
if $!product == "app1" then { action (type="omfwd" Target="127.0.0.1"
Port="5161" Protocol="tcp") stop }
if $!product == "app2" then { action (type="omfwd" Target="127.0.0.1"
Port="5146" Protocol="tcp") stop }
if $!product == "app3" then { action (type="omfwd" Target="127.0.0.1"
Port="5172" Protocol="tcp") stop }
if $!product == "app4" then { action (type="omfwd" Target="127.0.0.1"
Port="5162" Protocol="tcp") stop }
if $!product == "app5" then { action (type="omfwd" Target="127.0.0.1"
Port="5166" Protocol="tcp") stop }
if $!product == "app6" then { action (type="omfwd" Target="127.0.0.1"
Port="5163" Protocol="tcp") stop }
if $!product == "app7" then { action (type="omfwd" Target="127.0.0.1"
Port="5164" Protocol="tcp") stop }
if $!product == "app8" then { action (type="omfwd" Target="127.0.0.1"
Port="5177" Protocol="tcp") stop }
if $!product == "app9" then { action (type="omfwd" Target="127.0.0.1"
Port="5144" Protocol="tcp") stop }
if $!product == "app10" then { action (type="omfwd"
Target="127.0.0.1"
Port="5145" Protocol="tcp") stop }
if $!product == "app11" then { action (type="omfwd"
Target="127.0.0.1"
Port="5148" Protocol="tcp") stop }
if $!product == "app12" then { action (type="omfwd"
Target="127.0.0.1"
Port="5180" Protocol="tcp") stop }
if $!product == "app13" then { action (type="omfwd"
Target="127.0.0.1"
Port="5147" Protocol="tcp") stop }
if $!product == "app14" then { action (type="omfwd"
Target="127.0.0.1"
Port="5149" Protocol="tcp") stop }
if $!product == "app15" then { action (type="omfwd"
Target="127.0.0.1"
Port="5150" Protocol="tcp") stop }
if $!product == "app16" then { action (type="omfwd"
Target="127.0.0.1"
Port="5151" Protocol="tcp") stop }
if $!product == "app17" then { action (type="omfwd"
Target="127.0.0.1"
Port="5152" Protocol="tcp") stop }
if $!product == "app18" then { action (type="omfwd"
Target="127.0.0.1"
Port="5153" Protocol="tcp") stop }
if $!product == "app19" then { action (type="omfwd"
Target="127.0.0.1"
Port="5155" Protocol="tcp") stop }
if $!product == "app20" then { action (type="omfwd"
Target="127.0.0.1"
Port="5156" Protocol="tcp") stop }
if $!product == "app21" then { action (type="omfwd"
Target="127.0.0.1"
Port="5157" Protocol="tcp") stop }
if $!product == "app22" then { action (type="omfwd"
Target="127.0.0.1"
Port="5158" Protocol="tcp") stop }
if $!product == "app23" then { action (type="omfwd"
Target="127.0.0.1"
Port="5159" Protocol="tcp") stop }
if $!product == "app24" then { action (type="omfwd"
Target="127.0.0.1"
Port="5160" Protocol="tcp") stop }
if $!product == "app25" then { action (type="omfwd"
Target="127.0.0.1"
Port="5178" Protocol="tcp") stop }
if $!product == "app26" then { action (type="omfwd"
Target="127.0.0.1"
Port="5165" Protocol="tcp") stop }
if $!product == "app27" then { action (type="omfwd"
Target="127.0.0.1"
Port="5165" Protocol="tcp") stop }
if $!product == "app28" then { action (type="omfwd"
Target="127.0.0.1"
Port="5167" Protocol="tcp") stop }
if $!product == "app29" then { action (type="omfwd"
Target="127.0.0.1"
Port="5167" Protocol="tcp") stop }
if $!product == "app30" then { action (type="omfwd"
Target="127.0.0.1"
Port="5179" Protocol="tcp") stop }
if $!product == "app31" then { action (type="omfwd"
Target="127.0.0.1"
Port="5169" Protocol="tcp") stop }
if $!product == "app32" then { action (type="omfwd"
Target="127.0.0.1"
Port="5170" Protocol="tcp") stop }
if $!product == "app33" then { action (type="omfwd"
Target="127.0.0.1"
Port="5171" Protocol="tcp") stop }
if $!product == "app34" then { action (type="omfwd"
Target="127.0.0.1"
Port="5174" Protocol="tcp") stop }
if $!product == "app35" then { action (type="omfwd"
Target="127.0.0.1"
Port="5173" Protocol="tcp") stop }
if $!product == "app36" then { action (type="omfwd"
Target="127.0.0.1"
Port="5175" Protocol="tcp") stop }
if $!product == "app37" then { action (type="omfwd"
Target="127.0.0.1"
Port="5176" Protocol="tcp") stop }
if $!product == "app38" then { action (type="omfwd"
Target="127.0.0.1"
Port="5154" Protocol="tcp") stop }
if $!product == "app39" then { action (type="omfwd"
Target="127.0.0.1"
Port="5181" Protocol="tcp") stop }
# Unfortunately, the four Arcsight Loggers also send us garbage so
whatever
could not be parsed/classified correctly by the rules above, gets
parked
in
a catchall file.
if $fromhost-ip == '10.1.1.100' or $fromhost-ip == '10.1.1.101' or
$fromhost-ip == '10.1.1.102' or $fromhost-ip == '10.1.1.103' then {
action
(type="omfile" FileOwner="joe" FileGroup="joe" DirOwner="joe"
DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644"
file="/var/log/joe/fallback/**logfile")
& action (type="omfwd" Target="127.0.0.1" Port="5182"
Protocol="tcp"
)
stop }
# Take care of all the non-CEF / BSD Syslog formatted streams coming
in
else {
action (type="omfile" ASyncWriting="on" IOBufferSize="8192K"
FileOwner="joe" FileGroup="joe" DirOwner="joe" DirGroup="joe"
DirCreateMode="0755" FileCreateMode="0644" DynaFile="noncefdynfile")
& action (type="omfwd" Target="127.0.0.1" Port="5182"
Protocol="tcp")
}
}
module(load="imtcp" ) # needs to be done just once
input(type="imtcp" port="514" ruleset="tcpcef")
==============================**==============================**
===========
Here's the logrotate code:
==============================**==============================**
===========
/var/log/joe/*/*/*
/var/log/joe/*/*/*/*
{
missingok
size 1G
rotate 0
sharedscripts
postrotate
if [ -f /var/run/syslog.pid ]; then \
kill -HUP `cat /var/run/syslog.pid`; \
fi;
endscript
}
==============================**==============================**
===========
On Tue, Jun 25, 2013 at 10:12 AM, David Lang <[email protected]> wrote:
On Tue, 25 Jun 2013, Xuri Nagarin wrote:
On RHEL 6.2 64-bit, I have Rsyslog 7.4.1 (actually, the issue has
existed
for earlier versions too).
After logrotation, the logrotate script HUPs the rsyslogd pid but
the
disk
space doesn't free up until I restart rsyslog. After a few hours
"df"
reports a full filesystem whereas "ls" shows much smaller file
sizes. In
between this confusion, rsyslog seems to stop receiving log
streams.
Is there a way for rsyslog to switch file handles when it is
HUP-ed?
rsyslog is already supposed to close and re-open files when it's
HUP-ed,
and it seems to be working for me and many others.
could you send your config file so we can see if there is anything
unusual
in it?
David Lang
______________________________****_________________
rsyslog mailing list
http://lists.adiscon.net/****mailman/listinfo/rsyslog<
http://lists.adiscon.net/**mailman/listinfo/rsyslog>
<http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<
http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/****professional-services/<
http://www.rsyslog.com/**professional-services/>
<http://**www.rsyslog.com/professional-**services/<
http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<
http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<
http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<
http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<
http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
