My suspicion was something to do with dynamic file creation but looking at the code, both static and dynamic file creation use the same function of file "open":
>From tools/omfile.c ------------xxxxxxxxxxxxxxxxx--------------- fd = open((char*) newFileName, O_WRONLY|O_APPEND|O_CREAT|O_NOCTTY|O_CLOEXEC, 556 pData->fCreateMode); ------------xxxxxxxxxxxxxxxxx--------------- "O_APPEND" should take care of file being rotated while rsyslog is trying to write to it. On Tue, Jun 25, 2013 at 11:47 AM, Xuri Nagarin <[email protected]> wrote: > Not a typo or error, in RHEL the rsyslogd start up script in init.d has an > explicit variable "PIDFILE" set as: > PIDFILE=/var/run/syslogd.pid > > :) > > I did not write the script, using whatever was bundled in the RPM I > grabbed from Adiscon. > > To confirm: > $ ps ax | grep rsyslog > 11331 ? Sl 61:18 /sbin/rsyslogd -i /var/run/syslogd.pid -x > > > > > > On Tue, Jun 25, 2013 at 11:41 AM, Soham Chakraborty < > [email protected]> wrote: > >> Hi, >> >> In the logrotate config, change it to /var/run/rsyslog.pid and test. On a >> quick skim, it looks like a typo. >> >> Soham >> >> >> On Tue, Jun 25, 2013 at 11:07 PM, David Lang <[email protected]> wrote: >> >> > well, one thing that looks wrong is that logrotate is looking for >> > /var/run/syslog.pid, but on my systems the pid is in >> /var/log/rsyslog.pid >> > >> > are you sure that rsyslog is actually getting the HUP? >> > >> > can you try sending it a HUP manually and see if it closes the files? >> > >> > David Lang >> > >> > >> > On Tue, 25 Jun 2013, Xuri Nagarin wrote: >> > >> > Date: Tue, 25 Jun 2013 11:34:08 -0700 >> >> From: Xuri Nagarin <[email protected]> >> >> Reply-To: rsyslog-users <[email protected]> >> >> To: rsyslog-users <[email protected]> >> >> Subject: Re: [rsyslog] HUP-ing rsyslog does not free up disk space >> >> >> >> >> >> Hi David, >> >> >> >> The master conf file is simple with few additions that are >> >> self-explanatory. The other conf in rsyslog.d, I have added comments >> >> inline >> >> in the conf that explain the config logic. >> >> >> >> >> >> /etc/rsyslog.conf >> >> ==============================**==============================** >> >> =========== >> >> module(load="impstats" interval="600" severity="7" >> >> >> >> log.syslog="off" /* need to turn log stream logging off! */ >> >> log.file="/var/log/rsyslog-**stats.log") >> >> >> >> $ModLoad imuxsock.so # Unix sockets >> >> $ModLoad imklog.so # Kernel logger >> >> $MainMsgQueueSize 1000000 >> >> $OMFileIOBufferSize 512k >> >> $MaxMessageSize 8k >> >> $MainMsgQueueWorkerThreads 64 >> >> $umask 0000 >> >> $FileOwner joe >> >> $FileGroup joe >> >> $DirOwner joe >> >> $DirGroup joe >> >> $DirCreateMode 0755 >> >> $FileCreateMode 0644 >> >> >> >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> >> $IncludeConfig /etc/rsyslog.d/*.conf >> >> >> >> $RuleSet local >> >> *.info;mail.none;authpriv.**none;cron.none >> >> /var/log/messages >> >> authpriv.* >> /var/log/secure >> >> mail.* >> -/var/log/maillog >> >> cron.* /var/log/cron >> >> *.emerg * >> >> uucp,news.crit >> /var/log/spooler >> >> local7.* >> /var/log/boot.log >> >> $DefaultRuleset local >> >> ==============================**==============================** >> >> =========== >> >> >> >> /etc/rsyslog.d/cef.conf >> >> ==============================**==============================** >> >> =========== >> >> # Senders are four Arcsight Logger devices that send logs in CEF format >> >> >> >> #template for writing CEF formatted logs >> >> template(name="cefdynfile" type="string" >> >> string="/var/log/joe/%$!**vendor%/%$!product%/logfile") >> >> >> >> #template for writing logs from non-CEF sources >> >> template(name="noncefdynfile" type="string" >> >> string="/var/log/joe/noncef/%**hostname%/%programname%/**logfile") >> >> >> >> ruleset(name="tcpcef") { >> >> >> >> #CEF uses the pipe delimiter, fields 2 and 3 are product vendor and >> >> product >> >> type respectively >> >> set $!vendor = field($msg, 124, 2); >> >> set $!product = field($msg, 124, 3); >> >> >> >> # Rules to write CEF formatted logs to disk and send logs by app type >> to >> >> their flume destinations that are listening on the same box >> >> >> >> if $syslogtag=="CEF:" then { action (type="omfile" ASyncWriting="on" >> >> IOBufferSize="8192K" FileOwner="joe" FileGroup="joe" DirOwner="joe" >> >> DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644" >> >> DynaFile="cefdynfile") } >> >> if $!product == "app1" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5161" Protocol="tcp") stop } >> >> if $!product == "app2" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5146" Protocol="tcp") stop } >> >> if $!product == "app3" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5172" Protocol="tcp") stop } >> >> if $!product == "app4" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5162" Protocol="tcp") stop } >> >> if $!product == "app5" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5166" Protocol="tcp") stop } >> >> if $!product == "app6" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5163" Protocol="tcp") stop } >> >> if $!product == "app7" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5164" Protocol="tcp") stop } >> >> if $!product == "app8" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5177" Protocol="tcp") stop } >> >> if $!product == "app9" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5144" Protocol="tcp") stop } >> >> if $!product == "app10" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5145" Protocol="tcp") stop } >> >> if $!product == "app11" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5148" Protocol="tcp") stop } >> >> if $!product == "app12" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5180" Protocol="tcp") stop } >> >> if $!product == "app13" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5147" Protocol="tcp") stop } >> >> if $!product == "app14" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5149" Protocol="tcp") stop } >> >> if $!product == "app15" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5150" Protocol="tcp") stop } >> >> if $!product == "app16" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5151" Protocol="tcp") stop } >> >> if $!product == "app17" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5152" Protocol="tcp") stop } >> >> if $!product == "app18" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5153" Protocol="tcp") stop } >> >> if $!product == "app19" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5155" Protocol="tcp") stop } >> >> if $!product == "app20" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5156" Protocol="tcp") stop } >> >> if $!product == "app21" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5157" Protocol="tcp") stop } >> >> if $!product == "app22" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5158" Protocol="tcp") stop } >> >> if $!product == "app23" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5159" Protocol="tcp") stop } >> >> if $!product == "app24" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5160" Protocol="tcp") stop } >> >> if $!product == "app25" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5178" Protocol="tcp") stop } >> >> if $!product == "app26" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5165" Protocol="tcp") stop } >> >> if $!product == "app27" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5165" Protocol="tcp") stop } >> >> if $!product == "app28" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5167" Protocol="tcp") stop } >> >> if $!product == "app29" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5167" Protocol="tcp") stop } >> >> if $!product == "app30" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5179" Protocol="tcp") stop } >> >> if $!product == "app31" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5169" Protocol="tcp") stop } >> >> if $!product == "app32" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5170" Protocol="tcp") stop } >> >> if $!product == "app33" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5171" Protocol="tcp") stop } >> >> if $!product == "app34" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5174" Protocol="tcp") stop } >> >> if $!product == "app35" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5173" Protocol="tcp") stop } >> >> if $!product == "app36" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5175" Protocol="tcp") stop } >> >> if $!product == "app37" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5176" Protocol="tcp") stop } >> >> if $!product == "app38" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5154" Protocol="tcp") stop } >> >> if $!product == "app39" then { action (type="omfwd" Target="127.0.0.1" >> >> Port="5181" Protocol="tcp") stop } >> >> >> >> # Unfortunately, the four Arcsight Loggers also send us garbage so >> >> whatever >> >> could not be parsed/classified correctly by the rules above, gets >> parked >> >> in >> >> a catchall file. >> >> >> >> if $fromhost-ip == '10.1.1.100' or $fromhost-ip == '10.1.1.101' or >> >> $fromhost-ip == '10.1.1.102' or $fromhost-ip == '10.1.1.103' then { >> action >> >> (type="omfile" FileOwner="joe" FileGroup="joe" DirOwner="joe" >> >> DirGroup="joe" DirCreateMode="0755" FileCreateMode="0644" >> >> file="/var/log/joe/fallback/**logfile") >> >> & action (type="omfwd" Target="127.0.0.1" Port="5182" >> Protocol="tcp" >> >> ) >> >> stop } >> >> >> >> # Take care of all the non-CEF / BSD Syslog formatted streams coming in >> >> else { >> >> action (type="omfile" ASyncWriting="on" IOBufferSize="8192K" >> >> FileOwner="joe" FileGroup="joe" DirOwner="joe" DirGroup="joe" >> >> DirCreateMode="0755" FileCreateMode="0644" DynaFile="noncefdynfile") >> >> & action (type="omfwd" Target="127.0.0.1" Port="5182" >> Protocol="tcp") >> >> } >> >> } >> >> >> >> module(load="imtcp" ) # needs to be done just once >> >> input(type="imtcp" port="514" ruleset="tcpcef") >> >> ==============================**==============================** >> >> =========== >> >> >> >> >> >> Here's the logrotate code: >> >> ==============================**==============================** >> >> =========== >> >> /var/log/joe/*/*/* >> >> /var/log/joe/*/*/*/* >> >> { >> >> missingok >> >> size 1G >> >> rotate 0 >> >> sharedscripts >> >> postrotate >> >> if [ -f /var/run/syslog.pid ]; then \ >> >> kill -HUP `cat /var/run/syslog.pid`; \ >> >> fi; >> >> endscript >> >> } >> >> ==============================**==============================** >> >> =========== >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Jun 25, 2013 at 10:12 AM, David Lang <[email protected]> wrote: >> >> >> >> On Tue, 25 Jun 2013, Xuri Nagarin wrote: >> >>> >> >>> On RHEL 6.2 64-bit, I have Rsyslog 7.4.1 (actually, the issue has >> >>> existed >> >>> >> >>>> for earlier versions too). >> >>>> >> >>>> After logrotation, the logrotate script HUPs the rsyslogd pid but the >> >>>> disk >> >>>> space doesn't free up until I restart rsyslog. After a few hours "df" >> >>>> reports a full filesystem whereas "ls" shows much smaller file >> sizes. In >> >>>> between this confusion, rsyslog seems to stop receiving log streams. >> >>>> >> >>>> Is there a way for rsyslog to switch file handles when it is HUP-ed? >> >>>> >> >>>> >> >>> rsyslog is already supposed to close and re-open files when it's >> HUP-ed, >> >>> and it seems to be working for me and many others. >> >>> >> >>> could you send your config file so we can see if there is anything >> >>> unusual >> >>> in it? >> >>> >> >>> David Lang >> >>> ______________________________****_________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog< >> http://lists.adiscon.net/**mailman/listinfo/rsyslog> >> >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog< >> http://lists.adiscon.net/mailman/listinfo/rsyslog> >> >>> > >> >>> http://www.rsyslog.com/****professional-services/< >> http://www.rsyslog.com/**professional-services/> >> >>> <http://**www.rsyslog.com/professional-**services/< >> http://www.rsyslog.com/professional-services/> >> >>> > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>> DON'T LIKE THAT. >> >>> >> >>> ______________________________**_________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< >> http://lists.adiscon.net/mailman/listinfo/rsyslog> >> >> http://www.rsyslog.com/**professional-services/< >> http://www.rsyslog.com/professional-services/> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> >> ______________________________**_________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/**mailman/listinfo/rsyslog< >> http://lists.adiscon.net/mailman/listinfo/rsyslog> >> > http://www.rsyslog.com/**professional-services/< >> http://www.rsyslog.com/professional-services/> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
