I'm stumped on a problem involving log consolidation and imfile creating duplicate events in the log and believe it's a bug. I have rsyslog using imfile to forward web access logs to a central server. Each of the source log files is 2.5 to 4.5 GB depending on the traffic that day. The destination ends up being 4 or 5 times that size. The duplicate events occur at very large intervals in the log. Is this possible a 32bit offset problem with files larger than 2GB? I'm doing the same thing with much smaller files and don't have this problem. In all cases rsyslog is restarted every night but at different times to account for log rotation, client and server. Client is Rsyslog 4.4.2 and collector is Rsyslog 5.5.6 both running on Ubuntu linux 8.04.4 x86_64.
I stopped the client rsyslog running the imfile module at an hour interval and copied the contents of the control file <Obj:1:strm:1: +iCurrFNum:2:1:1: +pszFName:1:38:/opt/wp/log/nginx/wpn_rails_access.log: +iMaxFiles:2:1:0: +bDeleteOnClose:2:1:0: +sType:2:1:2: +tOperationsMode:2:1:1: +tOpenMode:2:3:384: +iCurrOffs:2:10:1883810959: >End . <Obj:1:strm:1: +iCurrFNum:2:1:1: +pszFName:1:38:/opt/wp/log/nginx/wpn_rails_access.log: +iMaxFiles:2:1:0: +bDeleteOnClose:2:1:0: +sType:2:1:2: +tOperationsMode:2:1:1: +tOpenMode:2:3:384: +iCurrOffs:2:10:2083988720: >End . Example log lines prepended with lineNumber:byteOffset from grep -xbnF for entire line: 3443:1134236:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce"; "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" 2239059:821710731:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce"; "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" 4976860:1834241112:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce"; "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" 9546150:3535005742:65.161.86.4 192.168.2.12 - [01/Oct/2010:00:04:03 -0700] "GET /common/images/clear/802844224332.gif HTTP/1.1" 200 43 "http://www.whitepages.com/business/dairy-queen-zumbrota-mn?t=3dea4c5c833e45ec8f61051e0fc540ce"; "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AskTbBT5/3.8.0.12304 Firefox/3.6.10" "www.whitepages.com" "-" "-" >>Relevant rsyslog.conf on client<< $template commonForwardFormat,"<%PRI%>%timereported:::date-rfc3339% %fromhost% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" $ModLoad imfile $InputFileName /opt/wp/log/nginx/wpn_rails_access.log $InputFilePollInterval 5 $InputFileStateFile wpn_rails_access-offset_state-rt_weblog $InputFileFacility local6 $InputFileTag NginxAccess: $InputRunFileMonitor if $programname == 'NginxAccess' then @@syslog1:10518;commonForwardFormat & ~ >>Relevant rsyslog.conf on syslog collector<< # Remove the rsyslog appended timestamp host and tag $template wpnrailsAccessLogFixup,"%msg:15:$:%\n" $template DYNwpnrailsAccess,"/var/log/remote/wpnrails-access/%fromhost%/%fromhost%-access-%timestamp:1:10:date-rfc3339%.log" $template DYNwpnrailsAccessCombined,"/var/log/remote/wpnrails-access-combined/access-combined-%timestamp:1:10:date-rfc3339%.log" $RuleSet wpnrailsAccess $RulesetCreateMainQueue on *.* ?DYNwpnrailsAccess;wpnrailsAccessLogFixup *.* ?DYNwpnrailsAccessCombined;wpnrailsAccessLogFixup $InputTCPServerBindRuleset wpnrailsAccess $InputTCPServerRun 10518 _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
