While waiting on this response, I decided to update to 7.4.6-1.el6.x86_64. This is that I'm running as of now. I modified the default rsyslog.conf (making a backup of course) to load the tcp and udp modules, uncommented the forwarding rule and added my SIEM server's ip address, and added these lines for writing local files:
$template DailyPerHostLogs,"/var/log/syslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/messages.log" *.* -?DailyPerHostLogs I restarted the daemon and received an error about the -r option no longer being supported as well as the -m (the old /etc/sysconfig/rsyslog file is still there with SYSLOGD_OPTIONS="-r514 -m0" in it, I should probably update that soon). Watching the syslogs come in via tcpdump and tailing the newly created file for one of my hosts, I can see that the logs are being written to the file at the same time that they're being received. This is precisely what I need to happen! On Wed, Nov 6, 2013 at 10:03 AM, David Lang <[email protected]> wrote: > With your configuration, the logs are split into different files based on > when rsyslog is writing them out, not the timestamp in the log messages, so > if logs are delayed (either at the sender, or receiver) that can cause some > issues. > > You are running rsyslog 5.x, the current community supported version is 7.4, > and it includes a _lot_ of performance improvements. can you upgrade to > that? > > After you are on a reasonably current version, we can do a lot more to help > track down any remaining problems. > > David Lang > > > On Wed, 6 Nov 2013, G Jones wrote: > >> I left rsyslog running overnight to see if it stopped writing to the >> file again, the last entry it shows happened at 16:49.56, I used kill >> -HUP and after the process finally stopped I rechecked the file. >> Nothing changed. However, I didn't realize that, for some reason, >> rsyslog decided that 16:49 was the end of a day and created a new file >> dated for 11/6 with it's first entry starting at 16:50 on 11/5 (which >> will need to be addressed as well since this also happened with my >> last config). I checked the newly created (though incorrectly dated) >> file and the last entry it shows is for 2013-11-05 at 18:23:52. On the >> relay side, I've checked my SIEM and the last received entry came in >> right before I sent rsyslog the kill command. >> >> What I fail to see here is why rsyslog will relay messages but not >> write those same messages to a file or simply stop writing them all >> together. Then there's the issue of what constitutes the end of a day >> before rsyslog creates a new file. >> >> On Tue, Nov 5, 2013 at 5:19 PM, David Lang <[email protected]> wrote: >>> >>> some of it is probably buffering on the output, if you send rsyslog a >>> kill >>> -HUP it will flush all data and close the files. Check and see if this >>> gets >>> the data in them as you expect. >>> >>> David Lang >>> >>> On Tue, 5 Nov 2013, G Jones wrote: >>> >>>> Date: Tue, 5 Nov 2013 16:25:03 -0800 >>>> From: G Jones <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: [email protected] >>>> Subject: [rsyslog] rsyslog writing to file lagging behind relayed logs >>>> >>>> >>>> Hello, new to the list and to Rsyslog. >>>> >>>> I have an Rsyslog machine I'm standing up on CentOS x64 to replace >>>> some Kiwi log servers. I need to be able to receive syslog on TCP and >>>> UDP, relay them to my SIEM and also store those logs locally on each >>>> Rsyslog machine. Relaying is working just fine. The problem is with >>>> the local files. I noticed that the files being written for each host >>>> I recieve is lagging behind by minutes to upwards of a full day >>>> behind. I'm running a tcpdump on both the rsyslog machine and the SIEM >>>> and can see when the logs are received on both, however running a tail >>>> -f on the files show that those entries are lagging behind. It gets >>>> progressively worse as time goes on. They start out in near perfect >>>> sync for the first couple of minutes, then the messages start to lag >>>> horribly. >>>> >>>> I just tried using this config, modified a bit for my environment: >>>> >>>> >>>> # Modules$ModLoad imtcp >>>> $ModLoad imudp >>>> $ModLoad imuxsock >>>> $ModLoad imklog# Templates# log every host in its own directory >>>> $template >>>> >>>> RemoteHost,"/var/opt/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"### >>>> Rulesets# Local Logging >>>> $RuleSet local >>>> kern.* /var/log/messages >>>> *.info;mail.none;authpriv.none;cron.none >>>> /var/log/messages >>>> authpriv.* /var/log/secure >>>> mail.* >>>> -/var/log/maillog >>>> cron.* /var/log/cron >>>> *.emerg * >>>> uucp,news.crit /var/log/spooler >>>> local7.* >>>> /var/log/boot.log >>>> # use the local RuleSet as default if not specified otherwise >>>> $DefaultRuleset local >>>> # Remote Logging >>>> $RuleSet remote >>>> *.* ?RemoteHost >>>> # Send messages we receive to SIEM >>>> *.* @a.b.c.d:1514 >>>> ### Listeners >>>> # bind ruleset to tcp listener >>>> $InputTCPServerBindRuleset remote >>>> # and activate it: >>>> $InputTCPServerRun 1514 >>>> $InputUDPServerBindRuleset remote >>>> $UDPServerRun 514 >>>> >>>> This particular config does exactly the same thing: I can watch syslog >>>> messages come in, but they're not being written to the log file until >>>> much later, if at all. Any thoughts as to why this is happening? >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T >>>> LIKE THAT. >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
