Thanks David, Had to change the ziplevel '5' to "5".
Now it's compressing and writing to the file, but it isn't showing up in kibana when I run a test that creates logs. Is placement in my config the key? Or should I be adjusting the part for Elastic search accordingly? -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Thursday, May 08, 2014 3:26 PM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server On Thu, 8 May 2014, Josh Bitto wrote: > Yes your correct, but my template should be writing the logs to a file. So I > guess what I should have said is that omelasticsearch sends the logs to ES > while rsyslog "writes" the logs to a file. Would that be the better > terminology? > > As far as a omfile module are you saying that... > > module(load="builtin:omfile" FileCreateMode="0644" > DirCreateMode="0700" FileOwner="syslog" ZipLevel="2") > *.* action(type="omfile" > File="/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log") > > Would be better than doing... > > $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log" > > *.* ?RemoteHost $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log.gz" action(name="PerHostFile" type="omfile" dynafile="RemoteHost" DynaFileCacheSize="1000" ziplevel='5') with v7 you don't need *.* this creates a compressed file to begin with, and every day will get a new file if rsyslog crashes (as opposed to shutting down) it's best to mv the file before starting a new rsyslog. VeryRobustZip='on' solves this problem, at the cost of worse compression if the omfile falls behind, you can add ASyncWriting="on" to this so that a different thread does the compression than the rest of the work (only needed at high traffic levels) if you are writing for more than 1000 hosts/day, change the DynaFileCachSize variable to be larger than the number of hosts you write for (the default is 10) > Does that also recreate the file each day? Meaning....Day1 logs are written > and compressed. Then day2 new file is created and compressed and so on... it doesn't write it and then compress it, it compresses it and then writes it, so it's always compressed. David Lang > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of David Lang > Sent: Thursday, May 08, 2014 2:46 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > > On Thu, 8 May 2014, Josh Bitto wrote: > >> I wanted to thank everyone that gave their input into this question. It >> helped out tremendously. >> >> So I'm still in the test phase of creating a central log server, but I think >> I've landed what my final setup is going to be. I'm going give a rundown of >> what that is and ask to see if I might have potential problems. >> >> Rsyslog->Elasticsearch->Kibana >> >> The rsyslog.conf is pretty much the default file that comes with CentOS >> repository with the exception of adding the following: >> >> ################################### >> >> template(name="logstash-index" >> type="list") { >> constant(value="logstash-") >> property(name="timereported" dateFormat="rfc3339" position.from="1" >> position.to="4") >> constant(value=".") >> property(name="timereported" dateFormat="rfc3339" position.from="6" >> position.to="7") >> constant(value=".") >> property(name="timereported" dateFormat="rfc3339" position.from="9" >> position.to="10") } >> >> # this is for formatting our syslog in JSON with @timestamp >> template(name="plain-syslog" >> type="list") { >> constant(value="{") >> constant(value="\"@timestamp\":\"") property(name="timereported" >> dateFormat="rfc3339") >> constant(value="\",\"host\":\"") property(name="hostname") >> constant(value="\",\"severity\":\"") >> property(name="syslogseverity-text") >> constant(value="\",\"facility\":\"") >> property(name="syslogfacility-text") >> constant(value="\",\"tag\":\"") property(name="syslogtag" >> format="json") >> constant(value="\",\"message\":\"") property(name="msg" >> format="json") >> constant(value="\"}") >> } >> >> # this is where we actually send the logs to Elasticsearch >> (localhost:9200 by default) action(type="omelasticsearch" >> template="plain-syslog" >> searchIndex="logstash-index" >> dynSearchIndex="on") >> >> $ModLoad imfile # Load the imfile input module >> >> # Watch /var/log/httpd/access_log >> $InputFileName /var/log/httpd/access_log $InputFileTag apache-access: >> $InputFileStateFile state-apache-access $InputRunFileMonitor >> >> # Watch /var/log/httpd/error_log >> $InputFileName /var/log/httpd/error_log $InputFileTag apache-error: >> $InputFileStateFile state-apache-error $InputRunFileMonitor >> >> # Watch /var/ossec/logs/alerts/alerts.log $InputFileName >> /var/ossec/logs/alerts/alerts.log $InputFileTag ossec-alert: >> $InputFileStateFile state-ossec-alert >> $InputRunFileMonitor >> >> >> $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log" >> >> *.* ?RemoteHost >> >> >> if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' >> or $msg contains 'Received SNMP packet(s) from UDP' ) then ~ >> >> ############################################################# >> >> So basically I have rsyslog writing incoming logs to a file and then they >> are being sent for ES and kibana. Eventually I will have a logrotation done >> so that they can be rotated and compressed to save space. >> >> Thanks guys! > > a couple nits > > omelasticsearch isn't writing them to a file (your description of write them > to a file then send them is misleading) > > take a look at the options for omfile that can compress the output in place. > It means that the logs are buffered a little bit more, and things can get a > bit interesting if rsyslog crashes in the middle of a write, but it avoids > the need to play the rotation game. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This > is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our > control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
