Yes your correct, but my template should be writing the logs to a file. So I guess what I should have said is that omelasticsearch sends the logs to ES while rsyslog "writes" the logs to a file. Would that be the better terminology?
As far as a omfile module are you saying that... module(load="builtin:omfile" FileCreateMode="0644" DirCreateMode="0700" FileOwner="syslog" ZipLevel="2") *.* action(type="omfile" File="/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log") Would be better than doing... $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log" *.* ?RemoteHost Does that also recreate the file each day? Meaning....Day1 logs are written and compressed. Then day2 new file is created and compressed and so on... -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Thursday, May 08, 2014 2:46 PM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server On Thu, 8 May 2014, Josh Bitto wrote: > I wanted to thank everyone that gave their input into this question. It > helped out tremendously. > > So I'm still in the test phase of creating a central log server, but I think > I've landed what my final setup is going to be. I'm going give a rundown of > what that is and ask to see if I might have potential problems. > > Rsyslog->Elasticsearch->Kibana > > The rsyslog.conf is pretty much the default file that comes with CentOS > repository with the exception of adding the following: > > ################################### > > template(name="logstash-index" > type="list") { > constant(value="logstash-") > property(name="timereported" dateFormat="rfc3339" position.from="1" > position.to="4") > constant(value=".") > property(name="timereported" dateFormat="rfc3339" position.from="6" > position.to="7") > constant(value=".") > property(name="timereported" dateFormat="rfc3339" position.from="9" > position.to="10") } > > # this is for formatting our syslog in JSON with @timestamp > template(name="plain-syslog" > type="list") { > constant(value="{") > constant(value="\"@timestamp\":\"") property(name="timereported" > dateFormat="rfc3339") > constant(value="\",\"host\":\"") property(name="hostname") > constant(value="\",\"severity\":\"") > property(name="syslogseverity-text") > constant(value="\",\"facility\":\"") > property(name="syslogfacility-text") > constant(value="\",\"tag\":\"") property(name="syslogtag" > format="json") > constant(value="\",\"message\":\"") property(name="msg" format="json") > constant(value="\"}") > } > > # this is where we actually send the logs to Elasticsearch > (localhost:9200 by default) action(type="omelasticsearch" > template="plain-syslog" > searchIndex="logstash-index" > dynSearchIndex="on") > > $ModLoad imfile # Load the imfile input module > > # Watch /var/log/httpd/access_log > $InputFileName /var/log/httpd/access_log $InputFileTag apache-access: > $InputFileStateFile state-apache-access $InputRunFileMonitor > > # Watch /var/log/httpd/error_log > $InputFileName /var/log/httpd/error_log $InputFileTag apache-error: > $InputFileStateFile state-apache-error $InputRunFileMonitor > > # Watch /var/ossec/logs/alerts/alerts.log $InputFileName > /var/ossec/logs/alerts/alerts.log $InputFileTag ossec-alert: > $InputFileStateFile state-ossec-alert > $InputRunFileMonitor > > > $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log" > > *.* ?RemoteHost > > > if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' > or $msg contains 'Received SNMP packet(s) from UDP' ) then ~ > > ############################################################# > > So basically I have rsyslog writing incoming logs to a file and then they are > being sent for ES and kibana. Eventually I will have a logrotation done so > that they can be rotated and compressed to save space. > > Thanks guys! a couple nits omelasticsearch isn't writing them to a file (your description of write them to a file then send them is misleading) take a look at the options for omfile that can compress the output in place. It means that the logs are buffered a little bit more, and things can get a bit interesting if rsyslog crashes in the middle of a write, but it avoids the need to play the rotation game. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
