On Thu, 8 May 2014, Josh Bitto wrote:
Yes your correct, but my template should be writing the logs to a file. So I guess what I
should have said is that omelasticsearch sends the logs to ES while rsyslog
"writes" the logs to a file. Would that be the better terminology?
As far as a omfile module are you saying that...
module(load="builtin:omfile" FileCreateMode="0644" DirCreateMode="0700"
FileOwner="syslog" ZipLevel="2")
*.* action(type="omfile"
File="/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log")
Would be better than doing...
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log"
*.* ?RemoteHost
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log.gz"
action(name="PerHostFile" type="omfile" dynafile="RemoteHost"
DynaFileCacheSize="1000" ziplevel='5')
with v7 you don't need *.*
this creates a compressed file to begin with, and every day will get a new file
if rsyslog crashes (as opposed to shutting down) it's best to mv the file before
starting a new rsyslog. VeryRobustZip='on' solves this problem, at the cost of
worse compression
if the omfile falls behind, you can add ASyncWriting="on" to this so that a
different thread does the compression than the rest of the work (only needed at
high traffic levels)
if you are writing for more than 1000 hosts/day, change the DynaFileCachSize
variable to be larger than the number of hosts you write for (the default is 10)
Does that also recreate the file each day? Meaning....Day1 logs are written and
compressed. Then day2 new file is created and compressed and so on...
it doesn't write it and then compress it, it compresses it and then writes it,
so it's always compressed.
David Lang
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Thursday, May 08, 2014 2:46 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
On Thu, 8 May 2014, Josh Bitto wrote:
I wanted to thank everyone that gave their input into this question. It helped
out tremendously.
So I'm still in the test phase of creating a central log server, but I think
I've landed what my final setup is going to be. I'm going give a rundown of
what that is and ask to see if I might have potential problems.
Rsyslog->Elasticsearch->Kibana
The rsyslog.conf is pretty much the default file that comes with CentOS
repository with the exception of adding the following:
###################################
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10") }
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch
(localhost:9200 by default) action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log $InputFileTag apache-access:
$InputFileStateFile state-apache-access $InputRunFileMonitor
# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log $InputFileTag apache-error:
$InputFileStateFile state-apache-error $InputRunFileMonitor
# Watch /var/ossec/logs/alerts/alerts.log $InputFileName
/var/ossec/logs/alerts/alerts.log $InputFileTag ossec-alert:
$InputFileStateFile state-ossec-alert
$InputRunFileMonitor
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log"
*.* ?RemoteHost
if $programname == 'snmpd' and ( $msg contains 'Connection from UDP'
or $msg contains 'Received SNMP packet(s) from UDP' ) then ~
#############################################################
So basically I have rsyslog writing incoming logs to a file and then they are
being sent for ES and kibana. Eventually I will have a logrotation done so that
they can be rotated and compressed to save space.
Thanks guys!
a couple nits
omelasticsearch isn't writing them to a file (your description of write them to
a file then send them is misleading)
take a look at the options for omfile that can compress the output in place. It
means that the logs are buffered a little bit more, and things can get a bit
interesting if rsyslog crashes in the middle of a write, but it avoids the need
to play the rotation game.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
