On Thu, 8 May 2014, Josh Bitto wrote:
I wanted to thank everyone that gave their input into this question. It helped
out tremendously.
So I'm still in the test phase of creating a central log server, but I think
I've landed what my final setup is going to be. I'm going give a rundown of
what that is and ask to see if I might have potential problems.
Rsyslog->Elasticsearch->Kibana
The rsyslog.conf is pretty much the default file that comes with CentOS
repository with the exception of adding the following:
###################################
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by
default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log
$InputFileTag apache-access:
$InputFileStateFile state-apache-access
$InputRunFileMonitor
# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log
$InputFileTag apache-error:
$InputFileStateFile state-apache-error
$InputRunFileMonitor
# Watch /var/ossec/logs/alerts/alerts.log
$InputFileName /var/ossec/logs/alerts/alerts.log
$InputFileTag ossec-alert:
$InputFileStateFile state-ossec-alert
$InputRunFileMonitor
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log"
*.* ?RemoteHost
if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' or $msg
contains 'Received SNMP packet(s) from UDP' ) then ~
#############################################################
So basically I have rsyslog writing incoming logs to a file and then they are
being sent for ES and kibana. Eventually I will have a logrotation done so that
they can be rotated and compressed to save space.
Thanks guys!
a couple nits
omelasticsearch isn't writing them to a file (your description of write them to
a file then send them is misleading)
take a look at the options for omfile that can compress the output in place. It
means that the logs are buffered a little bit more, and things can get a bit
interesting if rsyslog crashes in the middle of a write, but it avoids the need
to play the rotation game.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
