I wanted to thank everyone that gave their input into this question. It helped
out tremendously.
So I'm still in the test phase of creating a central log server, but I think
I've landed what my final setup is going to be. I'm going give a rundown of
what that is and ask to see if I might have potential problems.
Rsyslog->Elasticsearch->Kibana
The rsyslog.conf is pretty much the default file that comes with CentOS
repository with the exception of adding the following:
###################################
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by
default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log
$InputFileTag apache-access:
$InputFileStateFile state-apache-access
$InputRunFileMonitor
# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log
$InputFileTag apache-error:
$InputFileStateFile state-apache-error
$InputRunFileMonitor
# Watch /var/ossec/logs/alerts/alerts.log
$InputFileName /var/ossec/logs/alerts/alerts.log
$InputFileTag ossec-alert:
$InputFileStateFile state-ossec-alert
$InputRunFileMonitor
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log"
*.* ?RemoteHost
if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' or $msg
contains 'Received SNMP packet(s) from UDP' ) then ~
#############################################################
So basically I have rsyslog writing incoming logs to a file and then they are
being sent for ES and kibana. Eventually I will have a logrotation done so that
they can be rotated and compressed to save space.
Thanks guys!
Josh
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Friday, May 02, 2014 4:20 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
Ok, thanks things were wrapped oddly.
I don't see anything obvious here, I would suggest writing to a file with the
format plain-syslog and to another file with the format logstash-index
It's very possible that something is going in there that's odd
David Lang
On Fri, 2 May 2014, Josh Bitto wrote:
> Date: Fri, 2 May 2014 15:03:20 -0700
> From: Josh Bitto <[email protected]>
> Reply-To: rsyslog-users <[email protected]>
> To: rsyslog-users <[email protected]>
> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
>
> I will repost the entire config.
>
> ######################################################################
> ####
>
>
> # rsyslog configuration file
> # note that most of this config file uses old-style format, # because
> it is well-known AND quite suitable for simple cases # like we have
> with the default config. For more advanced # things, RainerScript
> configuration is suggested.
>
> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
> # If you experience problems, see
> http://www.rsyslog.com/doc/troubleshoot.html
>
> #### MODULES ####
>
> module(load="imuxsock") # provides support for local system logging (e.g. via
> logger command)
> module(load="imklog") # provides kernel logging support (previously done by
> rklogd)
> #module(load"immark") # provides --MARK-- message capability
>
> # Provides UDP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imudp.html
> module(load="imudp") # needs to be done just once input(type="imudp"
> port="514")
>
> # Provides TCP syslog reception
> # for parameters see http://www.rsyslog.com/doc/imtcp.html
> #module(load="imtcp") # needs to be done just once #input(type="imtcp"
> port="514")
>
> syslog.* /var/log/rsyslogd.log # per Rainers suggestion..note email
>
>
> #### GLOBAL DIRECTIVES ####
>
> # Use default timestamp format
> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>
> # File syncing capability is disabled by default. This feature is
> usually not required, # not useful and an extreme performance hit
> #$ActionFileEnableSync on
>
> # Include all config files in /etc/rsyslog.d/ $IncludeConfig
> /etc/rsyslog.d/*.conf
>
> module(load="impstats" interval="30" severity="7")
>
>
> #### RULES ####
>
> # Log all kernel messages to the console.
> # Logging much else clutters up the screen.
> #kern.* /dev/console
>
> # Log anything (except mail) of level info or higher.
> # Don't log private authentication messages!
> *.info;mail.none;authpriv.none;cron.none /var/log/messages
>
> # The authpriv file has restricted access.
> authpriv.* /var/log/secure
>
> # Log all the mail messages in one place.
> mail.* /var/log/maillog
>
>
> # Log cron stuff
> cron.* /var/log/cron
>
> # Everybody gets emergency messages
> *.emerg :omusrmsg:*
>
> # Save news errors of level crit and higher in a special file.
> uucp,news.crit /var/log/spooler
>
> # Save boot messages also to boot.log
> local7.* /var/log/boot.log
>
> template(name="logstash-index"
> type="list") {
> constant(value="logstash-")
> property(name="timereported" dateFormat="rfc3339" position.from="1"
> position.to="4")
> constant(value=".")
> property(name="timereported" dateFormat="rfc3339" position.from="6"
> position.to="7")
> constant(value=".")
> property(name="timereported" dateFormat="rfc3339" position.from="9"
> position.to="10") }
>
> # this is for formatting our syslog in JSON with @timestamp
> template(name="plain-syslog"
> type="list") {
> constant(value="{")
> constant(value="\"@timestamp\":\"") property(name="timereported"
> dateFormat="rfc3339")
> constant(value="\",\"host\":\"") property(name="hostname")
> constant(value="\",\"severity\":\"")
> property(name="syslogseverity-text")
> constant(value="\",\"facility\":\"")
> property(name="syslogfacility-text")
> constant(value="\",\"tag\":\"") property(name="syslogtag"
> format="json")
> constant(value="\",\"message\":\"") property(name="msg" format="json")
> constant(value="\"}")
> }
>
> # this is where we actually send the logs to Elasticsearch
> (localhost:9200 by default) action(type="omelasticsearch"
> template="plain-syslog"
> searchIndex="logstash-index"
> dynSearchIndex="on")
>
> $ModLoad imfile # Load the imfile input module
>
> # Watch /var/log/httpd/access_log
> #$InputFileName /var/log/httpd/access_log #$InputFileTag
> apache-access:
> #$InputFileStateFile state-apache-access #$InputRunFileMonitor
>
> # Watch /var/log/httpd/error_log
> #$InputFileName /var/log/httpd/error_log #$InputFileTag apache-error:
> #$InputFileStateFile state-apache-error #$InputRunFileMonitor
>
>
> # ### begin forwarding rule ###
> # The statement between the begin ... end define a SINGLE forwarding #
> rule. They belong together, do NOT split them. If you create multiple
> # forwarding rules, duplicate the whole block!
> # Remote Logging (we use TCP for reliable delivery) # # An on-disk
> queue is created for this action. If the remote host is # down,
> messages are spooled to disk and sent when it is up again.
> #$WorkDirectory /var/lib/rsyslog # where to place spool files
> #$ActionQueueFileName fwdRule1 # unique name prefix for spool files
> #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
> #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
> #$ActionQueueType LinkedList # run asynchronously
> #$ActionResumeRetryCount -1 # infinite retries if host is down
> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
> #*.* @@192.168.1.88:514
> # ### end of the forwarding rule ###
>
> ######################################################################
> ################### _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
