On Tue, 29 Apr 2014, Josh Bitto wrote:
Ok so after everyone's input I decided to go with
Rsyslog->Elasticsearch->Kibana setup.
So I'm running CentOS 6.5 with apache. On a virtualbox machine.
Rsyslog version rsyslog-7.6.3-1.el6.x86_64
Kibana and elasticsearch are the latest editions. I added a repo to just do a
yum install of ES, and kibana is in my webroot directory.
Basically I've just created a test server to see how well the setup will be
compared to a live server and I'm running into some issues. I've followed the
instructions from here.
http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/ and I
added the config part in that tutorial to the rsyslog.conf.
I've tested to make sure that rsyslog is running "logger blah blah blah" at the
command line and It returns in the messages logs. So I think where I'm missing is from
rsyslog to elasticsearch..
When I go to my kibana webpage and try to search for logs that I know are there
it doesn't return anything.
Here is my rsyslog.conf
do you have impstats running? what does it have to say about the action to put
logs into elasticsearch?
David Lang
# rsyslog configuration file
# note that most of this config file uses old-style format,
# because it is well-known AND quite suitable for simple cases
# like we have with the default config. For more advanced
# things, RainerScript configuration is suggested.
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
#module(load="imuxsock") # provides support for local system logging (e.g. via
logger command)
module(load="imklog") # provides kernel logging support (previously done by
rklogd)
#module(load"immark") # provides --MARK-- message capability
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by
default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
$ModLoad imklog # for reading kernel log messages
$ModLoad imuxsock # for reading local syslog messages
# Watch /var/log/apache2/access.log
$InputFileName /var/log/apache2/access.log
$InputFileTag apache-access:
$InputFileStateFile state-apache-access
$InputRunFileMonitor
# Watch /var/log/apache2/error.log
$InputFileName /var/log/apache2/error.log
$InputFileTag apache-error:
$InputFileStateFile state-apache-error
$InputRunFileMonitor
# Watch /var/log/mysql/mysql.log
$InputFileName /var/log/mysql/mysql.log
$InputFileTag mysql:
$InputFileStateFile state-mysql
$InputRunFileMonito
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@192.168.1.88:514
# ### end of the forwarding rule ###
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
