Well it looks like my celebrations for success was pre-mature. Initially when I
stated that it was working. It indexed 230 lines of logs over to kibana. Now it
isn't doing anything. I then started researching to see what the issue is and I
came across an article that said in order to have rsyslog drop logs into
elasticsearch you have to have omelasticsearch. Is this true?
>From other tutorials it shows to use omelasticsearch in the rsyslog.conf, but
>I have never configured it. Maybe someone can help me with my config. Here is
>the portion that I think should be going to elasticsearch.
########################################################################################
module(load="omelasticsearch") # for outputting to Elasticsearch
# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by
default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log
$InputFileTag apache-access:
$InputFileStateFile state-apache-access
$InputRunFileMonitor
# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log
$InputFileTag apache-error:
$InputFileStateFile state-apache-error
$InputRunFileMonitor
###############################################################################
When I look up my debug file I don't show any errors. When I look at the
rsyslogd.log this is the output.
2014-05-02T11:43:52.902217-07:00 syslogtest rsyslogd-pstats: resource-usage:
utime=4999 stime=7998 maxrss=2964 minflt=551 majflt=0 inblock=0 oublock=40
nvcsw=20 nivcsw=33
2014-05-02T11:43:52.902221-07:00 syslogtest rsyslogd-pstats: main Q: size=16
enqueued=58 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:43:52.902223-07:00 syslogtest rsyslogd-pstats: imudp(w0):
called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:22.932423-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=2 ratelimit.discarded=0 ratelimit.numratelimiters=1
2014-05-02T11:44:22.932451-07:00 syslogtest rsyslogd-pstats: action 1:
processed=60 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932456-07:00 syslogtest rsyslogd-pstats: omelasticsearch:
submitted=2 failed.http=2 failed.httprequests=2 failed.es=0
2014-05-02T11:44:22.932461-07:00 syslogtest rsyslogd-pstats: action 2:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932465-07:00 syslogtest rsyslogd-pstats: action 3:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932468-07:00 syslogtest rsyslogd-pstats: action 4:
processed=6 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932471-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932474-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932476-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932480-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932483-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932485-07:00 syslogtest rsyslogd-pstats: action 10:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932489-07:00 syslogtest rsyslogd-pstats: action 11:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932492-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:22.932495-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:22.932500-07:00 syslogtest rsyslogd-pstats: resource-usage:
utime=4999 stime=10998 maxrss=2964 minflt=567 majflt=0 inblock=0 oublock=56
nvcsw=28 nivcsw=36
2014-05-02T11:44:22.932505-07:00 syslogtest rsyslogd-pstats: main Q: size=16
enqueued=78 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:44:22.932509-07:00 syslogtest rsyslogd-pstats: imudp(w0):
called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:52.960753-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=3 ratelimit.discarded=0 ratelimit.numratelimiters=2
2014-05-02T11:44:52.960780-07:00 syslogtest rsyslogd-pstats: action 1:
processed=78 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960785-07:00 syslogtest rsyslogd-pstats: omelasticsearch:
submitted=2 failed.http=2 failed.httprequests=2 failed.es=0
2014-05-02T11:44:52.960789-07:00 syslogtest rsyslogd-pstats: action 2:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960792-07:00 syslogtest rsyslogd-pstats: action 3:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960795-07:00 syslogtest rsyslogd-pstats: action 4:
processed=7 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960799-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960801-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960804-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960807-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960810-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960813-07:00 syslogtest rsyslogd-pstats: action 10:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960816-07:00 syslogtest rsyslogd-pstats: action 11:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960819-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:52.960821-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:52.960826-07:00 syslogtest rsyslogd-pstats: resource-usage:
utime=6998 stime=12998 maxrss=2984 minflt=576 majflt=0 inblock=0 oublock=80
nvcsw=36 nivcsw=49
2014-05-02T11:44:52.960831-07:00 syslogtest rsyslogd-pstats: main Q: size=16
enqueued=97 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:44:52.960835-07:00 syslogtest rsyslogd-pstats: imudp(w0):
called.recvmmsg=0 called.recvmsg=0 msgs.received=0
--------------------------------------------------------------------------------------------------------------------------------------------------
So after reviewing the debug file for the action 2, 3, and 11; I couldn't find
any errors at all. Most of the output shows it being processed. Nothing
indicates that there was a problem. I'm stuck...
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
