action 11 is failing, unless it has a separate queue it can be causing you
significant other problems. I would suggest either commenting that out and
trying again, or looking into what is failing there.
using omelasticsearch is suggested because it results in the most direct
connection (allowing better detections of failures.
David Lang
On Fri, 2 May 2014, Josh Bitto wrote:
Well it looks like my celebrations for success was pre-mature. Initially when
I stated that it was working. It indexed 230 lines of logs over to kibana. Now
it isn't doing anything. I then started researching to see what the issue is
and I came across an article that said in order to have rsyslog drop logs into
elasticsearch you have to have omelasticsearch. Is this true?
From other tutorials it shows to use omelasticsearch in the rsyslog.conf, but
I have never configured it. Maybe someone can help me with my config. Here is
the portion that I think should be going to elasticsearch.
########################################################################################
module(load="omelasticsearch") # for outputting to Elasticsearch
# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by
default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log
$InputFileTag apache-access:
$InputFileStateFile state-apache-access
$InputRunFileMonitor
# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log
$InputFileTag apache-error:
$InputFileStateFile state-apache-error
$InputRunFileMonitor
###############################################################################
When I look up my debug file I don't show any errors. When I look at the
rsyslogd.log this is the output.
2014-05-02T11:43:52.902217-07:00 syslogtest rsyslogd-pstats: resource-usage:
utime=4999 stime=7998 maxrss=2964 minflt=551 majflt=0 inblock=0 oublock=40
nvcsw=20 nivcsw=33
2014-05-02T11:43:52.902221-07:00 syslogtest rsyslogd-pstats: main Q: size=16
enqueued=58 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:43:52.902223-07:00 syslogtest rsyslogd-pstats: imudp(w0):
called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:22.932423-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=2 ratelimit.discarded=0 ratelimit.numratelimiters=1
2014-05-02T11:44:22.932451-07:00 syslogtest rsyslogd-pstats: action 1:
processed=60 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932456-07:00 syslogtest rsyslogd-pstats: omelasticsearch:
submitted=2 failed.http=2 failed.httprequests=2 failed.es=0
2014-05-02T11:44:22.932461-07:00 syslogtest rsyslogd-pstats: action 2:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932465-07:00 syslogtest rsyslogd-pstats: action 3:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932468-07:00 syslogtest rsyslogd-pstats: action 4:
processed=6 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932471-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932474-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932476-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932480-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932483-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932485-07:00 syslogtest rsyslogd-pstats: action 10:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932489-07:00 syslogtest rsyslogd-pstats: action 11:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932492-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:22.932495-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:22.932500-07:00 syslogtest rsyslogd-pstats: resource-usage:
utime=4999 stime=10998 maxrss=2964 minflt=567 majflt=0 inblock=0 oublock=56
nvcsw=28 nivcsw=36
2014-05-02T11:44:22.932505-07:00 syslogtest rsyslogd-pstats: main Q: size=16
enqueued=78 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:44:22.932509-07:00 syslogtest rsyslogd-pstats: imudp(w0):
called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:52.960753-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=3 ratelimit.discarded=0 ratelimit.numratelimiters=2
2014-05-02T11:44:52.960780-07:00 syslogtest rsyslogd-pstats: action 1:
processed=78 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960785-07:00 syslogtest rsyslogd-pstats: omelasticsearch:
submitted=2 failed.http=2 failed.httprequests=2 failed.es=0
2014-05-02T11:44:52.960789-07:00 syslogtest rsyslogd-pstats: action 2:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960792-07:00 syslogtest rsyslogd-pstats: action 3:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960795-07:00 syslogtest rsyslogd-pstats: action 4:
processed=7 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960799-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960801-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960804-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960807-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960810-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960813-07:00 syslogtest rsyslogd-pstats: action 10:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960816-07:00 syslogtest rsyslogd-pstats: action 11:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960819-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:52.960821-07:00 syslogtest rsyslogd-pstats: imudp(*:514):
submitted=0
2014-05-02T11:44:52.960826-07:00 syslogtest rsyslogd-pstats: resource-usage:
utime=6998 stime=12998 maxrss=2984 minflt=576 majflt=0 inblock=0 oublock=80
nvcsw=36 nivcsw=49
2014-05-02T11:44:52.960831-07:00 syslogtest rsyslogd-pstats: main Q: size=16
enqueued=97 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:44:52.960835-07:00 syslogtest rsyslogd-pstats: imudp(w0):
called.recvmmsg=0 called.recvmsg=0 msgs.received=0
--------------------------------------------------------------------------------------------------------------------------------------------------
So after reviewing the debug file for the action 2, 3, and 11; I couldn't find
any errors at all. Most of the output shows it being processed. Nothing
indicates that there was a problem. I'm stuck...
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
