On Fri, 2 May 2014, Josh Bitto wrote:
David,
Ok figure this one out...So I completely deleted all my elasticsearch coding
in the rsyslog.conf file and went back to the default rsyslog.conf file that
you get in centos.
I get no failures on any of the actions. So I clear my debug and rsyslogd.log
files to start fresh.
I stop the rsyslog service. Run debug from the command line. (/sbin/rsyslogd
-c3 -dn > /var/log/debug.log)
one thing, -c3 says to use the config language of version 3, it's not needed on
current versions (I don't remember when it was dropped, I think a lot of v5
stuff still needed at least -c4)
So it begins to debug. This continues to run so it fills my rsyslogd.log file
and kibana is picking up the logs from that file. So I start the rsyslog
service....which in turn kibana stops reading the logs.
Ok, it sounds as if there is locking going on, kibana doesn't want to read from
a file that may be modified as it's reading it. you could work around this by
rotating the file every minute. This is where omelasticsearch would be better.
So I add back in the first part of my config part for elasticsearch and the
same results happen. It still picks up the logs in the rsyslogd.log file but
nothing else. When I stop the rsyslog service kibana picks up the logs and
when I start the service it stops picking up the logs. It only accounts for
this one file. It won't read any other file.
is there a debug mode for kibana so it can tell you why it's not doing what you
expect it to?
David Lang
So I know fundamentally there is communication. Now my question is why isn't
it working when I start the rsyslog service? I've read 3-4 different tutorials
on coding the rsyslog.conf file for elasticsearch and they all have the same
configuration.
Josh
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Friday, May 02, 2014 12:07 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
action 11 is failing, unless it has a separate queue it can be causing you
significant other problems. I would suggest either commenting that out and
trying again, or looking into what is failing there.
using omelasticsearch is suggested because it results in the most direct
connection (allowing better detections of failures.
David Lang
On Fri, 2 May 2014, Josh Bitto wrote:
Well it looks like my celebrations for success was pre-mature.
Initially when I stated that it was working. It indexed 230 lines of
logs over to kibana. Now it isn't doing anything. I then started
researching to see what the issue is and I came across an article that
said in order to have rsyslog drop logs into elasticsearch you have to have
omelasticsearch. Is this true?
From other tutorials it shows to use omelasticsearch in the
rsyslog.conf, but I have never configured it. Maybe someone can help
me with my config. Here is the portion that I think should be going to
elasticsearch.
######################################################################
##################
module(load="omelasticsearch") # for outputting to Elasticsearch #
this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10") }
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch
(localhost:9200 by default) action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$ModLoad imfile # Load the imfile input module
# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log $InputFileTag apache-access:
$InputFileStateFile state-apache-access $InputRunFileMonitor
# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log $InputFileTag apache-error:
$InputFileStateFile state-apache-error $InputRunFileMonitor
######################################################################
#########
When I look up my debug file I don't show any errors. When I look at the
rsyslogd.log this is the output.
2014-05-02T11:43:52.902217-07:00 syslogtest rsyslogd-pstats:
resource-usage: utime=4999 stime=7998 maxrss=2964 minflt=551 majflt=0
inblock=0 oublock=40 nvcsw=20 nivcsw=33
2014-05-02T11:43:52.902221-07:00 syslogtest rsyslogd-pstats: main Q:
size=16 enqueued=58 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:43:52.902223-07:00 syslogtest rsyslogd-pstats:
imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:22.932423-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=2 ratelimit.discarded=0 ratelimit.numratelimiters=1
2014-05-02T11:44:22.932451-07:00 syslogtest rsyslogd-pstats: action 1:
processed=60 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932456-07:00 syslogtest rsyslogd-pstats:
omelasticsearch: submitted=2 failed.http=2 failed.httprequests=2
failed.es=0
2014-05-02T11:44:22.932461-07:00 syslogtest rsyslogd-pstats: action 2:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932465-07:00 syslogtest rsyslogd-pstats: action 3:
processed=62 failed=62 suspended=1 suspended.duration=120 resumed=0
2014-05-02T11:44:22.932468-07:00 syslogtest rsyslogd-pstats: action 4:
processed=6 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932471-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932474-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932476-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932480-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932483-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932485-07:00 syslogtest rsyslogd-pstats: action
10: processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932489-07:00 syslogtest rsyslogd-pstats: action
11: processed=62 failed=62 suspended=1 suspended.duration=120
resumed=0
2014-05-02T11:44:22.932492-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:22.932495-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:22.932500-07:00 syslogtest rsyslogd-pstats:
resource-usage: utime=4999 stime=10998 maxrss=2964 minflt=567 majflt=0
inblock=0 oublock=56 nvcsw=28 nivcsw=36
2014-05-02T11:44:22.932505-07:00 syslogtest rsyslogd-pstats: main Q:
size=16 enqueued=78 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:44:22.932509-07:00 syslogtest rsyslogd-pstats:
imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:52.960753-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=3 ratelimit.discarded=0 ratelimit.numratelimiters=2
2014-05-02T11:44:52.960780-07:00 syslogtest rsyslogd-pstats: action 1:
processed=78 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960785-07:00 syslogtest rsyslogd-pstats:
omelasticsearch: submitted=2 failed.http=2 failed.httprequests=2
failed.es=0
2014-05-02T11:44:52.960789-07:00 syslogtest rsyslogd-pstats: action 2:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960792-07:00 syslogtest rsyslogd-pstats: action 3:
processed=81 failed=81 suspended=1 suspended.duration=150 resumed=0
2014-05-02T11:44:52.960795-07:00 syslogtest rsyslogd-pstats: action 4:
processed=7 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960799-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960801-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960804-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960807-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960810-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960813-07:00 syslogtest rsyslogd-pstats: action
10: processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960816-07:00 syslogtest rsyslogd-pstats: action
11: processed=81 failed=81 suspended=1 suspended.duration=150
resumed=0
2014-05-02T11:44:52.960819-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:52.960821-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:52.960826-07:00 syslogtest rsyslogd-pstats:
resource-usage: utime=6998 stime=12998 maxrss=2984 minflt=576 majflt=0
inblock=0 oublock=80 nvcsw=36 nivcsw=49
2014-05-02T11:44:52.960831-07:00 syslogtest rsyslogd-pstats: main Q:
size=16 enqueued=97 full=0 discarded.full=0 discarded.nf=0 maxqsize=18
2014-05-02T11:44:52.960835-07:00 syslogtest rsyslogd-pstats:
imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0
----------------------------------------------------------------------
----------------------------------------------------------------------
------ So after reviewing the debug file for the action 2, 3, and 11;
I couldn't find any errors at all. Most of the output shows it being processed.
Nothing indicates that there was a problem. I'm stuck...
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
