Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server

David Lang Fri, 02 May 2014 14:59:33 -0700

On Fri, 2 May 2014, Josh Bitto wrote:

Yes you are correct it should be rsyslog+omelasticsearch -> Elasticsearch -> 
Kibana


You are correct to a point. The only logs that I see are the ones that are 
generated from the rsyslogd.log file.
Any of the other files that I would like to include don't show up.
(messages, httpd errors, other facility levels in general...etc for example)
I am using version 7.6.3
-------------------
I have this in the rsyslog.conf
syslog.* /var/log/rsyslogd.log
per Rainers suggestion.

As far as dropping privileges could you elaborate on what you are referring to? 
This is a test server on my laptop running on virtualbox. So I haven't changed 
any privileges since the creation of the VM.

ahh, you confused me when you said that it only showed logs from one file, thatimplied to me that you were trying to read from multiple files

if some logs are being delivered, but not all of them, then we need to look atthe filters that you are using for the omelasticsearch action.

could you repost your config (I'm worried that in the many quotings, things mayhave gotten wrapped wierdly)


David Lang

-----Original Message-----
From: [email protected] 
[mailto:[email protected]] On Behalf Of Radu Gheorghe
Sent: Friday, May 02, 2014 2:36 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server

Hi,

I think there's a misunderstanding here. Kibana isn't reading from the log 
files, Kibana is the Web UI that connects to Elasticsearch. So your setup
is:

rsyslog+omelasticsearch -> Elasticsearch -> Kibana

Right?

What I also understand is that in debug mode your logs are sent to 
Elasticsearch (and you can see them with Kibana). If you just start it 
normally, rsyslog doesn't send stuff to Elasticsearch.

I think I've seen this before, and I'm using CentOS, too. And you're running 
rsyslog 7.6.3? This should be recent enough. Do you drop privileges? If yes, 
can you try without dropping? What happens if you start rsyslogd without debug 
(just -n)? If that works well, how about just rsyslogd with no parameters?

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch 
Support * http://sematext.com/

On Sat, May 3, 2014 at 12:20 AM, Josh Bitto <[email protected]> wrote:

To my understanding from all the reference material that I have read
and setup. The way it is supposed to work is rsyslog sends the logs to
elasticsearch in a "logstash" format. From there kibana reads those
logs since kibana is setup to read logstash to begin with on the basic install.



-----Original Message-----
From: [email protected] [mailto:
[email protected]] On Behalf Of David Lang
Sent: Friday, May 02, 2014 2:16 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server

On Fri, 2 May 2014, Josh Bitto wrote:

I think kibana doesn't have any problem with reading the file as it
is being modified, because it would continually update when it was
refreshed. That is with rsyslog service off.


the only thing that could be different with the rsyslog service on is
that the file is held open. When you say that kibana is reading the
file when it's refreshed, is whatever is refreshing it keeping the
file open and writing to it?
or is it opening the file,  writing to the file, closing the file, at
which point kibana is reading the file.

I'm a little confused when you recommend using omelasticsearch. I
thought that module already gets loaded.


If you are having kibana read a file to get the data into
elasticsearch, you aren't using omelasticsearch

omelasticsearch does a http call to elasticsearch to send the data
directly, it doesn't write it to a file for another process to read.


David Lang



-----Original Message-----
From: [email protected] [mailto:

[email protected]] On Behalf Of David Lang

Sent: Friday, May 02, 2014 1:36 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
server

 On Fri, 2 May 2014, Josh Bitto wrote:

David,

Ok figure this one out...So I completely deleted all my
elasticsearch coding in the rsyslog.conf file and went back to the
default rsyslog.conf file that you get in centos.

I get no failures on any of the actions. So I clear my debug and
rsyslogd.log files to start fresh.

I stop the rsyslog service. Run debug from the command line.
(/sbin/rsyslogd
-c3 -dn > /var/log/debug.log)


one thing, -c3 says to use the config language of version 3, it's
not

needed on current versions (I don't remember when it was dropped, I
think a lot of v5 stuff still needed at least -c4)

So it begins to debug. This continues to run so it fills my
rsyslogd.log file and kibana is picking up the logs from that file.
So I start the rsyslog service....which in turn kibana stops
reading the

logs.


Ok, it sounds as if there is locking going on, kibana doesn't want
to

read from a file that may be modified as it's reading it. you could
work around this by rotating the file every minute. This is where
omelasticsearch would be better.

So I add back in the first part of my config part for elasticsearch
and the same results happen. It still picks up the logs in the
rsyslogd.log file but nothing else. When I stop the rsyslog service
kibana picks up the logs and when I start the service it stops
picking up the logs. It only accounts for this one file. It won't
read any

other file.


is there a debug mode for kibana so it can tell you why it's not
doing

what you expect it to?


David Lang

So I know fundamentally there is communication. Now my question is
why isn't it working when I start the rsyslog service? I've read
3-4 different tutorials on coding the rsyslog.conf file for
elasticsearch and they all have the same configuration.

Josh









-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of David Lang
Sent: Friday, May 02, 2014 12:07 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
server

action 11 is failing, unless it has a separate queue it can be
causing

you significant other problems. I would suggest either commenting that
out and trying again, or looking into what is failing there.


using omelasticsearch is suggested because it results in the most

direct connection (allowing better detections of failures.


David Lang

On Fri, 2 May 2014, Josh Bitto wrote:

Well it looks like my celebrations for success was pre-mature.
Initially when I stated that it was working. It indexed 230 lines
of logs over to kibana. Now it isn't doing anything. I then
started researching to see what the issue is and I came across an
article that said in order to have rsyslog drop logs into
elasticsearch you

have to have omelasticsearch. Is this true?


From other tutorials it shows to use omelasticsearch in the
rsyslog.conf, but I have never configured it. Maybe someone can
help me with my config. Here is the portion that I think should be
going to

elasticsearch.


##################################################################
###
#
##################

module(load="omelasticsearch") # for outputting to Elasticsearch #
this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
 type="list") {
   constant(value="logstash-")
   property(name="timereported" dateFormat="rfc3339" position.from="1"

position.to="4")

   constant(value=".")
   property(name="timereported" dateFormat="rfc3339" position.from="6"

position.to="7")

   constant(value=".")
   property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10") }

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
 type="list") {
   constant(value="{")
     constant(value="\"@timestamp\":\"")

property(name="timereported" dateFormat="rfc3339")

     constant(value="\",\"host\":\"")        property(name="hostname")
     constant(value="\",\"severity\":\"")

 property(name="syslogseverity-text")

     constant(value="\",\"facility\":\"")

 property(name="syslogfacility-text")

     constant(value="\",\"tag\":\"")   property(name="syslogtag"

format="json")

     constant(value="\",\"message\":\"")    property(name="msg"

format="json")

   constant(value="\"}")
}

# this is where we actually send the logs to Elasticsearch
(localhost:9200 by default) action(type="omelasticsearch"
   template="plain-syslog"
   searchIndex="logstash-index"
   dynSearchIndex="on")

$ModLoad imfile   # Load the imfile input module

# Watch /var/log/httpd/access_log
$InputFileName /var/log/httpd/access_log $InputFileTag apache-access:
$InputFileStateFile state-apache-access $InputRunFileMonitor

# Watch /var/log/httpd/error_log
$InputFileName /var/log/httpd/error_log $InputFileTag apache-error:
$InputFileStateFile state-apache-error $InputRunFileMonitor
##################################################################
###
#
#########

When I look up my debug file I don't show any errors. When I look
at

the rsyslogd.log this is the output.


2014-05-02T11:43:52.902217-07:00 syslogtest rsyslogd-pstats:
resource-usage: utime=4999 stime=7998 maxrss=2964 minflt=551
majflt=0
inblock=0 oublock=40 nvcsw=20 nivcsw=33
2014-05-02T11:43:52.902221-07:00 syslogtest rsyslogd-pstats: main Q:
size=16 enqueued=58 full=0 discarded.full=0 discarded.nf=0
maxqsize=18
2014-05-02T11:43:52.902223-07:00 syslogtest rsyslogd-pstats:
imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:22.932423-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=2 ratelimit.discarded=0 ratelimit.numratelimiters=1
2014-05-02T11:44:22.932451-07:00 syslogtest rsyslogd-pstats: action 1:
processed=60 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932456-07:00 syslogtest rsyslogd-pstats:
omelasticsearch: submitted=2 failed.http=2 failed.httprequests=2
failed.es=0
2014-05-02T11:44:22.932461-07:00 syslogtest rsyslogd-pstats: action 2:
processed=62 failed=62 suspended=1 suspended.duration=120
resumed=0
2014-05-02T11:44:22.932465-07:00 syslogtest rsyslogd-pstats: action 3:
processed=62 failed=62 suspended=1 suspended.duration=120
resumed=0
2014-05-02T11:44:22.932468-07:00 syslogtest rsyslogd-pstats: action 4:
processed=6 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932471-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932474-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932476-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932480-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932483-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:22.932485-07:00 syslogtest rsyslogd-pstats:
action
10: processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2014-05-02T11:44:22.932489-07:00 syslogtest rsyslogd-pstats:
action
11: processed=62 failed=62 suspended=1 suspended.duration=120
resumed=0
2014-05-02T11:44:22.932492-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:22.932495-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:22.932500-07:00 syslogtest rsyslogd-pstats:
resource-usage: utime=4999 stime=10998 maxrss=2964 minflt=567
majflt=0
inblock=0 oublock=56 nvcsw=28 nivcsw=36
2014-05-02T11:44:22.932505-07:00 syslogtest rsyslogd-pstats: main Q:
size=16 enqueued=78 full=0 discarded.full=0 discarded.nf=0
maxqsize=18
2014-05-02T11:44:22.932509-07:00 syslogtest rsyslogd-pstats:
imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0
2014-05-02T11:44:52.960753-07:00 syslogtest rsyslogd-pstats: imuxsock:
submitted=3 ratelimit.discarded=0 ratelimit.numratelimiters=2
2014-05-02T11:44:52.960780-07:00 syslogtest rsyslogd-pstats: action 1:
processed=78 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960785-07:00 syslogtest rsyslogd-pstats:
omelasticsearch: submitted=2 failed.http=2 failed.httprequests=2
failed.es=0
2014-05-02T11:44:52.960789-07:00 syslogtest rsyslogd-pstats: action 2:
processed=81 failed=81 suspended=1 suspended.duration=150
resumed=0
2014-05-02T11:44:52.960792-07:00 syslogtest rsyslogd-pstats: action 3:
processed=81 failed=81 suspended=1 suspended.duration=150
resumed=0
2014-05-02T11:44:52.960795-07:00 syslogtest rsyslogd-pstats: action 4:
processed=7 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960799-07:00 syslogtest rsyslogd-pstats: action 5:
processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960801-07:00 syslogtest rsyslogd-pstats: action 6:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960804-07:00 syslogtest rsyslogd-pstats: action 7:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960807-07:00 syslogtest rsyslogd-pstats: action 8:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960810-07:00 syslogtest rsyslogd-pstats: action 9:
processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0
2014-05-02T11:44:52.960813-07:00 syslogtest rsyslogd-pstats:
action
10: processed=0 failed=0 suspended=0 suspended.duration=0
resumed=0
2014-05-02T11:44:52.960816-07:00 syslogtest rsyslogd-pstats:
action
11: processed=81 failed=81 suspended=1 suspended.duration=150
resumed=0
2014-05-02T11:44:52.960819-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:52.960821-07:00 syslogtest rsyslogd-pstats:
imudp(*:514): submitted=0
2014-05-02T11:44:52.960826-07:00 syslogtest rsyslogd-pstats:
resource-usage: utime=6998 stime=12998 maxrss=2984 minflt=576
majflt=0
inblock=0 oublock=80 nvcsw=36 nivcsw=49
2014-05-02T11:44:52.960831-07:00 syslogtest rsyslogd-pstats: main Q:
size=16 enqueued=97 full=0 discarded.full=0 discarded.nf=0
maxqsize=18
2014-05-02T11:44:52.960835-07:00 syslogtest rsyslogd-pstats:
imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0
------------------------------------------------------------------
---
-
------------------------------------------------------------------
---
-
------ So after reviewing the debug file for the action 2, 3, and
11; I couldn't find any errors at all. Most of the output shows it
being

processed. Nothing indicates that there was a problem. I'm stuck...




_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of

sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE

WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of

sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:

This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is 
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our 
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server

Reply via email to