Yes you are correct it should be rsyslog+omelasticsearch -> Elasticsearch -> Kibana
You are correct to a point. The only logs that I see are the ones that are generated from the rsyslogd.log file. Any of the other files that I would like to include don't show up. (messages, httpd errors, other facility levels in general...etc for example) I am using version 7.6.3 ------------------- I have this in the rsyslog.conf syslog.* /var/log/rsyslogd.log per Rainers suggestion. As far as dropping privileges could you elaborate on what you are referring to? This is a test server on my laptop running on virtualbox. So I haven't changed any privileges since the creation of the VM. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Radu Gheorghe Sent: Friday, May 02, 2014 2:36 PM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server Hi, I think there's a misunderstanding here. Kibana isn't reading from the log files, Kibana is the Web UI that connects to Elasticsearch. So your setup is: rsyslog+omelasticsearch -> Elasticsearch -> Kibana Right? What I also understand is that in debug mode your logs are sent to Elasticsearch (and you can see them with Kibana). If you just start it normally, rsyslog doesn't send stuff to Elasticsearch. I think I've seen this before, and I'm using CentOS, too. And you're running rsyslog 7.6.3? This should be recent enough. Do you drop privileges? If yes, can you try without dropping? What happens if you start rsyslogd without debug (just -n)? If that works well, how about just rsyslogd with no parameters? Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, May 3, 2014 at 12:20 AM, Josh Bitto <[email protected]> wrote: > To my understanding from all the reference material that I have read > and setup. The way it is supposed to work is rsyslog sends the logs to > elasticsearch in a "logstash" format. From there kibana reads those > logs since kibana is setup to read logstash to begin with on the basic > install. > > > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of David Lang > Sent: Friday, May 02, 2014 2:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > > On Fri, 2 May 2014, Josh Bitto wrote: > > > I think kibana doesn't have any problem with reading the file as it > > is being modified, because it would continually update when it was > > refreshed. That is with rsyslog service off. > > the only thing that could be different with the rsyslog service on is > that the file is held open. When you say that kibana is reading the > file when it's refreshed, is whatever is refreshing it keeping the > file open and writing to it? > or is it opening the file, writing to the file, closing the file, at > which point kibana is reading the file. > > > I'm a little confused when you recommend using omelasticsearch. I > > thought that module already gets loaded. > > If you are having kibana read a file to get the data into > elasticsearch, you aren't using omelasticsearch > > omelasticsearch does a http call to elasticsearch to send the data > directly, it doesn't write it to a file for another process to read. > > > David Lang > > > > > > -----Original Message----- > > From: [email protected] [mailto: > [email protected]] On Behalf Of David Lang > > Sent: Friday, May 02, 2014 1:36 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > > server > > > > On Fri, 2 May 2014, Josh Bitto wrote: > > > >> David, > >> > >> Ok figure this one out...So I completely deleted all my > >> elasticsearch coding in the rsyslog.conf file and went back to the > >> default rsyslog.conf file that you get in centos. > >> > >> I get no failures on any of the actions. So I clear my debug and > >> rsyslogd.log files to start fresh. > >> > >> I stop the rsyslog service. Run debug from the command line. > >> (/sbin/rsyslogd > >> -c3 -dn > /var/log/debug.log) > > > > one thing, -c3 says to use the config language of version 3, it's > > not > needed on current versions (I don't remember when it was dropped, I > think a lot of v5 stuff still needed at least -c4) > > > >> So it begins to debug. This continues to run so it fills my > >> rsyslogd.log file and kibana is picking up the logs from that file. > >> So I start the rsyslog service....which in turn kibana stops > >> reading the > logs. > > > > Ok, it sounds as if there is locking going on, kibana doesn't want > > to > read from a file that may be modified as it's reading it. you could > work around this by rotating the file every minute. This is where > omelasticsearch would be better. > > > >> So I add back in the first part of my config part for elasticsearch > >> and the same results happen. It still picks up the logs in the > >> rsyslogd.log file but nothing else. When I stop the rsyslog service > >> kibana picks up the logs and when I start the service it stops > >> picking up the logs. It only accounts for this one file. It won't > >> read any > other file. > > > > is there a debug mode for kibana so it can tell you why it's not > > doing > what you expect it to? > > > > David Lang > > > >> So I know fundamentally there is communication. Now my question is > >> why isn't it working when I start the rsyslog service? I've read > >> 3-4 different tutorials on coding the rsyslog.conf file for > >> elasticsearch and they all have the same configuration. > > > >> Josh > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> -----Original Message----- > >> From: [email protected] > >> [mailto:[email protected]] On Behalf Of David Lang > >> Sent: Friday, May 02, 2014 12:07 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > >> server > >> > >> action 11 is failing, unless it has a separate queue it can be > >> causing > you significant other problems. I would suggest either commenting that > out and trying again, or looking into what is failing there. > >> > >> using omelasticsearch is suggested because it results in the most > direct connection (allowing better detections of failures. > >> > >> David Lang > >> > >> On Fri, 2 May 2014, Josh Bitto wrote: > >> > >>> Well it looks like my celebrations for success was pre-mature. > >>> Initially when I stated that it was working. It indexed 230 lines > >>> of logs over to kibana. Now it isn't doing anything. I then > >>> started researching to see what the issue is and I came across an > >>> article that said in order to have rsyslog drop logs into > >>> elasticsearch you > have to have omelasticsearch. Is this true? > >>> > >>> From other tutorials it shows to use omelasticsearch in the > >>> rsyslog.conf, but I have never configured it. Maybe someone can > >>> help me with my config. Here is the portion that I think should be > >>> going to > elasticsearch. > >>> > >>> ################################################################## > >>> ### > >>> # > >>> ################## > >>> > >>> module(load="omelasticsearch") # for outputting to Elasticsearch # > >>> this is for index names to be like: logstash-YYYY.MM.DD > >>> template(name="logstash-index" > >>> type="list") { > >>> constant(value="logstash-") > >>> property(name="timereported" dateFormat="rfc3339" position.from="1" > position.to="4") > >>> constant(value=".") > >>> property(name="timereported" dateFormat="rfc3339" position.from="6" > position.to="7") > >>> constant(value=".") > >>> property(name="timereported" dateFormat="rfc3339" position.from="9" > >>> position.to="10") } > >>> > >>> # this is for formatting our syslog in JSON with @timestamp > >>> template(name="plain-syslog" > >>> type="list") { > >>> constant(value="{") > >>> constant(value="\"@timestamp\":\"") > property(name="timereported" dateFormat="rfc3339") > >>> constant(value="\",\"host\":\"") property(name="hostname") > >>> constant(value="\",\"severity\":\"") > property(name="syslogseverity-text") > >>> constant(value="\",\"facility\":\"") > property(name="syslogfacility-text") > >>> constant(value="\",\"tag\":\"") property(name="syslogtag" > format="json") > >>> constant(value="\",\"message\":\"") property(name="msg" > format="json") > >>> constant(value="\"}") > >>> } > >>> > >>> # this is where we actually send the logs to Elasticsearch > >>> (localhost:9200 by default) action(type="omelasticsearch" > >>> template="plain-syslog" > >>> searchIndex="logstash-index" > >>> dynSearchIndex="on") > >>> > >>> $ModLoad imfile # Load the imfile input module > >>> > >>> # Watch /var/log/httpd/access_log > >>> $InputFileName /var/log/httpd/access_log $InputFileTag apache-access: > >>> $InputFileStateFile state-apache-access $InputRunFileMonitor > >>> > >>> # Watch /var/log/httpd/error_log > >>> $InputFileName /var/log/httpd/error_log $InputFileTag apache-error: > >>> $InputFileStateFile state-apache-error $InputRunFileMonitor > >>> ################################################################## > >>> ### > >>> # > >>> ######### > >>> > >>> When I look up my debug file I don't show any errors. When I look > >>> at > the rsyslogd.log this is the output. > >>> > >>> 2014-05-02T11:43:52.902217-07:00 syslogtest rsyslogd-pstats: > >>> resource-usage: utime=4999 stime=7998 maxrss=2964 minflt=551 > >>> majflt=0 > >>> inblock=0 oublock=40 nvcsw=20 nivcsw=33 > >>> 2014-05-02T11:43:52.902221-07:00 syslogtest rsyslogd-pstats: main Q: > >>> size=16 enqueued=58 full=0 discarded.full=0 discarded.nf=0 > >>> maxqsize=18 > >>> 2014-05-02T11:43:52.902223-07:00 syslogtest rsyslogd-pstats: > >>> imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0 > >>> 2014-05-02T11:44:22.932423-07:00 syslogtest rsyslogd-pstats: imuxsock: > >>> submitted=2 ratelimit.discarded=0 ratelimit.numratelimiters=1 > >>> 2014-05-02T11:44:22.932451-07:00 syslogtest rsyslogd-pstats: action 1: > >>> processed=60 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932456-07:00 syslogtest rsyslogd-pstats: > >>> omelasticsearch: submitted=2 failed.http=2 failed.httprequests=2 > >>> failed.es=0 > >>> 2014-05-02T11:44:22.932461-07:00 syslogtest rsyslogd-pstats: action 2: > >>> processed=62 failed=62 suspended=1 suspended.duration=120 > >>> resumed=0 > >>> 2014-05-02T11:44:22.932465-07:00 syslogtest rsyslogd-pstats: action 3: > >>> processed=62 failed=62 suspended=1 suspended.duration=120 > >>> resumed=0 > >>> 2014-05-02T11:44:22.932468-07:00 syslogtest rsyslogd-pstats: action 4: > >>> processed=6 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932471-07:00 syslogtest rsyslogd-pstats: action 5: > >>> processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932474-07:00 syslogtest rsyslogd-pstats: action 6: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932476-07:00 syslogtest rsyslogd-pstats: action 7: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932480-07:00 syslogtest rsyslogd-pstats: action 8: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932483-07:00 syslogtest rsyslogd-pstats: action 9: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:22.932485-07:00 syslogtest rsyslogd-pstats: > >>> action > >>> 10: processed=0 failed=0 suspended=0 suspended.duration=0 > >>> resumed=0 > >>> 2014-05-02T11:44:22.932489-07:00 syslogtest rsyslogd-pstats: > >>> action > >>> 11: processed=62 failed=62 suspended=1 suspended.duration=120 > >>> resumed=0 > >>> 2014-05-02T11:44:22.932492-07:00 syslogtest rsyslogd-pstats: > >>> imudp(*:514): submitted=0 > >>> 2014-05-02T11:44:22.932495-07:00 syslogtest rsyslogd-pstats: > >>> imudp(*:514): submitted=0 > >>> 2014-05-02T11:44:22.932500-07:00 syslogtest rsyslogd-pstats: > >>> resource-usage: utime=4999 stime=10998 maxrss=2964 minflt=567 > >>> majflt=0 > >>> inblock=0 oublock=56 nvcsw=28 nivcsw=36 > >>> 2014-05-02T11:44:22.932505-07:00 syslogtest rsyslogd-pstats: main Q: > >>> size=16 enqueued=78 full=0 discarded.full=0 discarded.nf=0 > >>> maxqsize=18 > >>> 2014-05-02T11:44:22.932509-07:00 syslogtest rsyslogd-pstats: > >>> imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0 > >>> 2014-05-02T11:44:52.960753-07:00 syslogtest rsyslogd-pstats: imuxsock: > >>> submitted=3 ratelimit.discarded=0 ratelimit.numratelimiters=2 > >>> 2014-05-02T11:44:52.960780-07:00 syslogtest rsyslogd-pstats: action 1: > >>> processed=78 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960785-07:00 syslogtest rsyslogd-pstats: > >>> omelasticsearch: submitted=2 failed.http=2 failed.httprequests=2 > >>> failed.es=0 > >>> 2014-05-02T11:44:52.960789-07:00 syslogtest rsyslogd-pstats: action 2: > >>> processed=81 failed=81 suspended=1 suspended.duration=150 > >>> resumed=0 > >>> 2014-05-02T11:44:52.960792-07:00 syslogtest rsyslogd-pstats: action 3: > >>> processed=81 failed=81 suspended=1 suspended.duration=150 > >>> resumed=0 > >>> 2014-05-02T11:44:52.960795-07:00 syslogtest rsyslogd-pstats: action 4: > >>> processed=7 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960799-07:00 syslogtest rsyslogd-pstats: action 5: > >>> processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960801-07:00 syslogtest rsyslogd-pstats: action 6: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960804-07:00 syslogtest rsyslogd-pstats: action 7: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960807-07:00 syslogtest rsyslogd-pstats: action 8: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960810-07:00 syslogtest rsyslogd-pstats: action 9: > >>> processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 > >>> 2014-05-02T11:44:52.960813-07:00 syslogtest rsyslogd-pstats: > >>> action > >>> 10: processed=0 failed=0 suspended=0 suspended.duration=0 > >>> resumed=0 > >>> 2014-05-02T11:44:52.960816-07:00 syslogtest rsyslogd-pstats: > >>> action > >>> 11: processed=81 failed=81 suspended=1 suspended.duration=150 > >>> resumed=0 > >>> 2014-05-02T11:44:52.960819-07:00 syslogtest rsyslogd-pstats: > >>> imudp(*:514): submitted=0 > >>> 2014-05-02T11:44:52.960821-07:00 syslogtest rsyslogd-pstats: > >>> imudp(*:514): submitted=0 > >>> 2014-05-02T11:44:52.960826-07:00 syslogtest rsyslogd-pstats: > >>> resource-usage: utime=6998 stime=12998 maxrss=2984 minflt=576 > >>> majflt=0 > >>> inblock=0 oublock=80 nvcsw=36 nivcsw=49 > >>> 2014-05-02T11:44:52.960831-07:00 syslogtest rsyslogd-pstats: main Q: > >>> size=16 enqueued=97 full=0 discarded.full=0 discarded.nf=0 > >>> maxqsize=18 > >>> 2014-05-02T11:44:52.960835-07:00 syslogtest rsyslogd-pstats: > >>> imudp(w0): called.recvmmsg=0 called.recvmsg=0 msgs.received=0 > >>> ------------------------------------------------------------------ > >>> --- > >>> - > >>> ------------------------------------------------------------------ > >>> --- > >>> - > >>> ------ So after reviewing the debug file for the action 2, 3, and > >>> 11; I couldn't find any errors at all. Most of the output shows it > >>> being > processed. Nothing indicates that there was a problem. I'm stuck... > >>> > >>> > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
