Ok I think I got it to debug... I stopped the rsyslog service and then entered this at the command line. /sbin/rsyslogd -c3 -dn > /var/log/debug.log
Then it gave me a file. Should I post the entire contents? Josh -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of David Lang Sent: Tuesday, April 29, 2014 4:44 PM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server Ok, I woould guess that your elasticsarch actions are 10 and 11 (you would have to count the outputs in the file or add the name parameter to the action() calls) So this is saying that it's attempting to deliver the messages to elasticsearch but is failing. try starting in debug mode and see exactly what's happening when it tries to deliver a message. David Lang On Tue, 29 Apr 2014, Josh Bitto wrote: > Date: Tue, 29 Apr 2014 16:24:42 -0700 > From: Josh Bitto <[email protected]> > Reply-To: rsyslog-users <[email protected]> > To: rsyslog-users <[email protected]> > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > > This is the output > > Apr 29 16:23:15 syslogtest rsyslogd-pstats: imuxsock: submitted=1 > ratelimit.discarded=0 ratelimit.numratelimiters=1 Apr 29 16:23:15 > syslogtest rsyslogd-pstats: omelasticsearch: submitted=2 failed.http=2 > failed.httprequests=2 failed.es=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 1: processed=10 failed=10 suspended=1 > suspended.duration=60 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 2: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 3: processed=9 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 4: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 5: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 6: processed=1 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 7: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 8: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 9: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 10: processed=10 failed=10 suspended=1 > suspended.duration=60 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: action 11: processed=10 failed=10 suspended=1 > suspended.duration=60 resumed=0 Apr 29 16:23:15 syslogtest > rsyslogd-pstats: resource-usage: apache-error:=2999 stime=15997 > maxrss=2988 minflt=561 majflt=0 inblock=0 oublock=32 nvcsw=68 > nivcsw=31 Apr 29 16:23:15 syslogtest rsyslogd-pstats: main Q: size=14 > enqueued=24 full=0 discarded.full=0 discarded.nf=0 maxqsize=14 Apr 29 > 16:23:15 syslogtest rsyslogd-pstats: imudp(w0): called.recvmmsg=0 > called.recvmsg=0 msgs.received=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: imuxsock: submitted=0 ratelimit.discarded=0 > ratelimit.numratelimiters=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: omelasticsearch: submitted=2 failed.http=2 > failed.httprequests=2 failed.es=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 1: processed=9 failed=9 suspended=1 > suspended.duration=30 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 2: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 3: processed=9 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 4: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 5: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 6: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 7: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 8: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 9: processed=0 failed=0 suspended=0 > suspended.duration=0 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 10: processed=9 failed=9 suspended=1 > suspended.duration=30 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: action 11: processed=9 failed=9 suspended=1 > suspended.duration=30 resumed=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: imudp(*:514): submitted=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: imudp(*:514): submitted=0 Apr 29 16:23:46 syslogtest > rsyslogd-pstats: resource-usage: apache-error:=1999 stime=999 > maxrss=2916 minflt=523 majflt=0 inblock=0 oublock=16 nvcsw=10 > nivcsw=31 Apr 29 16:23:46 syslogtest rsyslogd-pstats: main Q: size=16 > enqueued=25 full=0 discarded.full=0 discarded.nf=0 maxqsize=16 Apr 29 > 16:23:46 syslogtest rsyslogd-pstats: imudp(w0): called.recvmmsg=0 > called.recvmsg=0 msgs.received=0 > > > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of David Lang > Sent: Tuesday, April 29, 2014 4:20 PM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server > > On Tue, 29 Apr 2014, Josh Bitto wrote: > >> I didn't have it running, but I added it and waiting on the 10 minute >> interval. If I set it to 300 would be go down to 5 minutes? > > Yes, for a test like this where it doesn't look like anything is > getting through, I'd suggest setting it to something really short, say > 10s so that you can debug quickly > > David Lang > >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of David Lang >> Sent: Tuesday, April 29, 2014 4:10 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >> server >> >> On Tue, 29 Apr 2014, Josh Bitto wrote: >> >>> Ok so after everyone's input I decided to go with >>> Rsyslog->Elasticsearch->Kibana setup. >>> >>> So I'm running CentOS 6.5 with apache. On a virtualbox machine. >>> Rsyslog version rsyslog-7.6.3-1.el6.x86_64 Kibana and elasticsearch >>> are the latest editions. I added a repo to just do a yum install of ES, and >>> kibana is in my webroot directory. >>> >>> Basically I've just created a test server to see how well the setup will be >>> compared to a live server and I'm running into some issues. I've followed >>> the instructions from here. >>> http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/ >>> and I added the config part in that tutorial to the rsyslog.conf. >>> >>> I've tested to make sure that rsyslog is running "logger blah blah blah" at >>> the command line and It returns in the messages logs. So I think where I'm >>> missing is from rsyslog to elasticsearch.. >>> >>> When I go to my kibana webpage and try to search for logs that I know are >>> there it doesn't return anything. >>> >>> Here is my rsyslog.conf >> >> do you have impstats running? what does it have to say about the action to >> put logs into elasticsearch? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This > is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our > control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
