One question: how are failed messages counted? Is there any distinction between a transient error (like a shard not being available - I don't have a clearer example in mind) and a permanent one (like a malformed JSON)? Or are all such logs marked as failed and retried indefinitely?
I assume connection errors (timeouts or connection refused) are treated separately (the connfail counter), right? I'm asking this because: a) I'd like to test it properly b) I'd like to update the documentation on impstats while I'm there Best regards, Radu 2013/8/26 Radu Gheorghe <[email protected]> > Hi Rainer, > > Unless someone else jumps in, I'm willing to do it. I'm hesitant because > I'm squeezed with time in this period. But it shouldn't take long, so I'll > find some time this week. Just let me know when the patch is available and > where should I apply it (dev/stable). > > Best regards, > Radu > > > 2013/8/26 Rainer Gerhards <[email protected]> > >> Would someone be willing to test counters (including fail cases) when I >> improve the instrumentation? (I want to save the time to setup a test lab >> just for this case...). >> >> Rainer >> >> On Thu, Aug 22, 2013 at 1:49 PM, Boylan, James <[email protected] >> >wrote: >> >> > That is interesting! We'll definitely need to narrow it down a bit. >> > >> > I'm currently running off 7.2.5. I haven't sat down and pulled in the >> > 7.4.x branch yet as I've been focusing on implementing the Rsyslog to >> > Elasticsearch infrastructure. I'm agree you were probably in the >> process of >> > correcting how the stats were being generated and weren't able to finish >> > it. The fact that it isn't incrementing failed at all tends to lend to >> that >> > fact since it appears it is the failed counter that is incrementing in >> an >> > unexpected way. >> > >> > Thankfully the way it is implemented in 7.2.5 does at least offer the >> > ability to recognize when it is failing. You just have to account for >> the >> > difference between the submits and fails. Whatever you have left when >> you >> > subtract submit from fail is the number of failures. >> > >> > --James >> > >> > >> > -----Original Message----- >> > From: [email protected] [mailto: >> > [email protected]] On Behalf Of Rainer Gerhards >> > Sent: Thursday, August 22, 2013 6:12 AM >> > To: rsyslog-users >> > Subject: Re: [rsyslog] Elasticsearch and impstats reading question >> > >> > On Tue, Aug 20, 2013 at 6:34 PM, Boylan, James <[email protected] >> > >wrote: >> > >> > > Rainer - >> > > >> > > I would double check. It looks like even when successful it increments >> > > both failed and submit. But it does an extra failed increment when >> > > there is an error. (This should be confirmed since I'm commenting >> > > based on observed behavior. >> > > >> > > >> > First of all, the current state really looks like I either did >> > experimental work or was dragged away right in the middle of some >> > implementation. I reviewed current git master branch and have to admit >> that >> > what I see makes limited sense (if it were complete). Anyhow, I have >> > documented the current state here: >> > >> > http://www.rsyslog.com/rsyslog-statistic-counter/ >> > >> > The interesting thing is that the current code does not even increment >> > "failed" at all... So which version are you working with? >> > >> > Let's see if we can clean up the mess ;) >> > >> > Rainer >> > >> > >> > > -- James >> > > >> > > -----Original Message----- >> > > From: [email protected] [mailto: >> > > [email protected]] On Behalf Of Boylan, James >> > > Sent: Friday, August 16, 2013 6:06 AM >> > > To: rsyslog-users >> > > Subject: Re: [rsyslog] Elasticsearch and impstats reading question >> > > >> > > That's exactly what I needed. Thanks Rainer. >> > > >> > > -- James T. Boylan >> > > >> > > -----Original Message----- >> > > From: [email protected] [mailto: >> > > [email protected]] On Behalf Of Rainer Gerhards >> > > Sent: Thursday, August 15, 2013 9:16 PM >> > > To: rsyslog-users >> > > Subject: Re: [rsyslog] Elasticsearch and impstats reading question >> > > >> > > On Thu, Aug 15, 2013 at 5:57 PM, Radu Gheorghe >> > > <[email protected] >> > > >wrote: >> > > >> > > > Hello, >> > > > >> > > > Disclaimer: I've never worked with impstats, only with >> Elaseticsearch. >> > > > But it sounds like I should, because it provides nice info. >> > > > >> > > > The fact that you have success=0 and failed=$A_LOT sounds like >> > > > something isn't working at all. So I'd start rsyslog in debug mode >> > > > and see what the reply from Elasticsearch looks like. Feel free to >> > > > post it here if you need additional help. For me, 9 times out of 10 >> > > > it's an invalid JSON :) >> > > > >> > > > connfail says 0, so I guess it connects OK. But I can't find any >> > > > documentation on impstats for elasticsearch here: >> > > > http://www.rsyslog.com/rsyslog-statistic-counter/ >> > > > >> > > > >> > > looks like this was a recent addition where I either forgot or did not >> > > manage to update the counters. >> > > >> > > >> > > > I'd gladly do some testing and contribute documentation to this (I'm >> > > > interested in these counters). Can anyone give some hints? I've >> > > > looked in omelasticsearch.c and I see the counters there (submit, >> > > > connfail, success, failed), but I don't understand how they're >> > gathered. >> > > > >> > > > >> > > had a quick look as well, it's essentially straightforward: >> > > >> > > connfail - connection to ES failed >> > > submit - successful submit of record >> > > failed - some error occured (Debug log will show more detail) success >> > > -- seems currently unused (need to check when I am back). >> > > >> > > Rainer >> > > >> > > > Best regards, >> > > > Radu >> > > > >> > > > >> > > > 2013/8/15 Boylan, James <[email protected]> >> > > > >> > > > > Hey Everyone! >> > > > > >> > > > > I'm trying to better understand what I'm seeing in the output of >> > > > > the impstats module. Could someone help confirm that I'm seeing >> > > > > that Elasticsearch is failing to receive a large portion of the >> > > > > messages being sent at it. >> > > > > >> > > > > 2013-07-02T14:27:26.298686-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: main Q: size=44400 >> > > > enqueued=1010337 full=0 >> > > > > discarded.full=0 discarded.nf=0 maxqsize=45886 >> > > > > 2013-07-02T14:27:26.298642-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: elasticsearch: >> > > > connfail=0 >> > > > submits=4763 failed=10538 >> > > > > success=0 >> > > > > 2013-07-02T14:27:26.298647-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 1: >> > > > processed=11913 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299756-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 2: >> > > > processed=11913 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299758-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 3: >> > > > processed=955943 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299761-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 4: >> > > > processed=955874 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299764-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 5: >> > > > processed=967728 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299772-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 7: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299775-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 8: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299777-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 9: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299779-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 10: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299781-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 11: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299783-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 12: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299785-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 13: processed=0 >> > > > failed=0 >> > > > > 2013-07-02T14:27:27.299789-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/22516/IPv4): >> > > > submitted=0 >> > > > > 2013-07-02T14:27:27.299791-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/22516/IPv6): >> > > > submitted=0 >> > > > > 2013-07-02T14:27:27.299794-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/21516/IPv4): >> > > > submitted=1000681 >> > > > > 2013-07-02T14:27:27.299795-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/21516/IPv6): >> > > > submitted=0 >> > > > > 2013-07-02T14:27:27.299799-05:00 >> > > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 4 queue: >> > > > size=65536 >> > > > enqueued=955874 full=72 >> > > > > discarded.full=0 discarded.nf=0 maxqsize=65536 >> > > > > >> > > > > Thanks! >> > > > > >> > > > > --James >> > > > > >> > > > > _______________________________________________ >> > > > > rsyslog mailing list >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > http://www.rsyslog.com/professional-services/ >> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> > > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> > > > > POST if you DON'T LIKE THAT. >> > > > > >> > > > _______________________________________________ >> > > > rsyslog mailing list >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > http://www.rsyslog.com/professional-services/ >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> > > > you DON'T LIKE THAT. >> > > > >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: >> > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> > > LIKE THAT. >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: >> > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> > > LIKE THAT. >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > > DON'T LIKE THAT. >> > > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >> > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites >> > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >> > THAT. >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
