That is interesting! We'll definitely need to narrow it down a bit. I'm currently running off 7.2.5. I haven't sat down and pulled in the 7.4.x branch yet as I've been focusing on implementing the Rsyslog to Elasticsearch infrastructure. I'm agree you were probably in the process of correcting how the stats were being generated and weren't able to finish it. The fact that it isn't incrementing failed at all tends to lend to that fact since it appears it is the failed counter that is incrementing in an unexpected way.
Thankfully the way it is implemented in 7.2.5 does at least offer the ability to recognize when it is failing. You just have to account for the difference between the submits and fails. Whatever you have left when you subtract submit from fail is the number of failures. --James -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Thursday, August 22, 2013 6:12 AM To: rsyslog-users Subject: Re: [rsyslog] Elasticsearch and impstats reading question On Tue, Aug 20, 2013 at 6:34 PM, Boylan, James <[email protected]>wrote: > Rainer - > > I would double check. It looks like even when successful it increments > both failed and submit. But it does an extra failed increment when > there is an error. (This should be confirmed since I'm commenting > based on observed behavior. > > First of all, the current state really looks like I either did experimental work or was dragged away right in the middle of some implementation. I reviewed current git master branch and have to admit that what I see makes limited sense (if it were complete). Anyhow, I have documented the current state here: http://www.rsyslog.com/rsyslog-statistic-counter/ The interesting thing is that the current code does not even increment "failed" at all... So which version are you working with? Let's see if we can clean up the mess ;) Rainer > -- James > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Boylan, James > Sent: Friday, August 16, 2013 6:06 AM > To: rsyslog-users > Subject: Re: [rsyslog] Elasticsearch and impstats reading question > > That's exactly what I needed. Thanks Rainer. > > -- James T. Boylan > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Rainer Gerhards > Sent: Thursday, August 15, 2013 9:16 PM > To: rsyslog-users > Subject: Re: [rsyslog] Elasticsearch and impstats reading question > > On Thu, Aug 15, 2013 at 5:57 PM, Radu Gheorghe > <[email protected] > >wrote: > > > Hello, > > > > Disclaimer: I've never worked with impstats, only with Elaseticsearch. > > But it sounds like I should, because it provides nice info. > > > > The fact that you have success=0 and failed=$A_LOT sounds like > > something isn't working at all. So I'd start rsyslog in debug mode > > and see what the reply from Elasticsearch looks like. Feel free to > > post it here if you need additional help. For me, 9 times out of 10 > > it's an invalid JSON :) > > > > connfail says 0, so I guess it connects OK. But I can't find any > > documentation on impstats for elasticsearch here: > > http://www.rsyslog.com/rsyslog-statistic-counter/ > > > > > looks like this was a recent addition where I either forgot or did not > manage to update the counters. > > > > I'd gladly do some testing and contribute documentation to this (I'm > > interested in these counters). Can anyone give some hints? I've > > looked in omelasticsearch.c and I see the counters there (submit, > > connfail, success, failed), but I don't understand how they're gathered. > > > > > had a quick look as well, it's essentially straightforward: > > connfail - connection to ES failed > submit - successful submit of record > failed - some error occured (Debug log will show more detail) success > -- seems currently unused (need to check when I am back). > > Rainer > > > Best regards, > > Radu > > > > > > 2013/8/15 Boylan, James <[email protected]> > > > > > Hey Everyone! > > > > > > I'm trying to better understand what I'm seeing in the output of > > > the impstats module. Could someone help confirm that I'm seeing > > > that Elasticsearch is failing to receive a large portion of the > > > messages being sent at it. > > > > > > 2013-07-02T14:27:26.298686-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: main Q: size=44400 > > enqueued=1010337 full=0 > > > discarded.full=0 discarded.nf=0 maxqsize=45886 > > > 2013-07-02T14:27:26.298642-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: elasticsearch: > > connfail=0 > > submits=4763 failed=10538 > > > success=0 > > > 2013-07-02T14:27:26.298647-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 1: > > processed=11913 > > failed=0 > > > 2013-07-02T14:27:27.299756-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 2: > > processed=11913 > > failed=0 > > > 2013-07-02T14:27:27.299758-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 3: > > processed=955943 > > failed=0 > > > 2013-07-02T14:27:27.299761-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 4: > > processed=955874 > > failed=0 > > > 2013-07-02T14:27:27.299764-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 5: > > processed=967728 > > failed=0 > > > 2013-07-02T14:27:27.299772-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 7: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299775-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 8: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299777-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 9: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299779-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 10: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299781-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 11: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299783-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 12: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299785-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 13: processed=0 > > failed=0 > > > 2013-07-02T14:27:27.299789-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/22516/IPv4): > > submitted=0 > > > 2013-07-02T14:27:27.299791-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/22516/IPv6): > > submitted=0 > > > 2013-07-02T14:27:27.299794-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/21516/IPv4): > > submitted=1000681 > > > 2013-07-02T14:27:27.299795-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: imptcp(*/21516/IPv6): > > submitted=0 > > > 2013-07-02T14:27:27.299799-05:00 > > wmlogserv01s.stag.orbitz.netrsyslogd-pstats: action 4 queue: > > size=65536 > > enqueued=955874 full=72 > > > discarded.full=0 discarded.nf=0 maxqsize=65536 > > > > > > Thanks! > > > > > > --James > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > > POST if you DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > you DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
