Re: Metron Not Reading From Kafka?

2018-09-19 Thread David McGinnis
-- > *From:* Simon Elliston Ball > *Sent:* Wednesday, September 12, 2018 7:06:54 AM > *To:* user@metron.apache.org > *Cc:* Ben Stoker > *Subject:* Re: Metron Not Reading From Kafka? > > That's a pretty out dated version, and sounds like you may have an old

Re: Metron Not Reading From Kafka?

2018-09-19 Thread James Sirota
6:54 AM To: user@metron.apache.org Cc: Ben Stoker Subject: Re: Metron Not Reading From Kafka?   That's a pretty out dated version, and sounds like you may have an old version of HDP if you have the jmx error.  From the info here, there are a lot of things it could potentially be. I would suggest

Re: Metron Not Reading From Kafka?

2018-09-14 Thread Michael Miklavcic
Sent:* Wednesday, September 12, 2018 7:06:54 AM > *To:* user@metron.apache.org > *Cc:* Ben Stoker > *Subject:* Re: Metron Not Reading From Kafka? > > That's a pretty out dated version, and sounds like you may have an old > version of HDP if you have the jmx error. > > From the inf

Re: Metron Not Reading From Kafka?

2018-09-14 Thread David McGinnis
-dropping-immediately-a.html -- David From: Simon Elliston Ball Sent: Wednesday, September 12, 2018 7:06:54 AM To: user@metron.apache.org Cc: Ben Stoker Subject: Re: Metron Not Reading From Kafka? That's a pretty out dated version, and sounds like you may have an old

Re: Metron Not Reading From Kafka?

2018-09-11 Thread David McGinnis
It's a CSV parser made to parse firewall logs. On Tue, Sep 11, 2018 at 10:25 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > What parser is this? > > On Tue, Sep 11, 2018, 9:20 PM David McGinnis > wrote: > >> I think that makes sense. Just so you know, however, we are seeing this

Re: Metron Not Reading From Kafka?

2018-09-11 Thread Michael Miklavcic
What parser is this? On Tue, Sep 11, 2018, 9:20 PM David McGinnis wrote: > I think that makes sense. Just so you know, however, we are seeing this > same error in the worker.log file as well. > > We are not seeing anything in the enrichments topic, so I am fairly > certain the issue is with the

Re: Metron Not Reading From Kafka?

2018-09-11 Thread David McGinnis
I think that makes sense. Just so you know, however, we are seeing this same error in the worker.log file as well. We are not seeing anything in the enrichments topic, so I am fairly certain the issue is with the parser topology. That's why we've tried to narrow in on any issues showing up in the

Re: Metron Not Reading From Kafka?

2018-09-11 Thread Michael Miklavcic
Does that error show up every time you restart the parser topology? I'm not sure why you would get that error. I believe at least as far as HDP is concerned, Ganglia hasn't been used in quite some time. Ambari Metrics Server is the approach that is now used. Before we get into the weeds, I would

Re: Metron Not Reading From Kafka?

2018-09-11 Thread David McGinnis
No, I don’t believe so. My understanding is that we installed Metron 0.4.3 from scratch onto an HDP 2.6.3 cluster, and we don’t know of any major changes to the cluster since then. I’m reading this to mean that we don’t have Ganglia available at all. We don’t have Ganglia installed on the cluster,

Re: Metron Not Reading From Kafka?

2018-09-11 Thread James Sirota
Did you mix and match versions of Metron?  Looks like you are having a classpath issue here11.09.2018, 12:01, "David McGinnis" :Sorry for the delay. We are finding a new error in the worker.log.err file for the parser Storm topology. It appears that it is trying to log to Ganglia and failing. We

Re: Metron Not Reading From Kafka?

2018-09-11 Thread David McGinnis
Sorry for the delay. We are finding a new error in the worker.log.err file for the parser Storm topology. It appears that it is trying to log to Ganglia and failing. We don't have Ganglia installed on our cluster, and would rather not install it, so is there some way to turn off logging to

Re: Metron Not Reading From Kafka?

2018-08-17 Thread David McGinnis
Thank you all for the quick recommendations. We'll follow up on these recommendations and report back. In the meantime, I can say that we don't have an error index even created in Elastic, so there's nothing in there of value. On Fri, Aug 17, 2018 at 10:13 AM, Simon Elliston Ball <

Re: Metron Not Reading From Kafka?

2018-08-17 Thread Simon Elliston Ball
It might be worth looking in the error and invalid topics, if you have any validation, or your parser is not producing proper timestamps (that's what I usually forget to check!) you may be getting messages routed to the error index. Are you indexing topologies picking any of this up? Is there

Metron Not Reading From Kafka?

2018-08-17 Thread David McGinnis
All, We have a Metron 0.4.3 installation on an HDP cluster which has a sensor set up to read from a Kafka topic and write the data out to Elasticsearch. Data is being inserted into the Kafka topic, and we can read that through Kafka console consumer, but the system is not reporting any data