I think that makes sense. Just so you know, however, we are seeing this same error in the worker.log file as well.
We are not seeing anything in the enrichments topic, so I am fairly certain the issue is with the parser topology. That's why we've tried to narrow in on any issues showing up in the logs for that particular topology. On Tue, Sep 11, 2018 at 9:27 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Does that error show up every time you restart the parser topology? I'm > not sure why you would get that error. I believe at least as far as HDP is > concerned, Ganglia hasn't been used in quite some time. Ambari Metrics > Server is the approach that is now used. Before we get into the weeds, I > would set that error aside for now. Generally when we're dealing with > exceptions in our topologies, we're looking for exceptions in worker.log, > meanwhile this particular issue is showing up in worker.log.err. Not to say > we should completely ignore it, but I suspect that may be a red herring. > It's possible changing the logging levels brought that out, but I can't say > for sure. > > Per @Casey Stella <ceste...@gmail.com> 's comments, were you able to > verify data is making its way to the enrichments Kafka topic? This is where > I'd start digging if I were you. > > To help your search, the typical dataflow in Metron looks like the > following: > > parser_name (e.g. 'bro') (kafka) -> parser (storm) -> enrichments (kafka) > -> enrichment topology (storm) -> indexing (kafka topic) -> > random_access_indexing & batch_indexing topology (storm) -> HDFS and > indexing engine (ES or Solr - in 0.4.3 you get ES only) > > - your parser topology polls from "parser" topic and writes to > "enrichments" topic > - enrichment topology polls from "enrichments" topic and writes to > "indexing" topic > - random_access/batch_indexing topologies poll from "indexing" and > write to ES/Solr and HDFS, respectively > > You've already validated data exists in your parser topic, but that's the > beginning of the chain. Validate data shows up in enrichments. If not, > there's an error in your parser. If it does, check topic indexing topic... > You get the idea. You're looking for the kink in your hose. > > Best, > Mike Miklavcic > > > On Tue, Sep 11, 2018 at 7:23 PM David McGinnis < > mcginn...@avalonconsult.com> wrote: > >> No, I don’t believe so. My understanding is that we installed Metron >> 0.4.3 from scratch onto an HDP 2.6.3 cluster, and we don’t know of any >> major changes to the cluster since then. I’m reading this to mean that we >> don’t have Ganglia available at all. We don’t have Ganglia installed on the >> cluster, but no where does it say we have to, so this is just confusing. >> >> >> On Tue, Sep 11, 2018 at 5:19 PM James Sirota <jsir...@apache.org> wrote: >> >>> Did you mix and match versions of Metron? Looks like you are having a >>> classpath issue here >>> >>> >>> 11.09.2018, 12:01, "David McGinnis" <mcginn...@avalonconsult.com>: >>> >>> Sorry for the delay. We are finding a new error in the worker.log.err >>> file for the parser Storm topology. It appears that it is trying to log to >>> Ganglia and failing. We don't have Ganglia installed on our cluster, and >>> would rather not install it, so is there some way to turn off logging to >>> Ganglia, or a minimal installation path we can do to get rid of this error? >>> Are we correct in assuming this error would be preventing data from getting >>> through the parser? >>> >>> FATAL ERROR in native method: processing of -javaagent failed >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> info/ganglia/gmetric4j/GMonitor >>> at java.lang.ClassLoader.defineClass1(Native Method) >>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763) >>> at >>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) >>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> at >>> sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:304) >>> at >>> sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) >>> Caused by: java.lang.ClassNotFoundException: >>> info.ganglia.gmetric4j.GMonitor >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> ... 14 more >>> >>> >>> On Fri, Aug 17, 2018 at 10:21 AM David McGinnis < >>> mcginn...@avalonconsult.com> wrote: >>> >>> Thank you all for the quick recommendations. We'll follow up on these >>> recommendations and report back. In the meantime, I can say that we don't >>> have an error index even created in Elastic, so there's nothing in there of >>> value. >>> >>> On Fri, Aug 17, 2018 at 10:13 AM, Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>> It might be worth looking in the error and invalid topics, if you have >>> any validation, or your parser is not producing proper timestamps (that's >>> what I usually forget to check!) you may be getting messages routed to the >>> error index. Are you indexing topologies picking any of this up? Is there >>> anything in the ES error index? >>> >>> The other thing worth doing is upgrading 0.4.3 is a very very old >>> version and a lot of changes have come in since then. >>> >>> Simon >>> >>> On 17 August 2018 at 16:07, David McGinnis <mcginn...@avalonconsult.com> >>> wrote: >>> >>> All, >>> >>> We have a Metron 0.4.3 installation on an HDP cluster which has a sensor >>> set up to read from a Kafka topic and write the data out to Elasticsearch. >>> Data is being inserted into the Kafka topic, and we can read that through >>> Kafka console consumer, but the system is not reporting any data coming >>> through. The Storm spout says no data has been processed, and the index >>> hasn't even been created in Elastic, despite running for nearly a month >>> now. >>> >>> We've searched the worker logs for Storm, and the only error that comes >>> up is a (we think) unrelated error about not being able to find the >>> jmxmetrics JAR file. Metron reports that the topic is found, and does not >>> tell us that the topic is not emitting, so we suspect it sees the data in >>> there. >>> >>> Do you all have any ideas on where we can look to determine the cause of >>> this issue, or things to try? >>> >>> Thanks! >>> >>> -- >>> David McGinnis >>> Staff Hadoop Consultant | Avalon Consulting, LLC >>> <http://www.avalonconsult.com/>M: (513) 439-0082 >>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | >>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter >>> <https://twitter.com/avalonconsult> >>> >>> ------------------------------------------------------------------------------------------------------------- >>> This message (including any attachments) contains confidential >>> information >>> intended for a specific individual and purpose, and is protected by law. >>> If >>> you are not the intended recipient, you should delete this message. Any >>> disclosure, copying, or distribution of this message, or the taking of >>> any >>> action based on it, is strictly prohibited. >>> >>> >>> >>> >>> -- >>> -- >>> simon elliston ball >>> @sireb >>> >>> >>> >>> >>> -- >>> David McGinnis >>> Staff Hadoop Consultant | Avalon Consulting, LLC >>> <http://www.avalonconsult.com/>M: (513) 439-0082 >>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | >>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter >>> <https://twitter.com/avalonconsult> >>> >>> ------------------------------------------------------------------------------------------------------------- >>> This message (including any attachments) contains confidential >>> information >>> intended for a specific individual and purpose, and is protected by law. >>> If >>> you are not the intended recipient, you should delete this message. Any >>> disclosure, copying, or distribution of this message, or the taking of >>> any >>> action based on it, is strictly prohibited. >>> >>> >>> -- >>> David McGinnis >>> Staff Hadoop Consultant | Avalon Consulting, LLC >>> <http://www.avalonconsult.com/>M: (513) 439-0082 >>> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | >>> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter >>> <https://twitter.com/avalonconsult> >>> >>> ------------------------------------------------------------------------------------------------------------- >>> This message (including any attachments) contains confidential >>> information >>> intended for a specific individual and purpose, and is protected by law. >>> If >>> you are not the intended recipient, you should delete this message. Any >>> disclosure, copying, or distribution of this message, or the taking of >>> any >>> action based on it, is strictly prohibited. >>> >>> >>> >>> ------------------- >>> Thank you, >>> >>> James Sirota >>> PMC- Apache Metron >>> jsirota AT apache DOT org >>> >>> >> >> -- >> David McGinnis >> Staff Hadoop Consultant | Avalon Consulting, LLC >> <http://www.avalonconsult.com/>M: (513) 439-0082 >> LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | >> Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter >> <https://twitter.com/avalonconsult> >> >> ------------------------------------------------------------------------------------------------------------- >> This message (including any attachments) contains confidential >> information >> intended for a specific individual and purpose, and is protected by law. >> If >> you are not the intended recipient, you should delete this message. Any >> disclosure, copying, or distribution of this message, or the taking of >> any >> action based on it, is strictly prohibited. >> > -- David McGinnis Staff Hadoop Consultant | Avalon Consulting, LLC <http://www.avalonconsult.com/>M: (513) 439-0082 LinkedIn <http://www.linkedin.com/company/avalon-consulting-llc> | Google+ <http://www.google.com/+AvalonConsultingLLC> | Twitter <https://twitter.com/avalonconsult> ------------------------------------------------------------------------------------------------------------- This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.
