No problem, I was following the Metron application tutorials in the Metron wiki: https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment
Regards, Tyler Moore Software Engineer Flyball Labs On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella <[email protected]> wrote: > Ah, so quick feedback here, that class path has changed from > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter > > There is probably some outdated documentation somewhere, would you mind > pointing out where you got that one? > > Casey > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore <[email protected]> wrote: > > > Casey, > > > > Thanks for the quick reply, love your work by the way! > > > > When I try to upload the parser I am getting a stack trace like this: > > 15:43:33.182 [main-EventThread] INFO o.a.c.f.s.ConnectionStateManager - > > State change: CONNECTED > > java.lang.IllegalStateException: Unable to instantiate connector: class > > not > > found > > at > > org.apache.metron.common.utils.ReflectionUtils.createInstance( > > ReflectionUtils.java:56) > > at > > org.apache.metron.parsers.topology.ParserTopologyBuilder. > createParserBolt( > > ParserTopologyBuilder.java:155) > > at > > org.apache.metron.parsers.topology.ParserTopologyBuilder.build( > > ParserTopologyBuilder.java:94) > > at > > org.apache.metron.parsers.topology.ParserTopologyCLI. > > main(ParserTopologyCLI.java:298) > > Caused by: java.lang.ClassNotFoundException: > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:264) > > at > > org.apache.metron.common.utils.ReflectionUtils.createInstance( > > ReflectionUtils.java:53) > > ... 3 more > > > > The storm supervisor log is saying the some of the prcosses aren't > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO] > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started > > and is throwing TimoutExceptions, I believe that is due to the parser. > > > > Without the parser though (when troubleshooting the enrichment config > from > > #1) I don't receive and errors from storm and the enrichment bolts seem > to > > be splitting the data but writer bolt emits 0 everytime. > > We are able to use the built-in hostname enrichment but the custom one I > > built (which will eventually be converted into asset discovery > enrichment) > > doesn't seem to be writing to elastic search. Do I need to setup a new > > index template to receive the data from the new enrichment config? Or > > should I be looking at creating a new spout / bolt to transfer the data? > > > > Regards, > > > > Tyler > > > > > > Regards, > > > > Tyler Moore > > Software Engineer > > Flyball Labs > > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella <[email protected]> wrote: > > > > > First off Tyler, thanks for using Metron. > > > > > > Do you have any errors or stack traces that are being thrown (keeping > in > > > mind that in storm, they may be in the storm logs (/var/log/storm on > the > > > supervisor nodes)? > > > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore <[email protected]> > > wrote: > > > > > > > Hey everyone, > > > > > > > > I've had a few sticking points that I encountered while trying to > > create > > > > some custom solutions using the Metron platform and could use some > > > > guidance. > > > > > > > > 1) My custom enrichment config is not writing to elasticsearch or may > > be > > > > configured improperly. > > > > > > > > My extractor config: > > > > { > > > > "config" : { > > > > "columns" : { > > > > "ip" : 0, > > > > "host" : 1 > > > > }, > > > > "indicator_column" : "ip", > > > > "type" : "hostname", > > > > "separator" : "," > > > > }, > > > > "extractor" : "CSV" > > > > } > > > > > > > > My enrichment config: > > > > { > > > > "zkQuorum" : "node1:2181", > > > > "sensorToFieldList" : { > > > > "bro" : { > > > > "type" : "ENRICHMENT", > > > > "fieldToEnrichmentTypes" : { > > > > "ip_src_addr" : ["hostname"], > > > > "ip_dst_addr" : ["hostname"] > > > > } > > > > } > > > > } > > > > } > > > > > > > > A sample of the data i'm uploading: > > > > 0.0.0.0, "IGMP" > > > > 10.113.145.135, "GLAZER" > > > > 10.113.145.137, "GLAZER" > > > > 10.113.145.138, "GLAZER" > > > > > > > > i'm uploading to zookeeper using the following command: > > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n > > > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c > > > hosts > > > > -e hostname_extractor_config.json > > > > > > > > 2) We eventually want to parse this data as a live stream but the > > parser > > > > errors out when I try sending data in. Here is the parser config: > > > > { > > > > "parserClassName" : "org.apache.metron.parsers.csv.CSVParser", > > > > "writerClassName" : > > > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter", > > > > "sensorTopic":"hostname", > > > > "parserConfig": > > > > { > > > > "shew.table" : "enrichment", > > > > "shew.cf" : "hosts", > > > > "shew.keyColumns" : "ip", > > > > "shew.enrichmentType" : "hostname", > > > > "columns" : { > > > > "ip" : 0, > > > > "host" : 1 > > > > } > > > > } > > > > } > > > > > > > > 3) We will be moving from replay to using kafka-python for sending > data > > > > captures and I am able to send bytes to a new topic, but when I try > > using > > > > the json serializer via kafka producer my program exits without error > > and > > > > no data is sent. > > > > Here is the section of the python code i'm having trouble with: > > > > > > > > producer = KafkaProducer(bootstrap_servers='50.253.243.17:6667', > > > > value_serializer=lambda m: json.dumps(m).encode('ascii'), > > api_version=(0, > > > > 9)) > > > > > > > > for _ in range(100): > > > > producer.send('pcap', {'key': 'value'}) > > > > producer.flush() > > > > > > > > If anyone could point me in the right direction that would be great!! > > I'm > > > > not sure if the first 2 problems are related to indexing or maybe I > > need > > > to > > > > create a bolt to pass on the data in storm? > > > > > > > > Regards, > > > > > > > > Tyler Moore > > > > Software Engineer > > > > Flyball Labs > > > > > > > > > >
