Haven't heard of the acronym before, i'm kinda new to the dev game :D Do you have any idea why my the enriched data isn't being written to elasticsearch?
Regards, Tyler Moore Software Engineer Flyball Labs On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella <[email protected]> wrote: > Thanks for finding that; I fixed it in the wiki. Isn't OSS awesome? ;) > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore <[email protected]> wrote: > > > No problem, > > > > I was following the Metron application tutorials in the Metron wiki: > > https://cwiki.apache.org/confluence/display/METRON/ > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment > > > > > > > > Regards, > > > > Tyler Moore > > Software Engineer > > Flyball Labs > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella <[email protected]> > wrote: > > > > > Ah, so quick feedback here, that class path has changed from > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to > > > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter > > > > > > There is probably some outdated documentation somewhere, would you mind > > > pointing out where you got that one? > > > > > > Casey > > > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore <[email protected]> > > wrote: > > > > > > > Casey, > > > > > > > > Thanks for the quick reply, love your work by the way! > > > > > > > > When I try to upload the parser I am getting a stack trace like this: > > > > 15:43:33.182 [main-EventThread] INFO o.a.c.f.s. > ConnectionStateManager > > - > > > > State change: CONNECTED > > > > java.lang.IllegalStateException: Unable to instantiate connector: > > class > > > > not > > > > found > > > > at > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance( > > > > ReflectionUtils.java:56) > > > > at > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder. > > > createParserBolt( > > > > ParserTopologyBuilder.java:155) > > > > at > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.build( > > > > ParserTopologyBuilder.java:94) > > > > at > > > > org.apache.metron.parsers.topology.ParserTopologyCLI. > > > > main(ParserTopologyCLI.java:298) > > > > Caused by: java.lang.ClassNotFoundException: > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > at java.lang.Class.forName0(Native Method) > > > > at java.lang.Class.forName(Class.java:264) > > > > at > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance( > > > > ReflectionUtils.java:53) > > > > ... 3 more > > > > > > > > The storm supervisor log is saying the some of the prcosses aren't > > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO] > > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started > > > > and is throwing TimoutExceptions, I believe that is due to the > parser. > > > > > > > > Without the parser though (when troubleshooting the enrichment config > > > from > > > > #1) I don't receive and errors from storm and the enrichment bolts > seem > > > to > > > > be splitting the data but writer bolt emits 0 everytime. > > > > We are able to use the built-in hostname enrichment but the custom > one > > I > > > > built (which will eventually be converted into asset discovery > > > enrichment) > > > > doesn't seem to be writing to elastic search. Do I need to setup a > new > > > > index template to receive the data from the new enrichment config? Or > > > > should I be looking at creating a new spout / bolt to transfer the > > data? > > > > > > > > Regards, > > > > > > > > Tyler > > > > > > > > > > > > Regards, > > > > > > > > Tyler Moore > > > > Software Engineer > > > > Flyball Labs > > > > > > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella <[email protected]> > > wrote: > > > > > > > > > First off Tyler, thanks for using Metron. > > > > > > > > > > Do you have any errors or stack traces that are being thrown > (keeping > > > in > > > > > mind that in storm, they may be in the storm logs (/var/log/storm > on > > > the > > > > > supervisor nodes)? > > > > > > > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore <[email protected] > > > > > > wrote: > > > > > > > > > > > Hey everyone, > > > > > > > > > > > > I've had a few sticking points that I encountered while trying to > > > > create > > > > > > some custom solutions using the Metron platform and could use > some > > > > > > guidance. > > > > > > > > > > > > 1) My custom enrichment config is not writing to elasticsearch or > > may > > > > be > > > > > > configured improperly. > > > > > > > > > > > > My extractor config: > > > > > > { > > > > > > "config" : { > > > > > > "columns" : { > > > > > > "ip" : 0, > > > > > > "host" : 1 > > > > > > }, > > > > > > "indicator_column" : "ip", > > > > > > "type" : "hostname", > > > > > > "separator" : "," > > > > > > }, > > > > > > "extractor" : "CSV" > > > > > > } > > > > > > > > > > > > My enrichment config: > > > > > > { > > > > > > "zkQuorum" : "node1:2181", > > > > > > "sensorToFieldList" : { > > > > > > "bro" : { > > > > > > "type" : "ENRICHMENT", > > > > > > "fieldToEnrichmentTypes" : { > > > > > > "ip_src_addr" : ["hostname"], > > > > > > "ip_dst_addr" : ["hostname"] > > > > > > } > > > > > > } > > > > > > } > > > > > > } > > > > > > > > > > > > A sample of the data i'm uploading: > > > > > > 0.0.0.0, "IGMP" > > > > > > 10.113.145.135, "GLAZER" > > > > > > 10.113.145.137, "GLAZER" > > > > > > 10.113.145.138, "GLAZER" > > > > > > > > > > > > i'm uploading to zookeeper using the following command: > > > > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n > > > > > > hostname_enrichment_config.json -i hostname_ref.csv -t > enrichment > > -c > > > > > hosts > > > > > > -e hostname_extractor_config.json > > > > > > > > > > > > 2) We eventually want to parse this data as a live stream but the > > > > parser > > > > > > errors out when I try sending data in. Here is the parser config: > > > > > > { > > > > > > "parserClassName" : "org.apache.metron.parsers.csv.CSVParser", > > > > > > "writerClassName" : > > > > > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter", > > > > > > "sensorTopic":"hostname", > > > > > > "parserConfig": > > > > > > { > > > > > > "shew.table" : "enrichment", > > > > > > "shew.cf" : "hosts", > > > > > > "shew.keyColumns" : "ip", > > > > > > "shew.enrichmentType" : "hostname", > > > > > > "columns" : { > > > > > > "ip" : 0, > > > > > > "host" : 1 > > > > > > } > > > > > > } > > > > > > } > > > > > > > > > > > > 3) We will be moving from replay to using kafka-python for > sending > > > data > > > > > > captures and I am able to send bytes to a new topic, but when I > try > > > > using > > > > > > the json serializer via kafka producer my program exits without > > error > > > > and > > > > > > no data is sent. > > > > > > Here is the section of the python code i'm having trouble with: > > > > > > > > > > > > producer = KafkaProducer(bootstrap_servers='50.253.243.17:6667', > > > > > > value_serializer=lambda m: json.dumps(m).encode('ascii'), > > > > api_version=(0, > > > > > > 9)) > > > > > > > > > > > > for _ in range(100): > > > > > > producer.send('pcap', {'key': 'value'}) > > > > > > producer.flush() > > > > > > > > > > > > If anyone could point me in the right direction that would be > > great!! > > > > I'm > > > > > > not sure if the first 2 problems are related to indexing or > maybe I > > > > need > > > > > to > > > > > > create a bolt to pass on the data in storm? > > > > > > > > > > > > Regards, > > > > > > > > > > > > Tyler Moore > > > > > > Software Engineer > > > > > > Flyball Labs > > > > > > > > > > > > > > > > > > > > >
