Ah, so quick feedback here, that class path has changed from org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
There is probably some outdated documentation somewhere, would you mind pointing out where you got that one? Casey On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore <[email protected]> wrote: > Casey, > > Thanks for the quick reply, love your work by the way! > > When I try to upload the parser I am getting a stack trace like this: > 15:43:33.182 [main-EventThread] INFO o.a.c.f.s.ConnectionStateManager - > State change: CONNECTED > java.lang.IllegalStateException: Unable to instantiate connector: class > not > found > at > org.apache.metron.common.utils.ReflectionUtils.createInstance( > ReflectionUtils.java:56) > at > org.apache.metron.parsers.topology.ParserTopologyBuilder.createParserBolt( > ParserTopologyBuilder.java:155) > at > org.apache.metron.parsers.topology.ParserTopologyBuilder.build( > ParserTopologyBuilder.java:94) > at > org.apache.metron.parsers.topology.ParserTopologyCLI. > main(ParserTopologyCLI.java:298) > Caused by: java.lang.ClassNotFoundException: > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at > org.apache.metron.common.utils.ReflectionUtils.createInstance( > ReflectionUtils.java:53) > ... 3 more > > The storm supervisor log is saying the some of the prcosses aren't > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO] > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started > and is throwing TimoutExceptions, I believe that is due to the parser. > > Without the parser though (when troubleshooting the enrichment config from > #1) I don't receive and errors from storm and the enrichment bolts seem to > be splitting the data but writer bolt emits 0 everytime. > We are able to use the built-in hostname enrichment but the custom one I > built (which will eventually be converted into asset discovery enrichment) > doesn't seem to be writing to elastic search. Do I need to setup a new > index template to receive the data from the new enrichment config? Or > should I be looking at creating a new spout / bolt to transfer the data? > > Regards, > > Tyler > > > Regards, > > Tyler Moore > Software Engineer > Flyball Labs > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella <[email protected]> wrote: > > > First off Tyler, thanks for using Metron. > > > > Do you have any errors or stack traces that are being thrown (keeping in > > mind that in storm, they may be in the storm logs (/var/log/storm on the > > supervisor nodes)? > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore <[email protected]> > wrote: > > > > > Hey everyone, > > > > > > I've had a few sticking points that I encountered while trying to > create > > > some custom solutions using the Metron platform and could use some > > > guidance. > > > > > > 1) My custom enrichment config is not writing to elasticsearch or may > be > > > configured improperly. > > > > > > My extractor config: > > > { > > > "config" : { > > > "columns" : { > > > "ip" : 0, > > > "host" : 1 > > > }, > > > "indicator_column" : "ip", > > > "type" : "hostname", > > > "separator" : "," > > > }, > > > "extractor" : "CSV" > > > } > > > > > > My enrichment config: > > > { > > > "zkQuorum" : "node1:2181", > > > "sensorToFieldList" : { > > > "bro" : { > > > "type" : "ENRICHMENT", > > > "fieldToEnrichmentTypes" : { > > > "ip_src_addr" : ["hostname"], > > > "ip_dst_addr" : ["hostname"] > > > } > > > } > > > } > > > } > > > > > > A sample of the data i'm uploading: > > > 0.0.0.0, "IGMP" > > > 10.113.145.135, "GLAZER" > > > 10.113.145.137, "GLAZER" > > > 10.113.145.138, "GLAZER" > > > > > > i'm uploading to zookeeper using the following command: > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n > > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c > > hosts > > > -e hostname_extractor_config.json > > > > > > 2) We eventually want to parse this data as a live stream but the > parser > > > errors out when I try sending data in. Here is the parser config: > > > { > > > "parserClassName" : "org.apache.metron.parsers.csv.CSVParser", > > > "writerClassName" : > > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter", > > > "sensorTopic":"hostname", > > > "parserConfig": > > > { > > > "shew.table" : "enrichment", > > > "shew.cf" : "hosts", > > > "shew.keyColumns" : "ip", > > > "shew.enrichmentType" : "hostname", > > > "columns" : { > > > "ip" : 0, > > > "host" : 1 > > > } > > > } > > > } > > > > > > 3) We will be moving from replay to using kafka-python for sending data > > > captures and I am able to send bytes to a new topic, but when I try > using > > > the json serializer via kafka producer my program exits without error > and > > > no data is sent. > > > Here is the section of the python code i'm having trouble with: > > > > > > producer = KafkaProducer(bootstrap_servers='50.253.243.17:6667', > > > value_serializer=lambda m: json.dumps(m).encode('ascii'), > api_version=(0, > > > 9)) > > > > > > for _ in range(100): > > > producer.send('pcap', {'key': 'value'}) > > > producer.flush() > > > > > > If anyone could point me in the right direction that would be great!! > I'm > > > not sure if the first 2 problems are related to indexing or maybe I > need > > to > > > create a bolt to pass on the data in storm? > > > > > > Regards, > > > > > > Tyler Moore > > > Software Engineer > > > Flyball Labs > > > > > >
