Casey, Thanks for the quick reply, love your work by the way!
When I try to upload the parser I am getting a stack trace like this: 15:43:33.182 [main-EventThread] INFO o.a.c.f.s.ConnectionStateManager - State change: CONNECTED java.lang.IllegalStateException: Unable to instantiate connector: class not found at org.apache.metron.common.utils.ReflectionUtils.createInstance(ReflectionUtils.java:56) at org.apache.metron.parsers.topology.ParserTopologyBuilder.createParserBolt(ParserTopologyBuilder.java:155) at org.apache.metron.parsers.topology.ParserTopologyBuilder.build(ParserTopologyBuilder.java:94) at org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:298) Caused by: java.lang.ClassNotFoundException: org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.metron.common.utils.ReflectionUtils.createInstance(ReflectionUtils.java:53) ... 3 more The storm supervisor log is saying the some of the prcosses aren't starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO] 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started and is throwing TimoutExceptions, I believe that is due to the parser. Without the parser though (when troubleshooting the enrichment config from #1) I don't receive and errors from storm and the enrichment bolts seem to be splitting the data but writer bolt emits 0 everytime. We are able to use the built-in hostname enrichment but the custom one I built (which will eventually be converted into asset discovery enrichment) doesn't seem to be writing to elastic search. Do I need to setup a new index template to receive the data from the new enrichment config? Or should I be looking at creating a new spout / bolt to transfer the data? Regards, Tyler Regards, Tyler Moore Software Engineer Flyball Labs On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella <[email protected]> wrote: > First off Tyler, thanks for using Metron. > > Do you have any errors or stack traces that are being thrown (keeping in > mind that in storm, they may be in the storm logs (/var/log/storm on the > supervisor nodes)? > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore <[email protected]> wrote: > > > Hey everyone, > > > > I've had a few sticking points that I encountered while trying to create > > some custom solutions using the Metron platform and could use some > > guidance. > > > > 1) My custom enrichment config is not writing to elasticsearch or may be > > configured improperly. > > > > My extractor config: > > { > > "config" : { > > "columns" : { > > "ip" : 0, > > "host" : 1 > > }, > > "indicator_column" : "ip", > > "type" : "hostname", > > "separator" : "," > > }, > > "extractor" : "CSV" > > } > > > > My enrichment config: > > { > > "zkQuorum" : "node1:2181", > > "sensorToFieldList" : { > > "bro" : { > > "type" : "ENRICHMENT", > > "fieldToEnrichmentTypes" : { > > "ip_src_addr" : ["hostname"], > > "ip_dst_addr" : ["hostname"] > > } > > } > > } > > } > > > > A sample of the data i'm uploading: > > 0.0.0.0, "IGMP" > > 10.113.145.135, "GLAZER" > > 10.113.145.137, "GLAZER" > > 10.113.145.138, "GLAZER" > > > > i'm uploading to zookeeper using the following command: > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c > hosts > > -e hostname_extractor_config.json > > > > 2) We eventually want to parse this data as a live stream but the parser > > errors out when I try sending data in. Here is the parser config: > > { > > "parserClassName" : "org.apache.metron.parsers.csv.CSVParser", > > "writerClassName" : > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter", > > "sensorTopic":"hostname", > > "parserConfig": > > { > > "shew.table" : "enrichment", > > "shew.cf" : "hosts", > > "shew.keyColumns" : "ip", > > "shew.enrichmentType" : "hostname", > > "columns" : { > > "ip" : 0, > > "host" : 1 > > } > > } > > } > > > > 3) We will be moving from replay to using kafka-python for sending data > > captures and I am able to send bytes to a new topic, but when I try using > > the json serializer via kafka producer my program exits without error and > > no data is sent. > > Here is the section of the python code i'm having trouble with: > > > > producer = KafkaProducer(bootstrap_servers='50.253.243.17:6667', > > value_serializer=lambda m: json.dumps(m).encode('ascii'), api_version=(0, > > 9)) > > > > for _ in range(100): > > producer.send('pcap', {'key': 'value'}) > > producer.flush() > > > > If anyone could point me in the right direction that would be great!! I'm > > not sure if the first 2 problems are related to indexing or maybe I need > to > > create a bolt to pass on the data in storm? > > > > Regards, > > > > Tyler Moore > > Software Engineer > > Flyball Labs > > >
