Not sure about the python-kafka lib issues. Regarding enrichment data getting written to ES, how many records have you processed and what is your batch size? You might need to write more records or adjust this for the values to propagate through. See the "Sensor Enrichment Configuration" section - https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-enrichment
On Thu, Nov 3, 2016 at 1:03 PM, Tyler Moore <[email protected]> wrote: > Mike, > > I am using quick-dev vagrant deployment and at the moment testing locally > but we plan on having data from remote locations streaming in to be parsed. > I was able to get the parsers running, thanks to casey, looks like i missed > an update to the Hbase enrichment writer naming convention. > Still working on the enrichment configs though, they aren't throwing any > errors and storm says they are emitting data, but not being written to > elastic. > As well with the python-kafka library, can't figure out why the json > serializer isn't working, as long as I have a parser implemented I could > forego serializing the data > prior to sending to a kafka topic correct?? > > Thanks for all your help thus far! > > Regards, > > Tyler > > Regards, > > Tyler Moore > Software Engineer > Flyball Labs > > On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic < > [email protected]> wrote: > > > Tyler, > > > > Thanks for the interest in Metron and welcome to the community! :) > > > > Just curious, what type of environment are you running in? Full cluster > or > > are you using the full-dev or quick-dev vagrant deployment vagrant > scripts? > > > > Best, > > Mike Miklavcic > > > > > > On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore <[email protected]> > wrote: > > > > > Haven't heard of the acronym before, i'm kinda new to the dev game :D > > > > > > Do you have any idea why my the enriched data isn't being written to > > > elasticsearch? > > > > > > Regards, > > > > > > Tyler Moore > > > Software Engineer > > > Flyball Labs > > > > > > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella <[email protected]> > > wrote: > > > > > > > Thanks for finding that; I fixed it in the wiki. Isn't OSS awesome? > ;) > > > > > > > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore <[email protected]> > > > wrote: > > > > > > > > > No problem, > > > > > > > > > > I was following the Metron application tutorials in the Metron > wiki: > > > > > https://cwiki.apache.org/confluence/display/METRON/ > > > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+ > > > Streaming+Enrichment > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > Tyler Moore > > > > > Software Engineer > > > > > Flyball Labs > > > > > > > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella <[email protected]> > > > > wrote: > > > > > > > > > > > Ah, so quick feedback here, that class path has changed from > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to > > > > > > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter > > > > > > > > > > > > There is probably some outdated documentation somewhere, would > you > > > mind > > > > > > pointing out where you got that one? > > > > > > > > > > > > Casey > > > > > > > > > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Casey, > > > > > > > > > > > > > > Thanks for the quick reply, love your work by the way! > > > > > > > > > > > > > > When I try to upload the parser I am getting a stack trace like > > > this: > > > > > > > 15:43:33.182 [main-EventThread] INFO o.a.c.f.s. > > > > ConnectionStateManager > > > > > - > > > > > > > State change: CONNECTED > > > > > > > java.lang.IllegalStateException: Unable to instantiate > > connector: > > > > > class > > > > > > > not > > > > > > > found > > > > > > > at > > > > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance( > > > > > > > ReflectionUtils.java:56) > > > > > > > at > > > > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder. > > > > > > createParserBolt( > > > > > > > ParserTopologyBuilder.java:155) > > > > > > > at > > > > > > > org.apache.metron.parsers.topology. > ParserTopologyBuilder.build( > > > > > > > ParserTopologyBuilder.java:94) > > > > > > > at > > > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI. > > > > > > > main(ParserTopologyCLI.java:298) > > > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter > > > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass( > Launcher.java:331) > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > > > > at java.lang.Class.forName0(Native Method) > > > > > > > at java.lang.Class.forName(Class.java:264) > > > > > > > at > > > > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance( > > > > > > > ReflectionUtils.java:53) > > > > > > > ... 3 more > > > > > > > > > > > > > > The storm supervisor log is saying the some of the prcosses > > aren't > > > > > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO] > > > > > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started > > > > > > > and is throwing TimoutExceptions, I believe that is due to the > > > > parser. > > > > > > > > > > > > > > Without the parser though (when troubleshooting the enrichment > > > config > > > > > > from > > > > > > > #1) I don't receive and errors from storm and the enrichment > > bolts > > > > seem > > > > > > to > > > > > > > be splitting the data but writer bolt emits 0 everytime. > > > > > > > We are able to use the built-in hostname enrichment but the > > custom > > > > one > > > > > I > > > > > > > built (which will eventually be converted into asset discovery > > > > > > enrichment) > > > > > > > doesn't seem to be writing to elastic search. Do I need to > setup > > a > > > > new > > > > > > > index template to receive the data from the new enrichment > > config? > > > Or > > > > > > > should I be looking at creating a new spout / bolt to transfer > > the > > > > > data? > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Tyler > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Tyler Moore > > > > > > > Software Engineer > > > > > > > Flyball Labs > > > > > > > > > > > > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > First off Tyler, thanks for using Metron. > > > > > > > > > > > > > > > > Do you have any errors or stack traces that are being thrown > > > > (keeping > > > > > > in > > > > > > > > mind that in storm, they may be in the storm logs > > (/var/log/storm > > > > on > > > > > > the > > > > > > > > supervisor nodes)? > > > > > > > > > > > > > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hey everyone, > > > > > > > > > > > > > > > > > > I've had a few sticking points that I encountered while > > trying > > > to > > > > > > > create > > > > > > > > > some custom solutions using the Metron platform and could > use > > > > some > > > > > > > > > guidance. > > > > > > > > > > > > > > > > > > 1) My custom enrichment config is not writing to > > elasticsearch > > > or > > > > > may > > > > > > > be > > > > > > > > > configured improperly. > > > > > > > > > > > > > > > > > > My extractor config: > > > > > > > > > { > > > > > > > > > "config" : { > > > > > > > > > "columns" : { > > > > > > > > > "ip" : 0, > > > > > > > > > "host" : 1 > > > > > > > > > }, > > > > > > > > > "indicator_column" : "ip", > > > > > > > > > "type" : "hostname", > > > > > > > > > "separator" : "," > > > > > > > > > }, > > > > > > > > > "extractor" : "CSV" > > > > > > > > > } > > > > > > > > > > > > > > > > > > My enrichment config: > > > > > > > > > { > > > > > > > > > "zkQuorum" : "node1:2181", > > > > > > > > > "sensorToFieldList" : { > > > > > > > > > "bro" : { > > > > > > > > > "type" : "ENRICHMENT", > > > > > > > > > "fieldToEnrichmentTypes" : { > > > > > > > > > "ip_src_addr" : ["hostname"], > > > > > > > > > "ip_dst_addr" : ["hostname"] > > > > > > > > > } > > > > > > > > > } > > > > > > > > > } > > > > > > > > > } > > > > > > > > > > > > > > > > > > A sample of the data i'm uploading: > > > > > > > > > 0.0.0.0, "IGMP" > > > > > > > > > 10.113.145.135, "GLAZER" > > > > > > > > > 10.113.145.137, "GLAZER" > > > > > > > > > 10.113.145.138, "GLAZER" > > > > > > > > > > > > > > > > > > i'm uploading to zookeeper using the following command: > > > > > > > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n > > > > > > > > > hostname_enrichment_config.json -i hostname_ref.csv -t > > > > enrichment > > > > > -c > > > > > > > > hosts > > > > > > > > > -e hostname_extractor_config.json > > > > > > > > > > > > > > > > > > 2) We eventually want to parse this data as a live stream > but > > > the > > > > > > > parser > > > > > > > > > errors out when I try sending data in. Here is the parser > > > config: > > > > > > > > > { > > > > > > > > > "parserClassName" : "org.apache.metron.parsers. > > > csv.CSVParser", > > > > > > > > > "writerClassName" : > > > > > > > > > "org.apache.metron.writer.hbase. > > SimpleHbaseEnrichmentWriter", > > > > > > > > > "sensorTopic":"hostname", > > > > > > > > > "parserConfig": > > > > > > > > > { > > > > > > > > > "shew.table" : "enrichment", > > > > > > > > > "shew.cf" : "hosts", > > > > > > > > > "shew.keyColumns" : "ip", > > > > > > > > > "shew.enrichmentType" : "hostname", > > > > > > > > > "columns" : { > > > > > > > > > "ip" : 0, > > > > > > > > > "host" : 1 > > > > > > > > > } > > > > > > > > > } > > > > > > > > > } > > > > > > > > > > > > > > > > > > 3) We will be moving from replay to using kafka-python for > > > > sending > > > > > > data > > > > > > > > > captures and I am able to send bytes to a new topic, but > > when I > > > > try > > > > > > > using > > > > > > > > > the json serializer via kafka producer my program exits > > without > > > > > error > > > > > > > and > > > > > > > > > no data is sent. > > > > > > > > > Here is the section of the python code i'm having trouble > > with: > > > > > > > > > > > > > > > > > > producer = KafkaProducer(bootstrap_servers=' > > 50.253.243.17:6667 > > > ', > > > > > > > > > value_serializer=lambda m: json.dumps(m).encode('ascii'), > > > > > > > api_version=(0, > > > > > > > > > 9)) > > > > > > > > > > > > > > > > > > for _ in range(100): > > > > > > > > > producer.send('pcap', {'key': 'value'}) > > > > > > > > > producer.flush() > > > > > > > > > > > > > > > > > > If anyone could point me in the right direction that would > be > > > > > great!! > > > > > > > I'm > > > > > > > > > not sure if the first 2 problems are related to indexing or > > > > maybe I > > > > > > > need > > > > > > > > to > > > > > > > > > create a bolt to pass on the data in storm? > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Tyler Moore > > > > > > > > > Software Engineer > > > > > > > > > Flyball Labs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
