Michael, Thanks for the reply, and to everyone helping y'all rock! I am getting data through the "assets" kafka spout, which is the spout for my parser, but I am not seeing anything coming through indexing spout. I didn't see any logs for the storm supervisor but I attached the enrichment log named exceptions.txt Seems like it is not connecting to the socket properly? I recently started getting an out of memory error form the Hbase enrichment bolt which may clear up why some tuples are being acked and some aren't. The log is attached as well. Also included is the log for my parser topology named asset_lot.txt not sure if that will help or not. The indexing topology doesn't seem to be working properly after I load my parser into zookeeper and it is not starting its executors or producing logs logs are located here: https://drive.google.com/drive/folders/0B0i9txEWVH4DNmRSS0pWckgzeTQ?usp=sharing
Regards, Tyler Regards, Tyler Moore Software Engineer Flyball Labs On Fri, Nov 4, 2016 at 3:28 PM, Michael Miklavcic < [email protected]> wrote: > Can you check for any exceptions in the enrichment logs using the following > grep? > grep --color=auto -C 3 -R -iE "exception" /var/log/storm > > It would also be good to know where the data is getting hung up. Can you > check if you're getting tuples transferring and acking through the indexing > Kafka spout? > > On Thu, Nov 3, 2016 at 3:41 PM, Tyler Moore <[email protected]> wrote: > > > The sample i'm sending has over logs about 40,000 records so I don't > think > > that is the issue. > > > > My batch size is 5 and the this is what it looks like when i dump it from > > zookeeper > > ENRICHMENT Config: bro > > { > > "index" : "bro", > > "batchSize" : 5, > > "enrichment" : { > > "fieldMap" : { > > "geo" : [ "ip_dst_addr", "ip_src_addr" ], > > "host" : [ "ip_src_addr", "ip_dst_addr" ], > > "hbaseEnrichment" : [ "ip_src_addr", "ip_dst_addr" ] > > }, > > "fieldToTypeMap" : { > > "ip_dst_addr" : [ "hostname", "asset" ], > > "ip_src_addr" : [ "hostname", "asset" ] > > }, > > "config" : { } > > }, > > "threatIntel" : { > > "fieldMap" : { > > "hbaseThreatIntel" : [ "ip_src_addr", "ip_dst_addr" ] > > }, > > "fieldToTypeMap" : { > > "ip_src_addr" : [ "malicious_ip" ], > > "ip_dst_addr" : [ "malicious_ip" ] > > }, > > "config" : { }, > > "triageConfig" : { > > "riskLevelRules" : { }, > > "aggregator" : "MAX", > > "aggregationConfig" : { } > > } > > }, > > "configuration" : { } > > } > > > > I loaded an extractor config file with it so I'm wondering if that should > > have populated the config fields here or maybe I need to add mappings to > > the column families in there? > > > > Regards, > > > > Tyler > > > > Regards, > > > > Tyler Moore > > Software Engineer > > Flyball Labs > > > > On Thu, Nov 3, 2016 at 3:55 PM, Michael Miklavcic < > > [email protected]> wrote: > > > > > Not sure about the python-kafka lib issues. Regarding enrichment data > > > getting written to ES, how many records have you processed and what is > > your > > > batch size? You might need to write more records or adjust this for the > > > values to propagate through. See the "Sensor Enrichment Configuration" > > > section - > > > https://github.com/apache/incubator-metron/tree/master/ > > > metron-platform/metron-enrichment > > > > > > > > > On Thu, Nov 3, 2016 at 1:03 PM, Tyler Moore <[email protected]> > > wrote: > > > > > > > Mike, > > > > > > > > I am using quick-dev vagrant deployment and at the moment testing > > locally > > > > but we plan on having data from remote locations streaming in to be > > > parsed. > > > > I was able to get the parsers running, thanks to casey, looks like i > > > missed > > > > an update to the Hbase enrichment writer naming convention. > > > > Still working on the enrichment configs though, they aren't throwing > > any > > > > errors and storm says they are emitting data, but not being written > to > > > > elastic. > > > > As well with the python-kafka library, can't figure out why the json > > > > serializer isn't working, as long as I have a parser implemented I > > could > > > > forego serializing the data > > > > prior to sending to a kafka topic correct?? > > > > > > > > Thanks for all your help thus far! > > > > > > > > Regards, > > > > > > > > Tyler > > > > > > > > Regards, > > > > > > > > Tyler Moore > > > > Software Engineer > > > > Flyball Labs > > > > > > > > On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic < > > > > [email protected]> wrote: > > > > > > > > > Tyler, > > > > > > > > > > Thanks for the interest in Metron and welcome to the community! :) > > > > > > > > > > Just curious, what type of environment are you running in? Full > > cluster > > > > or > > > > > are you using the full-dev or quick-dev vagrant deployment vagrant > > > > scripts? > > > > > > > > > > Best, > > > > > Mike Miklavcic > > > > > > > > > > > > > > > On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore <[email protected] > > > > > > wrote: > > > > > > > > > > > Haven't heard of the acronym before, i'm kinda new to the dev > game > > :D > > > > > > > > > > > > Do you have any idea why my the enriched data isn't being written > > to > > > > > > elasticsearch? > > > > > > > > > > > > Regards, > > > > > > > > > > > > Tyler Moore > > > > > > Software Engineer > > > > > > Flyball Labs > > > > > > > > > > > > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > Thanks for finding that; I fixed it in the wiki. Isn't OSS > > > awesome? > > > > ;) > > > > > > > > > > > > > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > No problem, > > > > > > > > > > > > > > > > I was following the Metron application tutorials in the > Metron > > > > wiki: > > > > > > > > https://cwiki.apache.org/confluence/display/METRON/ > > > > > > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+ > > > > > > Streaming+Enrichment > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Tyler Moore > > > > > > > > Software Engineer > > > > > > > > Flyball Labs > > > > > > > > > > > > > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Ah, so quick feedback here, that class path has changed > from > > > > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter > > to > > > > > > > > > org.apache.metron.enrichment.writer. > > > SimpleHbaseEnrichmentWriter > > > > > > > > > > > > > > > > > > There is probably some outdated documentation somewhere, > > would > > > > you > > > > > > mind > > > > > > > > > pointing out where you got that one? > > > > > > > > > > > > > > > > > > Casey > > > > > > > > > > > > > > > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Casey, > > > > > > > > > > > > > > > > > > > > Thanks for the quick reply, love your work by the way! > > > > > > > > > > > > > > > > > > > > When I try to upload the parser I am getting a stack > trace > > > like > > > > > > this: > > > > > > > > > > 15:43:33.182 [main-EventThread] INFO o.a.c.f.s. > > > > > > > ConnectionStateManager > > > > > > > > - > > > > > > > > > > State change: CONNECTED > > > > > > > > > > java.lang.IllegalStateException: Unable to instantiate > > > > > connector: > > > > > > > > class > > > > > > > > > > not > > > > > > > > > > found > > > > > > > > > > at > > > > > > > > > > org.apache.metron.common.utils.ReflectionUtils. > > > createInstance( > > > > > > > > > > ReflectionUtils.java:56) > > > > > > > > > > at > > > > > > > > > > org.apache.metron.parsers.topology. > ParserTopologyBuilder. > > > > > > > > > createParserBolt( > > > > > > > > > > ParserTopologyBuilder.java:155) > > > > > > > > > > at > > > > > > > > > > org.apache.metron.parsers.topology. > > > > ParserTopologyBuilder.build( > > > > > > > > > > ParserTopologyBuilder.java:94) > > > > > > > > > > at > > > > > > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI. > > > > > > > > > > main(ParserTopologyCLI.java:298) > > > > > > > > > > Caused by: java.lang.ClassNotFoundException: > > > > > > > > > > org.apache.metron.writer.hbase. > SimpleHbaseEnrichmentWriter > > > > > > > > > > at java.net.URLClassLoader. > findClass(URLClassLoader.java: > > > 381) > > > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > > > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass( > > > > Launcher.java:331) > > > > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > > > > > > > at java.lang.Class.forName0(Native Method) > > > > > > > > > > at java.lang.Class.forName(Class.java:264) > > > > > > > > > > at > > > > > > > > > > org.apache.metron.common.utils.ReflectionUtils. > > > createInstance( > > > > > > > > > > ReflectionUtils.java:53) > > > > > > > > > > ... 3 more > > > > > > > > > > > > > > > > > > > > The storm supervisor log is saying the some of the > prcosses > > > > > aren't > > > > > > > > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO] > > > > > > > > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't > started > > > > > > > > > > and is throwing TimoutExceptions, I believe that is due > to > > > the > > > > > > > parser. > > > > > > > > > > > > > > > > > > > > Without the parser though (when troubleshooting the > > > enrichment > > > > > > config > > > > > > > > > from > > > > > > > > > > #1) I don't receive and errors from storm and the > > enrichment > > > > > bolts > > > > > > > seem > > > > > > > > > to > > > > > > > > > > be splitting the data but writer bolt emits 0 everytime. > > > > > > > > > > We are able to use the built-in hostname enrichment but > the > > > > > custom > > > > > > > one > > > > > > > > I > > > > > > > > > > built (which will eventually be converted into asset > > > discovery > > > > > > > > > enrichment) > > > > > > > > > > doesn't seem to be writing to elastic search. Do I need > to > > > > setup > > > > > a > > > > > > > new > > > > > > > > > > index template to receive the data from the new > enrichment > > > > > config? > > > > > > Or > > > > > > > > > > should I be looking at creating a new spout / bolt to > > > transfer > > > > > the > > > > > > > > data? > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > Tyler > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > Tyler Moore > > > > > > > > > > Software Engineer > > > > > > > > > > Flyball Labs > > > > > > > > > > > > > > > > > > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > First off Tyler, thanks for using Metron. > > > > > > > > > > > > > > > > > > > > > > Do you have any errors or stack traces that are being > > > thrown > > > > > > > (keeping > > > > > > > > > in > > > > > > > > > > > mind that in storm, they may be in the storm logs > > > > > (/var/log/storm > > > > > > > on > > > > > > > > > the > > > > > > > > > > > supervisor nodes)? > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hey everyone, > > > > > > > > > > > > > > > > > > > > > > > > I've had a few sticking points that I encountered > while > > > > > trying > > > > > > to > > > > > > > > > > create > > > > > > > > > > > > some custom solutions using the Metron platform and > > could > > > > use > > > > > > > some > > > > > > > > > > > > guidance. > > > > > > > > > > > > > > > > > > > > > > > > 1) My custom enrichment config is not writing to > > > > > elasticsearch > > > > > > or > > > > > > > > may > > > > > > > > > > be > > > > > > > > > > > > configured improperly. > > > > > > > > > > > > > > > > > > > > > > > > My extractor config: > > > > > > > > > > > > { > > > > > > > > > > > > "config" : { > > > > > > > > > > > > "columns" : { > > > > > > > > > > > > "ip" : 0, > > > > > > > > > > > > "host" : 1 > > > > > > > > > > > > }, > > > > > > > > > > > > "indicator_column" : "ip", > > > > > > > > > > > > "type" : "hostname", > > > > > > > > > > > > "separator" : "," > > > > > > > > > > > > }, > > > > > > > > > > > > "extractor" : "CSV" > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > My enrichment config: > > > > > > > > > > > > { > > > > > > > > > > > > "zkQuorum" : "node1:2181", > > > > > > > > > > > > "sensorToFieldList" : { > > > > > > > > > > > > "bro" : { > > > > > > > > > > > > "type" : "ENRICHMENT", > > > > > > > > > > > > "fieldToEnrichmentTypes" : { > > > > > > > > > > > > "ip_src_addr" : ["hostname"], > > > > > > > > > > > > "ip_dst_addr" : ["hostname"] > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > A sample of the data i'm uploading: > > > > > > > > > > > > 0.0.0.0, "IGMP" > > > > > > > > > > > > 10.113.145.135, "GLAZER" > > > > > > > > > > > > 10.113.145.137, "GLAZER" > > > > > > > > > > > > 10.113.145.138, "GLAZER" > > > > > > > > > > > > > > > > > > > > > > > > i'm uploading to zookeeper using the following > command: > > > > > > > > > > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n > > > > > > > > > > > > hostname_enrichment_config.json -i hostname_ref.csv > -t > > > > > > > enrichment > > > > > > > > -c > > > > > > > > > > > hosts > > > > > > > > > > > > -e hostname_extractor_config.json > > > > > > > > > > > > > > > > > > > > > > > > 2) We eventually want to parse this data as a live > > stream > > > > but > > > > > > the > > > > > > > > > > parser > > > > > > > > > > > > errors out when I try sending data in. Here is the > > parser > > > > > > config: > > > > > > > > > > > > { > > > > > > > > > > > > "parserClassName" : "org.apache.metron.parsers. > > > > > > csv.CSVParser", > > > > > > > > > > > > "writerClassName" : > > > > > > > > > > > > "org.apache.metron.writer.hbase. > > > > > SimpleHbaseEnrichmentWriter", > > > > > > > > > > > > "sensorTopic":"hostname", > > > > > > > > > > > > "parserConfig": > > > > > > > > > > > > { > > > > > > > > > > > > "shew.table" : "enrichment", > > > > > > > > > > > > "shew.cf" : "hosts", > > > > > > > > > > > > "shew.keyColumns" : "ip", > > > > > > > > > > > > "shew.enrichmentType" : "hostname", > > > > > > > > > > > > "columns" : { > > > > > > > > > > > > "ip" : 0, > > > > > > > > > > > > "host" : 1 > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > 3) We will be moving from replay to using > kafka-python > > > for > > > > > > > sending > > > > > > > > > data > > > > > > > > > > > > captures and I am able to send bytes to a new topic, > > but > > > > > when I > > > > > > > try > > > > > > > > > > using > > > > > > > > > > > > the json serializer via kafka producer my program > exits > > > > > without > > > > > > > > error > > > > > > > > > > and > > > > > > > > > > > > no data is sent. > > > > > > > > > > > > Here is the section of the python code i'm having > > trouble > > > > > with: > > > > > > > > > > > > > > > > > > > > > > > > producer = KafkaProducer(bootstrap_servers=' > > > > > 50.253.243.17:6667 > > > > > > ', > > > > > > > > > > > > value_serializer=lambda m: > > json.dumps(m).encode('ascii'), > > > > > > > > > > api_version=(0, > > > > > > > > > > > > 9)) > > > > > > > > > > > > > > > > > > > > > > > > for _ in range(100): > > > > > > > > > > > > producer.send('pcap', {'key': 'value'}) > > > > > > > > > > > > producer.flush() > > > > > > > > > > > > > > > > > > > > > > > > If anyone could point me in the right direction that > > > would > > > > be > > > > > > > > great!! > > > > > > > > > > I'm > > > > > > > > > > > > not sure if the first 2 problems are related to > > indexing > > > or > > > > > > > maybe I > > > > > > > > > > need > > > > > > > > > > > to > > > > > > > > > > > > create a bolt to pass on the data in storm? > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > > Tyler Moore > > > > > > > > > > > > Software Engineer > > > > > > > > > > > > Flyball Labs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
