Re: Help with custom enrichment / parser

[Michael Miklavcic]

Can you check for any exceptions in the enrichment logs using the following
grep?
grep --color=auto -C 3 -R -iE "exception" /var/log/storm

It would also be good to know where the data is getting hung up. Can you
check if you're getting tuples transferring and acking through the indexing
Kafka spout?

On Thu, Nov 3, 2016 at 3:41 PM, Tyler Moore  wrote:

> The sample i'm sending has over logs about 40,000 records so I don't think
> that is the issue.
>
> My batch size is 5 and the this is what it looks like when i dump it from
> zookeeper
> ENRICHMENT Config: bro
> {
>   "index" : "bro",
>   "batchSize" : 5,
>   "enrichment" : {
> "fieldMap" : {
>   "geo" : [ "ip_dst_addr", "ip_src_addr" ],
>   "host" : [ "ip_src_addr", "ip_dst_addr" ],
>   "hbaseEnrichment" : [ "ip_src_addr", "ip_dst_addr" ]
> },
> "fieldToTypeMap" : {
>   "ip_dst_addr" : [ "hostname", "asset" ],
>   "ip_src_addr" : [ "hostname", "asset" ]
> },
> "config" : { }
>   },
>   "threatIntel" : {
> "fieldMap" : {
>   "hbaseThreatIntel" : [ "ip_src_addr", "ip_dst_addr" ]
> },
> "fieldToTypeMap" : {
>   "ip_src_addr" : [ "malicious_ip" ],
>   "ip_dst_addr" : [ "malicious_ip" ]
> },
> "config" : { },
> "triageConfig" : {
>   "riskLevelRules" : { },
>   "aggregator" : "MAX",
>   "aggregationConfig" : { }
> }
>   },
>   "configuration" : { }
> }
>
> I loaded an extractor config file with it so I'm wondering if that should
> have populated the config fields here or maybe I need to add mappings to
> the column families in there?
>
> Regards,
>
> Tyler
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>
> On Thu, Nov 3, 2016 at 3:55 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Not sure about the python-kafka lib issues. Regarding enrichment data
> > getting written to ES, how many records have you processed and what is
> your
> > batch size? You might need to write more records or adjust this for the
> > values to propagate through. See the "Sensor Enrichment Configuration"
> > section -
> > https://github.com/apache/incubator-metron/tree/master/
> > metron-platform/metron-enrichment
> >
> >
> > On Thu, Nov 3, 2016 at 1:03 PM, Tyler Moore 
> wrote:
> >
> > > Mike,
> > >
> > > I am using quick-dev vagrant deployment and at the moment testing
> locally
> > > but we plan on having data from remote locations streaming in to be
> > parsed.
> > > I was able to get the parsers running, thanks to casey, looks like i
> > missed
> > > an update to the Hbase enrichment writer naming convention.
> > > Still working on the enrichment configs though, they aren't throwing
> any
> > > errors and storm says they are emitting data, but not being written to
> > > elastic.
> > > As well with the python-kafka library, can't figure out why the json
> > > serializer isn't working, as long as I have a parser implemented I
> could
> > > forego serializing the data
> > > prior to sending to a kafka topic correct??
> > >
> > > Thanks for all your help thus far!
> > >
> > > Regards,
> > >
> > > Tyler
> > >
> > > Regards,
> > >
> > > Tyler Moore
> > > Software Engineer
> > > Flyball Labs
> > >
> > > On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Tyler,
> > > >
> > > > Thanks for the interest in Metron and welcome to the community! :)
> > > >
> > > > Just curious, what type of environment are you running in? Full
> cluster
> > > or
> > > > are you using the full-dev or quick-dev vagrant deployment vagrant
> > > scripts?
> > > >
> > > > Best,
> > > > Mike Miklavcic
> > > >
> > > >
> > > > On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore 
> > > wrote:
> > > >
> > > > > Haven't heard of the acronym before, i'm kinda new to the dev game
> :D
> > > > >
> > > > > Do you have any idea why my the enriched data isn't being written
> to
> > > > > elasticsearch?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Tyler Moore
> > > > > Software Engineer
> > > > > Flyball Labs
> > > > >
> > > > > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella 
> > > > wrote:
> > > > >
> > > > > > Thanks for finding that; I fixed it in the wiki.  Isn't OSS
> > awesome?
> > > ;)
> > > > > >
> > > > > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore <
> tmo...@goflyball.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > No problem,
> > > > > > >
> > > > > > > I was following the Metron application tutorials in the Metron
> > > wiki:
> > > > > > > https://cwiki.apache.org/confluence/display/METRON/
> > > > > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+
> > > > > Streaming+Enrichment
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Tyler Moore
> > > > > > > Software Engineer
> > > > > > > Flyball Labs
> > > > > > >
> > > > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella <
> > ceste...@gmail.com>

Re: Help with custom enrichment / parser

[Tyler Moore]

The sample i'm sending has over logs about 40,000 records so I don't think
that is the issue.

My batch size is 5 and the this is what it looks like when i dump it from
zookeeper
ENRICHMENT Config: bro
{
  "index" : "bro",
  "batchSize" : 5,
  "enrichment" : {
"fieldMap" : {
  "geo" : [ "ip_dst_addr", "ip_src_addr" ],
  "host" : [ "ip_src_addr", "ip_dst_addr" ],
  "hbaseEnrichment" : [ "ip_src_addr", "ip_dst_addr" ]
},
"fieldToTypeMap" : {
  "ip_dst_addr" : [ "hostname", "asset" ],
  "ip_src_addr" : [ "hostname", "asset" ]
},
"config" : { }
  },
  "threatIntel" : {
"fieldMap" : {
  "hbaseThreatIntel" : [ "ip_src_addr", "ip_dst_addr" ]
},
"fieldToTypeMap" : {
  "ip_src_addr" : [ "malicious_ip" ],
  "ip_dst_addr" : [ "malicious_ip" ]
},
"config" : { },
"triageConfig" : {
  "riskLevelRules" : { },
  "aggregator" : "MAX",
  "aggregationConfig" : { }
}
  },
  "configuration" : { }
}

I loaded an extractor config file with it so I'm wondering if that should
have populated the config fields here or maybe I need to add mappings to
the column families in there?

Regards,

Tyler

Regards,

Tyler Moore
Software Engineer
Flyball Labs

On Thu, Nov 3, 2016 at 3:55 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Not sure about the python-kafka lib issues. Regarding enrichment data
> getting written to ES, how many records have you processed and what is your
> batch size? You might need to write more records or adjust this for the
> values to propagate through. See the "Sensor Enrichment Configuration"
> section -
> https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-enrichment
>
>
> On Thu, Nov 3, 2016 at 1:03 PM, Tyler Moore  wrote:
>
> > Mike,
> >
> > I am using quick-dev vagrant deployment and at the moment testing locally
> > but we plan on having data from remote locations streaming in to be
> parsed.
> > I was able to get the parsers running, thanks to casey, looks like i
> missed
> > an update to the Hbase enrichment writer naming convention.
> > Still working on the enrichment configs though, they aren't throwing any
> > errors and storm says they are emitting data, but not being written to
> > elastic.
> > As well with the python-kafka library, can't figure out why the json
> > serializer isn't working, as long as I have a parser implemented I could
> > forego serializing the data
> > prior to sending to a kafka topic correct??
> >
> > Thanks for all your help thus far!
> >
> > Regards,
> >
> > Tyler
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Flyball Labs
> >
> > On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Tyler,
> > >
> > > Thanks for the interest in Metron and welcome to the community! :)
> > >
> > > Just curious, what type of environment are you running in? Full cluster
> > or
> > > are you using the full-dev or quick-dev vagrant deployment vagrant
> > scripts?
> > >
> > > Best,
> > > Mike Miklavcic
> > >
> > >
> > > On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore 
> > wrote:
> > >
> > > > Haven't heard of the acronym before, i'm kinda new to the dev game :D
> > > >
> > > > Do you have any idea why my the enriched data isn't being written to
> > > > elasticsearch?
> > > >
> > > > Regards,
> > > >
> > > > Tyler Moore
> > > > Software Engineer
> > > > Flyball Labs
> > > >
> > > > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella 
> > > wrote:
> > > >
> > > > > Thanks for finding that; I fixed it in the wiki.  Isn't OSS
> awesome?
> > ;)
> > > > >
> > > > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore  >
> > > > wrote:
> > > > >
> > > > > > No problem,
> > > > > >
> > > > > > I was following the Metron application tutorials in the Metron
> > wiki:
> > > > > > https://cwiki.apache.org/confluence/display/METRON/
> > > > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+
> > > > Streaming+Enrichment
> > > > > >
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Tyler Moore
> > > > > > Software Engineer
> > > > > > Flyball Labs
> > > > > >
> > > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella <
> ceste...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Ah, so quick feedback here, that class path has changed from
> > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > > > > > > org.apache.metron.enrichment.writer.
> SimpleHbaseEnrichmentWriter
> > > > > > >
> > > > > > > There is probably some outdated documentation somewhere, would
> > you
> > > > mind
> > > > > > > pointing out where you got that one?
> > > > > > >
> > > > > > > Casey
> > > > > > >
> > > > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore <
> > tmo...@goflyball.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Casey,
> > > > > > > >
> > > > > > > > Thanks for the quick reply, love your work 

Re: Help with custom enrichment / parser

[Michael Miklavcic]

Not sure about the python-kafka lib issues. Regarding enrichment data
getting written to ES, how many records have you processed and what is your
batch size? You might need to write more records or adjust this for the
values to propagate through. See the "Sensor Enrichment Configuration"
section -
https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-enrichment


On Thu, Nov 3, 2016 at 1:03 PM, Tyler Moore  wrote:

> Mike,
>
> I am using quick-dev vagrant deployment and at the moment testing locally
> but we plan on having data from remote locations streaming in to be parsed.
> I was able to get the parsers running, thanks to casey, looks like i missed
> an update to the Hbase enrichment writer naming convention.
> Still working on the enrichment configs though, they aren't throwing any
> errors and storm says they are emitting data, but not being written to
> elastic.
> As well with the python-kafka library, can't figure out why the json
> serializer isn't working, as long as I have a parser implemented I could
> forego serializing the data
> prior to sending to a kafka topic correct??
>
> Thanks for all your help thus far!
>
> Regards,
>
> Tyler
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>
> On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Tyler,
> >
> > Thanks for the interest in Metron and welcome to the community! :)
> >
> > Just curious, what type of environment are you running in? Full cluster
> or
> > are you using the full-dev or quick-dev vagrant deployment vagrant
> scripts?
> >
> > Best,
> > Mike Miklavcic
> >
> >
> > On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore 
> wrote:
> >
> > > Haven't heard of the acronym before, i'm kinda new to the dev game :D
> > >
> > > Do you have any idea why my the enriched data isn't being written to
> > > elasticsearch?
> > >
> > > Regards,
> > >
> > > Tyler Moore
> > > Software Engineer
> > > Flyball Labs
> > >
> > > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella 
> > wrote:
> > >
> > > > Thanks for finding that; I fixed it in the wiki.  Isn't OSS awesome?
> ;)
> > > >
> > > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore 
> > > wrote:
> > > >
> > > > > No problem,
> > > > >
> > > > > I was following the Metron application tutorials in the Metron
> wiki:
> > > > > https://cwiki.apache.org/confluence/display/METRON/
> > > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+
> > > Streaming+Enrichment
> > > > >
> > > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > Tyler Moore
> > > > > Software Engineer
> > > > > Flyball Labs
> > > > >
> > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella 
> > > > wrote:
> > > > >
> > > > > > Ah, so quick feedback here, that class path has changed from
> > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > > > > > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
> > > > > >
> > > > > > There is probably some outdated documentation somewhere, would
> you
> > > mind
> > > > > > pointing out where you got that one?
> > > > > >
> > > > > > Casey
> > > > > >
> > > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore <
> tmo...@goflyball.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Casey,
> > > > > > >
> > > > > > > Thanks for the quick reply, love your work by the way!
> > > > > > >
> > > > > > > When I try to upload the parser I am getting a stack trace like
> > > this:
> > > > > > > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.
> > > > ConnectionStateManager
> > > > > -
> > > > > > > State change: CONNECTED
> > > > > > > java.lang.IllegalStateException: Unable to instantiate
> > connector:
> > > > > class
> > > > > > > not
> > > > > > > found
> > > > > > > at
> > > > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > > > > ReflectionUtils.java:56)
> > > > > > > at
> > > > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> > > > > > createParserBolt(
> > > > > > > ParserTopologyBuilder.java:155)
> > > > > > > at
> > > > > > > org.apache.metron.parsers.topology.
> ParserTopologyBuilder.build(
> > > > > > > ParserTopologyBuilder.java:94)
> > > > > > > at
> > > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > > > > > > main(ParserTopologyCLI.java:298)
> > > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass(
> Launcher.java:331)
> > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > at
> > > > > > > 

Re: Help with custom enrichment / parser

[Tyler Moore]

Mike,

I am using quick-dev vagrant deployment and at the moment testing locally
but we plan on having data from remote locations streaming in to be parsed.
I was able to get the parsers running, thanks to casey, looks like i missed
an update to the Hbase enrichment writer naming convention.
Still working on the enrichment configs though, they aren't throwing any
errors and storm says they are emitting data, but not being written to
elastic.
As well with the python-kafka library, can't figure out why the json
serializer isn't working, as long as I have a parser implemented I could
forego serializing the data
prior to sending to a kafka topic correct??

Thanks for all your help thus far!

Regards,

Tyler

Regards,

Tyler Moore
Software Engineer
Flyball Labs

On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Tyler,
>
> Thanks for the interest in Metron and welcome to the community! :)
>
> Just curious, what type of environment are you running in? Full cluster or
> are you using the full-dev or quick-dev vagrant deployment vagrant scripts?
>
> Best,
> Mike Miklavcic
>
>
> On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore  wrote:
>
> > Haven't heard of the acronym before, i'm kinda new to the dev game :D
> >
> > Do you have any idea why my the enriched data isn't being written to
> > elasticsearch?
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Flyball Labs
> >
> > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella 
> wrote:
> >
> > > Thanks for finding that; I fixed it in the wiki.  Isn't OSS awesome? ;)
> > >
> > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore 
> > wrote:
> > >
> > > > No problem,
> > > >
> > > > I was following the Metron application tutorials in the Metron wiki:
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+
> > Streaming+Enrichment
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Tyler Moore
> > > > Software Engineer
> > > > Flyball Labs
> > > >
> > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella 
> > > wrote:
> > > >
> > > > > Ah, so quick feedback here, that class path has changed from
> > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > > > > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
> > > > >
> > > > > There is probably some outdated documentation somewhere, would you
> > mind
> > > > > pointing out where you got that one?
> > > > >
> > > > > Casey
> > > > >
> > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore  >
> > > > wrote:
> > > > >
> > > > > > Casey,
> > > > > >
> > > > > > Thanks for the quick reply, love your work by the way!
> > > > > >
> > > > > > When I try to upload the parser I am getting a stack trace like
> > this:
> > > > > > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.
> > > ConnectionStateManager
> > > > -
> > > > > > State change: CONNECTED
> > > > > > java.lang.IllegalStateException: Unable to instantiate
> connector:
> > > > class
> > > > > > not
> > > > > > found
> > > > > > at
> > > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > > > ReflectionUtils.java:56)
> > > > > > at
> > > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> > > > > createParserBolt(
> > > > > > ParserTopologyBuilder.java:155)
> > > > > > at
> > > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> > > > > > ParserTopologyBuilder.java:94)
> > > > > > at
> > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > > > > > main(ParserTopologyCLI.java:298)
> > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > at
> > > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > > > ReflectionUtils.java:53)
> > > > > > ... 3 more
> > > > > >
> > > > > > The storm supervisor log is saying the some of the prcosses
> aren't
> > > > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > > > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > > > > > and is throwing TimoutExceptions, I believe that is due to the
> > > parser.
> > > > > >
> > > > > > Without the parser though (when troubleshooting the enrichment
> > config
> > > > > from
> > > > > > #1) I don't receive and errors from storm and the enrichment
> bolts
> > > seem
> > > > > to
> > > > > > be splitting the data but writer bolt emits 0 everytime.
> > > > > > We are able 

Re: Help with custom enrichment / parser

[Michael Miklavcic]

Tyler,

Thanks for the interest in Metron and welcome to the community! :)

Just curious, what type of environment are you running in? Full cluster or
are you using the full-dev or quick-dev vagrant deployment vagrant scripts?

Best,
Mike Miklavcic


On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore  wrote:

> Haven't heard of the acronym before, i'm kinda new to the dev game :D
>
> Do you have any idea why my the enriched data isn't being written to
> elasticsearch?
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>
> On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella  wrote:
>
> > Thanks for finding that; I fixed it in the wiki.  Isn't OSS awesome? ;)
> >
> > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore 
> wrote:
> >
> > > No problem,
> > >
> > > I was following the Metron application tutorials in the Metron wiki:
> > > https://cwiki.apache.org/confluence/display/METRON/
> > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+
> Streaming+Enrichment
> > >
> > >
> > >
> > > Regards,
> > >
> > > Tyler Moore
> > > Software Engineer
> > > Flyball Labs
> > >
> > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella 
> > wrote:
> > >
> > > > Ah, so quick feedback here, that class path has changed from
> > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > > > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
> > > >
> > > > There is probably some outdated documentation somewhere, would you
> mind
> > > > pointing out where you got that one?
> > > >
> > > > Casey
> > > >
> > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore 
> > > wrote:
> > > >
> > > > > Casey,
> > > > >
> > > > > Thanks for the quick reply, love your work by the way!
> > > > >
> > > > > When I try to upload the parser I am getting a stack trace like
> this:
> > > > > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.
> > ConnectionStateManager
> > > -
> > > > > State change: CONNECTED
> > > > > java.lang.IllegalStateException: Unable to instantiate connector:
> > > class
> > > > > not
> > > > > found
> > > > > at
> > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > > ReflectionUtils.java:56)
> > > > > at
> > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> > > > createParserBolt(
> > > > > ParserTopologyBuilder.java:155)
> > > > > at
> > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> > > > > ParserTopologyBuilder.java:94)
> > > > > at
> > > > > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > > > > main(ParserTopologyCLI.java:298)
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > at java.lang.Class.forName0(Native Method)
> > > > > at java.lang.Class.forName(Class.java:264)
> > > > > at
> > > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > > ReflectionUtils.java:53)
> > > > > ... 3 more
> > > > >
> > > > > The storm supervisor log is saying the some of the prcosses aren't
> > > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > > > > and is throwing TimoutExceptions, I believe that is due to the
> > parser.
> > > > >
> > > > > Without the parser though (when troubleshooting the enrichment
> config
> > > > from
> > > > > #1) I don't receive and errors from storm and the enrichment bolts
> > seem
> > > > to
> > > > > be splitting the data but writer bolt emits 0 everytime.
> > > > > We are able to use the built-in hostname enrichment but the custom
> > one
> > > I
> > > > > built (which will eventually be converted into asset discovery
> > > > enrichment)
> > > > > doesn't seem to be writing to elastic search. Do I need to setup a
> > new
> > > > > index template to receive the data from the new enrichment config?
> Or
> > > > > should I be looking at creating a new spout / bolt to transfer the
> > > data?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Tyler
> > > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > Tyler Moore
> > > > > Software Engineer
> > > > > Flyball Labs
> > > > >
> > > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella 
> > > wrote:
> > > > >
> > > > > > First off Tyler, thanks for using Metron.
> > > > > >
> > > > > > Do you have any errors or stack traces that are being thrown
> > (keeping
> > > > in
> > > > > > mind that in storm, they may be in the storm logs (/var/log/storm
> > on
> > > > the
> > > > > > supervisor nodes)?
> > > > > >
> > > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore <
> tmo...@goflyball.com
> 

Re: Help with custom enrichment / parser

[Tyler Moore]

log is empty

Regards,

Tyler Moore
Software Engineer
Flyball Labs

On Thu, Nov 3, 2016 at 1:41 PM, Casey Stella  wrote:

> Do you see any errors in the storm logs in /var/log on the storm
> supervisors?
>
> On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore  wrote:
>
> > Casey,
> >
> > Thanks for the quick reply, love your work by the way!
> >
> > When I try to upload the parser I am getting a stack trace like this:
> > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager -
> > State change: CONNECTED
> > java.lang.IllegalStateException: Unable to instantiate connector: class
> > not
> > found
> > at
> > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > ReflectionUtils.java:56)
> > at
> > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> createParserBolt(
> > ParserTopologyBuilder.java:155)
> > at
> > org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> > ParserTopologyBuilder.java:94)
> > at
> > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > main(ParserTopologyCLI.java:298)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:264)
> > at
> > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > ReflectionUtils.java:53)
> > ... 3 more
> >
> > The storm supervisor log is saying the some of the prcosses aren't
> > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > and is throwing TimoutExceptions, I believe that is due to the parser.
> >
> > Without the parser though (when troubleshooting the enrichment config
> from
> > #1) I don't receive and errors from storm and the enrichment bolts seem
> to
> > be splitting the data but writer bolt emits 0 everytime.
> > We are able to use the built-in hostname enrichment but the custom one I
> > built (which will eventually be converted into asset discovery
> enrichment)
> > doesn't seem to be writing to elastic search. Do I need to setup a new
> > index template to receive the data from the new enrichment config? Or
> > should I be looking at creating a new spout / bolt to transfer the data?
> >
> > Regards,
> >
> > Tyler
> >
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Flyball Labs
> >
> > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella  wrote:
> >
> > > First off Tyler, thanks for using Metron.
> > >
> > > Do you have any errors or stack traces that are being thrown (keeping
> in
> > > mind that in storm, they may be in the storm logs (/var/log/storm on
> the
> > > supervisor nodes)?
> > >
> > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore 
> > wrote:
> > >
> > > > Hey everyone,
> > > >
> > > > I've had a few sticking points that I encountered while trying to
> > create
> > > > some custom solutions using the Metron platform and could use some
> > > > guidance.
> > > >
> > > > 1) My custom enrichment config is not writing to elasticsearch or may
> > be
> > > > configured improperly.
> > > >
> > > > My extractor config:
> > > > {
> > > >   "config" : {
> > > > "columns" : {
> > > >  "ip" : 0,
> > > >  "host" : 1
> > > > },
> > > >  "indicator_column" : "ip",
> > > >  "type" : "hostname",
> > > >  "separator" : ","
> > > >   },
> > > >   "extractor" : "CSV"
> > > > }
> > > >
> > > > My enrichment config:
> > > > {
> > > >   "zkQuorum" : "node1:2181",
> > > >   "sensorToFieldList" : {
> > > >  "bro" : {
> > > >"type" : "ENRICHMENT",
> > > >"fieldToEnrichmentTypes" : {
> > > >  "ip_src_addr" : ["hostname"],
> > > >  "ip_dst_addr" : ["hostname"]
> > > >  }
> > > >   }
> > > >}
> > > > }
> > > >
> > > > A sample of the data i'm uploading:
> > > > 0.0.0.0, "IGMP"
> > > > 10.113.145.135, "GLAZER"
> > > > 10.113.145.137, "GLAZER"
> > > > 10.113.145.138, "GLAZER"
> > > >
> > > > i'm uploading to zookeeper using the following command:
> > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
> > > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c
> > > hosts
> > > > -e hostname_extractor_config.json
> > > >
> > > > 2) We eventually want to parse this data as a live stream but the
> > parser
> > > > errors out when I try sending data in. Here is the parser config:
> > > > {
> > > >   "parserClassName" : "org.apache.metron.parsers.csv.CSVParser",
> > > >   "writerClassName" :
> > > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
> > > >   "sensorTopic":"hostname",
> > > >   "parserConfig":
> > > >   {
> > > > 

Re: Help with custom enrichment / parser

[Casey Stella]

Do you see any errors in the storm logs in /var/log on the storm
supervisors?

On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore  wrote:

> Casey,
>
> Thanks for the quick reply, love your work by the way!
>
> When I try to upload the parser I am getting a stack trace like this:
> 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager -
> State change: CONNECTED
> java.lang.IllegalStateException: Unable to instantiate connector: class
> not
> found
> at
> org.apache.metron.common.utils.ReflectionUtils.createInstance(
> ReflectionUtils.java:56)
> at
> org.apache.metron.parsers.topology.ParserTopologyBuilder.createParserBolt(
> ParserTopologyBuilder.java:155)
> at
> org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> ParserTopologyBuilder.java:94)
> at
> org.apache.metron.parsers.topology.ParserTopologyCLI.
> main(ParserTopologyCLI.java:298)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.metron.common.utils.ReflectionUtils.createInstance(
> ReflectionUtils.java:53)
> ... 3 more
>
> The storm supervisor log is saying the some of the prcosses aren't
> starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> and is throwing TimoutExceptions, I believe that is due to the parser.
>
> Without the parser though (when troubleshooting the enrichment config from
> #1) I don't receive and errors from storm and the enrichment bolts seem to
> be splitting the data but writer bolt emits 0 everytime.
> We are able to use the built-in hostname enrichment but the custom one I
> built (which will eventually be converted into asset discovery enrichment)
> doesn't seem to be writing to elastic search. Do I need to setup a new
> index template to receive the data from the new enrichment config? Or
> should I be looking at creating a new spout / bolt to transfer the data?
>
> Regards,
>
> Tyler
>
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>
> On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella  wrote:
>
> > First off Tyler, thanks for using Metron.
> >
> > Do you have any errors or stack traces that are being thrown (keeping in
> > mind that in storm, they may be in the storm logs (/var/log/storm on the
> > supervisor nodes)?
> >
> > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore 
> wrote:
> >
> > > Hey everyone,
> > >
> > > I've had a few sticking points that I encountered while trying to
> create
> > > some custom solutions using the Metron platform and could use some
> > > guidance.
> > >
> > > 1) My custom enrichment config is not writing to elasticsearch or may
> be
> > > configured improperly.
> > >
> > > My extractor config:
> > > {
> > >   "config" : {
> > > "columns" : {
> > >  "ip" : 0,
> > >  "host" : 1
> > > },
> > >  "indicator_column" : "ip",
> > >  "type" : "hostname",
> > >  "separator" : ","
> > >   },
> > >   "extractor" : "CSV"
> > > }
> > >
> > > My enrichment config:
> > > {
> > >   "zkQuorum" : "node1:2181",
> > >   "sensorToFieldList" : {
> > >  "bro" : {
> > >"type" : "ENRICHMENT",
> > >"fieldToEnrichmentTypes" : {
> > >  "ip_src_addr" : ["hostname"],
> > >  "ip_dst_addr" : ["hostname"]
> > >  }
> > >   }
> > >}
> > > }
> > >
> > > A sample of the data i'm uploading:
> > > 0.0.0.0, "IGMP"
> > > 10.113.145.135, "GLAZER"
> > > 10.113.145.137, "GLAZER"
> > > 10.113.145.138, "GLAZER"
> > >
> > > i'm uploading to zookeeper using the following command:
> > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
> > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c
> > hosts
> > > -e hostname_extractor_config.json
> > >
> > > 2) We eventually want to parse this data as a live stream but the
> parser
> > > errors out when I try sending data in. Here is the parser config:
> > > {
> > >   "parserClassName" : "org.apache.metron.parsers.csv.CSVParser",
> > >   "writerClassName" :
> > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
> > >   "sensorTopic":"hostname",
> > >   "parserConfig":
> > >   {
> > > "shew.table" : "enrichment",
> > > "shew.cf" : "hosts",
> > > "shew.keyColumns" : "ip",
> > > "shew.enrichmentType" : "hostname",
> > > "columns" : {
> > >   "ip" : 0,
> > >   "host" : 1
> > > }
> > >   }
> > > }
> > >
> > > 3) We will be moving from replay to using kafka-python for sending data
> > > captures and I am able to send bytes to a new topic, but when I try
> using
> > > the json 

Re: Help with custom enrichment / parser

[Tyler Moore]

Haven't heard of the acronym before, i'm kinda new to the dev game :D

Do you have any idea why my the enriched data isn't being written to
elasticsearch?

Regards,

Tyler Moore
Software Engineer
Flyball Labs

On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella  wrote:

> Thanks for finding that; I fixed it in the wiki.  Isn't OSS awesome? ;)
>
> On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore  wrote:
>
> > No problem,
> >
> > I was following the Metron application tutorials in the Metron wiki:
> > https://cwiki.apache.org/confluence/display/METRON/
> > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment
> >
> >
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Flyball Labs
> >
> > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella 
> wrote:
> >
> > > Ah, so quick feedback here, that class path has changed from
> > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
> > >
> > > There is probably some outdated documentation somewhere, would you mind
> > > pointing out where you got that one?
> > >
> > > Casey
> > >
> > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore 
> > wrote:
> > >
> > > > Casey,
> > > >
> > > > Thanks for the quick reply, love your work by the way!
> > > >
> > > > When I try to upload the parser I am getting a stack trace like this:
> > > > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.
> ConnectionStateManager
> > -
> > > > State change: CONNECTED
> > > > java.lang.IllegalStateException: Unable to instantiate connector:
> > class
> > > > not
> > > > found
> > > > at
> > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > ReflectionUtils.java:56)
> > > > at
> > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> > > createParserBolt(
> > > > ParserTopologyBuilder.java:155)
> > > > at
> > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> > > > ParserTopologyBuilder.java:94)
> > > > at
> > > > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > > > main(ParserTopologyCLI.java:298)
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > at java.lang.Class.forName0(Native Method)
> > > > at java.lang.Class.forName(Class.java:264)
> > > > at
> > > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > > ReflectionUtils.java:53)
> > > > ... 3 more
> > > >
> > > > The storm supervisor log is saying the some of the prcosses aren't
> > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > > > and is throwing TimoutExceptions, I believe that is due to the
> parser.
> > > >
> > > > Without the parser though (when troubleshooting the enrichment config
> > > from
> > > > #1) I don't receive and errors from storm and the enrichment bolts
> seem
> > > to
> > > > be splitting the data but writer bolt emits 0 everytime.
> > > > We are able to use the built-in hostname enrichment but the custom
> one
> > I
> > > > built (which will eventually be converted into asset discovery
> > > enrichment)
> > > > doesn't seem to be writing to elastic search. Do I need to setup a
> new
> > > > index template to receive the data from the new enrichment config? Or
> > > > should I be looking at creating a new spout / bolt to transfer the
> > data?
> > > >
> > > > Regards,
> > > >
> > > > Tyler
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Tyler Moore
> > > > Software Engineer
> > > > Flyball Labs
> > > >
> > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella 
> > wrote:
> > > >
> > > > > First off Tyler, thanks for using Metron.
> > > > >
> > > > > Do you have any errors or stack traces that are being thrown
> (keeping
> > > in
> > > > > mind that in storm, they may be in the storm logs (/var/log/storm
> on
> > > the
> > > > > supervisor nodes)?
> > > > >
> > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore  >
> > > > wrote:
> > > > >
> > > > > > Hey everyone,
> > > > > >
> > > > > > I've had a few sticking points that I encountered while trying to
> > > > create
> > > > > > some custom solutions using the Metron platform and could use
> some
> > > > > > guidance.
> > > > > >
> > > > > > 1) My custom enrichment config is not writing to elasticsearch or
> > may
> > > > be
> > > > > > configured improperly.
> > > > > >
> > > > > > My extractor config:
> > > > > > {
> > > > > >   "config" : {
> > > > > > "columns" : {
> > > > > >  "ip" : 0,
> > > > > >  "host" : 1
> > > > > > },
> 

Re: Help with custom enrichment / parser

[Casey Stella]

Thanks for finding that; I fixed it in the wiki.  Isn't OSS awesome? ;)

On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore  wrote:

> No problem,
>
> I was following the Metron application tutorials in the Metron wiki:
> https://cwiki.apache.org/confluence/display/METRON/
> 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment
>
>
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>
> On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella  wrote:
>
> > Ah, so quick feedback here, that class path has changed from
> > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
> >
> > There is probably some outdated documentation somewhere, would you mind
> > pointing out where you got that one?
> >
> > Casey
> >
> > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore 
> wrote:
> >
> > > Casey,
> > >
> > > Thanks for the quick reply, love your work by the way!
> > >
> > > When I try to upload the parser I am getting a stack trace like this:
> > > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager
> -
> > > State change: CONNECTED
> > > java.lang.IllegalStateException: Unable to instantiate connector:
> class
> > > not
> > > found
> > > at
> > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > ReflectionUtils.java:56)
> > > at
> > > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> > createParserBolt(
> > > ParserTopologyBuilder.java:155)
> > > at
> > > org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> > > ParserTopologyBuilder.java:94)
> > > at
> > > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > > main(ParserTopologyCLI.java:298)
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > at java.lang.Class.forName0(Native Method)
> > > at java.lang.Class.forName(Class.java:264)
> > > at
> > > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > > ReflectionUtils.java:53)
> > > ... 3 more
> > >
> > > The storm supervisor log is saying the some of the prcosses aren't
> > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > > and is throwing TimoutExceptions, I believe that is due to the parser.
> > >
> > > Without the parser though (when troubleshooting the enrichment config
> > from
> > > #1) I don't receive and errors from storm and the enrichment bolts seem
> > to
> > > be splitting the data but writer bolt emits 0 everytime.
> > > We are able to use the built-in hostname enrichment but the custom one
> I
> > > built (which will eventually be converted into asset discovery
> > enrichment)
> > > doesn't seem to be writing to elastic search. Do I need to setup a new
> > > index template to receive the data from the new enrichment config? Or
> > > should I be looking at creating a new spout / bolt to transfer the
> data?
> > >
> > > Regards,
> > >
> > > Tyler
> > >
> > >
> > > Regards,
> > >
> > > Tyler Moore
> > > Software Engineer
> > > Flyball Labs
> > >
> > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella 
> wrote:
> > >
> > > > First off Tyler, thanks for using Metron.
> > > >
> > > > Do you have any errors or stack traces that are being thrown (keeping
> > in
> > > > mind that in storm, they may be in the storm logs (/var/log/storm on
> > the
> > > > supervisor nodes)?
> > > >
> > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore 
> > > wrote:
> > > >
> > > > > Hey everyone,
> > > > >
> > > > > I've had a few sticking points that I encountered while trying to
> > > create
> > > > > some custom solutions using the Metron platform and could use some
> > > > > guidance.
> > > > >
> > > > > 1) My custom enrichment config is not writing to elasticsearch or
> may
> > > be
> > > > > configured improperly.
> > > > >
> > > > > My extractor config:
> > > > > {
> > > > >   "config" : {
> > > > > "columns" : {
> > > > >  "ip" : 0,
> > > > >  "host" : 1
> > > > > },
> > > > >  "indicator_column" : "ip",
> > > > >  "type" : "hostname",
> > > > >  "separator" : ","
> > > > >   },
> > > > >   "extractor" : "CSV"
> > > > > }
> > > > >
> > > > > My enrichment config:
> > > > > {
> > > > >   "zkQuorum" : "node1:2181",
> > > > >   "sensorToFieldList" : {
> > > > >  "bro" : {
> > > > >"type" : "ENRICHMENT",
> > > > >"fieldToEnrichmentTypes" : {
> > > > >  "ip_src_addr" : ["hostname"],
> > > > >  "ip_dst_addr" : ["hostname"]
> > > > >  }
> > > > >   }
> > > > >}
> > > > > }

Re: Help with custom enrichment / parser

[Tyler Moore]

No problem,

I was following the Metron application tutorials in the Metron wiki:
https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment



Regards,

Tyler Moore
Software Engineer
Flyball Labs

On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella  wrote:

> Ah, so quick feedback here, that class path has changed from
> org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter
>
> There is probably some outdated documentation somewhere, would you mind
> pointing out where you got that one?
>
> Casey
>
> On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore  wrote:
>
> > Casey,
> >
> > Thanks for the quick reply, love your work by the way!
> >
> > When I try to upload the parser I am getting a stack trace like this:
> > 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager -
> > State change: CONNECTED
> > java.lang.IllegalStateException: Unable to instantiate connector: class
> > not
> > found
> > at
> > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > ReflectionUtils.java:56)
> > at
> > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> createParserBolt(
> > ParserTopologyBuilder.java:155)
> > at
> > org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> > ParserTopologyBuilder.java:94)
> > at
> > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > main(ParserTopologyCLI.java:298)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:264)
> > at
> > org.apache.metron.common.utils.ReflectionUtils.createInstance(
> > ReflectionUtils.java:53)
> > ... 3 more
> >
> > The storm supervisor log is saying the some of the prcosses aren't
> > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > and is throwing TimoutExceptions, I believe that is due to the parser.
> >
> > Without the parser though (when troubleshooting the enrichment config
> from
> > #1) I don't receive and errors from storm and the enrichment bolts seem
> to
> > be splitting the data but writer bolt emits 0 everytime.
> > We are able to use the built-in hostname enrichment but the custom one I
> > built (which will eventually be converted into asset discovery
> enrichment)
> > doesn't seem to be writing to elastic search. Do I need to setup a new
> > index template to receive the data from the new enrichment config? Or
> > should I be looking at creating a new spout / bolt to transfer the data?
> >
> > Regards,
> >
> > Tyler
> >
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Flyball Labs
> >
> > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella  wrote:
> >
> > > First off Tyler, thanks for using Metron.
> > >
> > > Do you have any errors or stack traces that are being thrown (keeping
> in
> > > mind that in storm, they may be in the storm logs (/var/log/storm on
> the
> > > supervisor nodes)?
> > >
> > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore 
> > wrote:
> > >
> > > > Hey everyone,
> > > >
> > > > I've had a few sticking points that I encountered while trying to
> > create
> > > > some custom solutions using the Metron platform and could use some
> > > > guidance.
> > > >
> > > > 1) My custom enrichment config is not writing to elasticsearch or may
> > be
> > > > configured improperly.
> > > >
> > > > My extractor config:
> > > > {
> > > >   "config" : {
> > > > "columns" : {
> > > >  "ip" : 0,
> > > >  "host" : 1
> > > > },
> > > >  "indicator_column" : "ip",
> > > >  "type" : "hostname",
> > > >  "separator" : ","
> > > >   },
> > > >   "extractor" : "CSV"
> > > > }
> > > >
> > > > My enrichment config:
> > > > {
> > > >   "zkQuorum" : "node1:2181",
> > > >   "sensorToFieldList" : {
> > > >  "bro" : {
> > > >"type" : "ENRICHMENT",
> > > >"fieldToEnrichmentTypes" : {
> > > >  "ip_src_addr" : ["hostname"],
> > > >  "ip_dst_addr" : ["hostname"]
> > > >  }
> > > >   }
> > > >}
> > > > }
> > > >
> > > > A sample of the data i'm uploading:
> > > > 0.0.0.0, "IGMP"
> > > > 10.113.145.135, "GLAZER"
> > > > 10.113.145.137, "GLAZER"
> > > > 10.113.145.138, "GLAZER"
> > > >
> > > > i'm uploading to zookeeper using the following command:
> > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
> > > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c
> > > hosts
> > > > -e hostname_extractor_config.json
> > > 

Re: Help with custom enrichment / parser

[Casey Stella]

Ah, so quick feedback here, that class path has changed from
org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter

There is probably some outdated documentation somewhere, would you mind
pointing out where you got that one?

Casey

On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore  wrote:

> Casey,
>
> Thanks for the quick reply, love your work by the way!
>
> When I try to upload the parser I am getting a stack trace like this:
> 15:43:33.182 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager -
> State change: CONNECTED
> java.lang.IllegalStateException: Unable to instantiate connector: class
> not
> found
> at
> org.apache.metron.common.utils.ReflectionUtils.createInstance(
> ReflectionUtils.java:56)
> at
> org.apache.metron.parsers.topology.ParserTopologyBuilder.createParserBolt(
> ParserTopologyBuilder.java:155)
> at
> org.apache.metron.parsers.topology.ParserTopologyBuilder.build(
> ParserTopologyBuilder.java:94)
> at
> org.apache.metron.parsers.topology.ParserTopologyCLI.
> main(ParserTopologyCLI.java:298)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.metron.common.utils.ReflectionUtils.createInstance(
> ReflectionUtils.java:53)
> ... 3 more
>
> The storm supervisor log is saying the some of the prcosses aren't
> starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> and is throwing TimoutExceptions, I believe that is due to the parser.
>
> Without the parser though (when troubleshooting the enrichment config from
> #1) I don't receive and errors from storm and the enrichment bolts seem to
> be splitting the data but writer bolt emits 0 everytime.
> We are able to use the built-in hostname enrichment but the custom one I
> built (which will eventually be converted into asset discovery enrichment)
> doesn't seem to be writing to elastic search. Do I need to setup a new
> index template to receive the data from the new enrichment config? Or
> should I be looking at creating a new spout / bolt to transfer the data?
>
> Regards,
>
> Tyler
>
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>
> On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella  wrote:
>
> > First off Tyler, thanks for using Metron.
> >
> > Do you have any errors or stack traces that are being thrown (keeping in
> > mind that in storm, they may be in the storm logs (/var/log/storm on the
> > supervisor nodes)?
> >
> > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore 
> wrote:
> >
> > > Hey everyone,
> > >
> > > I've had a few sticking points that I encountered while trying to
> create
> > > some custom solutions using the Metron platform and could use some
> > > guidance.
> > >
> > > 1) My custom enrichment config is not writing to elasticsearch or may
> be
> > > configured improperly.
> > >
> > > My extractor config:
> > > {
> > >   "config" : {
> > > "columns" : {
> > >  "ip" : 0,
> > >  "host" : 1
> > > },
> > >  "indicator_column" : "ip",
> > >  "type" : "hostname",
> > >  "separator" : ","
> > >   },
> > >   "extractor" : "CSV"
> > > }
> > >
> > > My enrichment config:
> > > {
> > >   "zkQuorum" : "node1:2181",
> > >   "sensorToFieldList" : {
> > >  "bro" : {
> > >"type" : "ENRICHMENT",
> > >"fieldToEnrichmentTypes" : {
> > >  "ip_src_addr" : ["hostname"],
> > >  "ip_dst_addr" : ["hostname"]
> > >  }
> > >   }
> > >}
> > > }
> > >
> > > A sample of the data i'm uploading:
> > > 0.0.0.0, "IGMP"
> > > 10.113.145.135, "GLAZER"
> > > 10.113.145.137, "GLAZER"
> > > 10.113.145.138, "GLAZER"
> > >
> > > i'm uploading to zookeeper using the following command:
> > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
> > > hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c
> > hosts
> > > -e hostname_extractor_config.json
> > >
> > > 2) We eventually want to parse this data as a live stream but the
> parser
> > > errors out when I try sending data in. Here is the parser config:
> > > {
> > >   "parserClassName" : "org.apache.metron.parsers.csv.CSVParser",
> > >   "writerClassName" :
> > > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
> > >   "sensorTopic":"hostname",
> > >   "parserConfig":
> > >   {
> > > "shew.table" : "enrichment",
> > > "shew.cf" : "hosts",
> > > "shew.keyColumns" : "ip",
> > > "shew.enrichmentType" : "hostname",
> > > "columns" : {
> > >   "ip" : 0,
> > >   

Re: Help with custom enrichment / parser

[Casey Stella]

First off Tyler, thanks for using Metron.

Do you have any errors or stack traces that are being thrown (keeping in
mind that in storm, they may be in the storm logs (/var/log/storm on the
supervisor nodes)?

On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore  wrote:

> Hey everyone,
>
> I've had a few sticking points that I encountered while trying to create
> some custom solutions using the Metron platform and could use some
> guidance.
>
> 1) My custom enrichment config is not writing to elasticsearch or may be
> configured improperly.
>
> My extractor config:
> {
>   "config" : {
> "columns" : {
>  "ip" : 0,
>  "host" : 1
> },
>  "indicator_column" : "ip",
>  "type" : "hostname",
>  "separator" : ","
>   },
>   "extractor" : "CSV"
> }
>
> My enrichment config:
> {
>   "zkQuorum" : "node1:2181",
>   "sensorToFieldList" : {
>  "bro" : {
>"type" : "ENRICHMENT",
>"fieldToEnrichmentTypes" : {
>  "ip_src_addr" : ["hostname"],
>  "ip_dst_addr" : ["hostname"]
>  }
>   }
>}
> }
>
> A sample of the data i'm uploading:
> 0.0.0.0, "IGMP"
> 10.113.145.135, "GLAZER"
> 10.113.145.137, "GLAZER"
> 10.113.145.138, "GLAZER"
>
> i'm uploading to zookeeper using the following command:
> /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
> hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c hosts
> -e hostname_extractor_config.json
>
> 2) We eventually want to parse this data as a live stream but the parser
> errors out when I try sending data in. Here is the parser config:
> {
>   "parserClassName" : "org.apache.metron.parsers.csv.CSVParser",
>   "writerClassName" :
> "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
>   "sensorTopic":"hostname",
>   "parserConfig":
>   {
> "shew.table" : "enrichment",
> "shew.cf" : "hosts",
> "shew.keyColumns" : "ip",
> "shew.enrichmentType" : "hostname",
> "columns" : {
>   "ip" : 0,
>   "host" : 1
> }
>   }
> }
>
> 3) We will be moving from replay to using kafka-python for sending data
> captures and I am able to send bytes to a new topic, but when I try using
> the json serializer via kafka producer my program exits without error and
> no data is sent.
> Here is the section of the python code i'm having trouble with:
>
> producer = KafkaProducer(bootstrap_servers='50.253.243.17:6667',
> value_serializer=lambda m: json.dumps(m).encode('ascii'), api_version=(0,
> 9))
>
> for _ in range(100):
> producer.send('pcap', {'key': 'value'})
> producer.flush()
>
> If anyone could point me in the right direction that would be great!! I'm
> not sure if the first 2 problems are related to indexing or maybe I need to
> create a bolt to pass on the data in storm?
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Flyball Labs
>

Help with custom enrichment / parser

[Tyler Moore]

Hey everyone,

I've had a few sticking points that I encountered while trying to create
some custom solutions using the Metron platform and could use some guidance.

1) My custom enrichment config is not writing to elasticsearch or may be
configured improperly.

My extractor config:
{
  "config" : {
"columns" : {
 "ip" : 0,
 "host" : 1
},
 "indicator_column" : "ip",
 "type" : "hostname",
 "separator" : ","
  },
  "extractor" : "CSV"
}

My enrichment config:
{
  "zkQuorum" : "node1:2181",
  "sensorToFieldList" : {
 "bro" : {
   "type" : "ENRICHMENT",
   "fieldToEnrichmentTypes" : {
 "ip_src_addr" : ["hostname"],
 "ip_dst_addr" : ["hostname"]
 }
  }
   }
}

A sample of the data i'm uploading:
0.0.0.0, "IGMP"
10.113.145.135, "GLAZER"
10.113.145.137, "GLAZER"
10.113.145.138, "GLAZER"

i'm uploading to zookeeper using the following command:
/usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
hostname_enrichment_config.json -i hostname_ref.csv -t enrichment -c hosts
-e hostname_extractor_config.json

2) We eventually want to parse this data as a live stream but the parser
errors out when I try sending data in. Here is the parser config:
{
  "parserClassName" : "org.apache.metron.parsers.csv.CSVParser",
  "writerClassName" :
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
  "sensorTopic":"hostname",
  "parserConfig":
  {
"shew.table" : "enrichment",
"shew.cf" : "hosts",
"shew.keyColumns" : "ip",
"shew.enrichmentType" : "hostname",
"columns" : {
  "ip" : 0,
  "host" : 1
}
  }
}

3) We will be moving from replay to using kafka-python for sending data
captures and I am able to send bytes to a new topic, but when I try using
the json serializer via kafka producer my program exits without error and
no data is sent.
Here is the section of the python code i'm having trouble with:

producer = KafkaProducer(bootstrap_servers='50.253.243.17:6667',
value_serializer=lambda m: json.dumps(m).encode('ascii'), api_version=(0,
9))

for _ in range(100):
producer.send('pcap', {'key': 'value'})
producer.flush()

If anyone could point me in the right direction that would be great!! I'm
not sure if the first 2 problems are related to indexing or maybe I need to
create a bolt to pass on the data in storm?

Regards,

Tyler Moore
Software Engineer
Flyball Labs