The sample i'm sending has over logs about 40,000 records so I don't think
that is the issue.
My batch size is 5 and the this is what it looks like when i dump it from
zookeeper
ENRICHMENT Config: bro
{
"index" : "bro",
"batchSize" : 5,
"enrichment" : {
"fieldMap" : {
"geo" : [ "ip_dst_addr", "ip_src_addr" ],
"host" : [ "ip_src_addr", "ip_dst_addr" ],
"hbaseEnrichment" : [ "ip_src_addr", "ip_dst_addr" ]
},
"fieldToTypeMap" : {
"ip_dst_addr" : [ "hostname", "asset" ],
"ip_src_addr" : [ "hostname", "asset" ]
},
"config" : { }
},
"threatIntel" : {
"fieldMap" : {
"hbaseThreatIntel" : [ "ip_src_addr", "ip_dst_addr" ]
},
"fieldToTypeMap" : {
"ip_src_addr" : [ "malicious_ip" ],
"ip_dst_addr" : [ "malicious_ip" ]
},
"config" : { },
"triageConfig" : {
"riskLevelRules" : { },
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
I loaded an extractor config file with it so I'm wondering if that should
have populated the config fields here or maybe I need to add mappings to
the column families in there?
Regards,
Tyler
Regards,
Tyler Moore
Software Engineer
Flyball Labs
On Thu, Nov 3, 2016 at 3:55 PM, Michael Miklavcic <
[email protected]> wrote:
> Not sure about the python-kafka lib issues. Regarding enrichment data
> getting written to ES, how many records have you processed and what is your
> batch size? You might need to write more records or adjust this for the
> values to propagate through. See the "Sensor Enrichment Configuration"
> section -
> https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-enrichment
>
>
> On Thu, Nov 3, 2016 at 1:03 PM, Tyler Moore <[email protected]> wrote:
>
> > Mike,
> >
> > I am using quick-dev vagrant deployment and at the moment testing locally
> > but we plan on having data from remote locations streaming in to be
> parsed.
> > I was able to get the parsers running, thanks to casey, looks like i
> missed
> > an update to the Hbase enrichment writer naming convention.
> > Still working on the enrichment configs though, they aren't throwing any
> > errors and storm says they are emitting data, but not being written to
> > elastic.
> > As well with the python-kafka library, can't figure out why the json
> > serializer isn't working, as long as I have a parser implemented I could
> > forego serializing the data
> > prior to sending to a kafka topic correct??
> >
> > Thanks for all your help thus far!
> >
> > Regards,
> >
> > Tyler
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Flyball Labs
> >
> > On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic <
> > [email protected]> wrote:
> >
> > > Tyler,
> > >
> > > Thanks for the interest in Metron and welcome to the community! :)
> > >
> > > Just curious, what type of environment are you running in? Full cluster
> > or
> > > are you using the full-dev or quick-dev vagrant deployment vagrant
> > scripts?
> > >
> > > Best,
> > > Mike Miklavcic
> > >
> > >
> > > On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore <[email protected]>
> > wrote:
> > >
> > > > Haven't heard of the acronym before, i'm kinda new to the dev game :D
> > > >
> > > > Do you have any idea why my the enriched data isn't being written to
> > > > elasticsearch?
> > > >
> > > > Regards,
> > > >
> > > > Tyler Moore
> > > > Software Engineer
> > > > Flyball Labs
> > > >
> > > > On Thu, Nov 3, 2016 at 12:15 PM, Casey Stella <[email protected]>
> > > wrote:
> > > >
> > > > > Thanks for finding that; I fixed it in the wiki. Isn't OSS
> awesome?
> > ;)
> > > > >
> > > > > On Thu, Nov 3, 2016 at 12:11 PM, Tyler Moore <[email protected]
> >
> > > > wrote:
> > > > >
> > > > > > No problem,
> > > > > >
> > > > > > I was following the Metron application tutorials in the Metron
> > wiki:
> > > > > > https://cwiki.apache.org/confluence/display/METRON/
> > > > > > 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+
> > > > Streaming+Enrichment
> > > > > >
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Tyler Moore
> > > > > > Software Engineer
> > > > > > Flyball Labs
> > > > > >
> > > > > > On Thu, Nov 3, 2016 at 11:59 AM, Casey Stella <
> [email protected]>
> > > > > wrote:
> > > > > >
> > > > > > > Ah, so quick feedback here, that class path has changed from
> > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter to
> > > > > > > org.apache.metron.enrichment.writer.
> SimpleHbaseEnrichmentWriter
> > > > > > >
> > > > > > > There is probably some outdated documentation somewhere, would
> > you
> > > > mind
> > > > > > > pointing out where you got that one?
> > > > > > >
> > > > > > > Casey
> > > > > > >
> > > > > > > On Thu, Nov 3, 2016 at 11:56 AM, Tyler Moore <
> > [email protected]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Casey,
> > > > > > > >
> > > > > > > > Thanks for the quick reply, love your work by the way!
> > > > > > > >
> > > > > > > > When I try to upload the parser I am getting a stack trace
> like
> > > > this:
> > > > > > > > 15:43:33.182 [main-EventThread] INFO o.a.c.f.s.
> > > > > ConnectionStateManager
> > > > > > -
> > > > > > > > State change: CONNECTED
> > > > > > > > java.lang.IllegalStateException: Unable to instantiate
> > > connector:
> > > > > > class
> > > > > > > > not
> > > > > > > > found
> > > > > > > > at
> > > > > > > > org.apache.metron.common.utils.ReflectionUtils.
> createInstance(
> > > > > > > > ReflectionUtils.java:56)
> > > > > > > > at
> > > > > > > > org.apache.metron.parsers.topology.ParserTopologyBuilder.
> > > > > > > createParserBolt(
> > > > > > > > ParserTopologyBuilder.java:155)
> > > > > > > > at
> > > > > > > > org.apache.metron.parsers.topology.
> > ParserTopologyBuilder.build(
> > > > > > > > ParserTopologyBuilder.java:94)
> > > > > > > > at
> > > > > > > > org.apache.metron.parsers.topology.ParserTopologyCLI.
> > > > > > > > main(ParserTopologyCLI.java:298)
> > > > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > > > org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter
> > > > > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:
> 381)
> > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass(
> > Launcher.java:331)
> > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > > > > > > at java.lang.Class.forName0(Native Method)
> > > > > > > > at java.lang.Class.forName(Class.java:264)
> > > > > > > > at
> > > > > > > > org.apache.metron.common.utils.ReflectionUtils.
> createInstance(
> > > > > > > > ReflectionUtils.java:53)
> > > > > > > > ... 3 more
> > > > > > > >
> > > > > > > > The storm supervisor log is saying the some of the prcosses
> > > aren't
> > > > > > > > starting 2016-11-03 15:32:25.730 b.s.d.supervisor [INFO]
> > > > > > > > 9b0734b2-5e5f-4109-aabd-cf343f54e3a4 still hasn't started
> > > > > > > > and is throwing TimoutExceptions, I believe that is due to
> the
> > > > > parser.
> > > > > > > >
> > > > > > > > Without the parser though (when troubleshooting the
> enrichment
> > > > config
> > > > > > > from
> > > > > > > > #1) I don't receive and errors from storm and the enrichment
> > > bolts
> > > > > seem
> > > > > > > to
> > > > > > > > be splitting the data but writer bolt emits 0 everytime.
> > > > > > > > We are able to use the built-in hostname enrichment but the
> > > custom
> > > > > one
> > > > > > I
> > > > > > > > built (which will eventually be converted into asset
> discovery
> > > > > > > enrichment)
> > > > > > > > doesn't seem to be writing to elastic search. Do I need to
> > setup
> > > a
> > > > > new
> > > > > > > > index template to receive the data from the new enrichment
> > > config?
> > > > Or
> > > > > > > > should I be looking at creating a new spout / bolt to
> transfer
> > > the
> > > > > > data?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Tyler
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Tyler Moore
> > > > > > > > Software Engineer
> > > > > > > > Flyball Labs
> > > > > > > >
> > > > > > > > On Thu, Nov 3, 2016 at 9:26 AM, Casey Stella <
> > [email protected]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > First off Tyler, thanks for using Metron.
> > > > > > > > >
> > > > > > > > > Do you have any errors or stack traces that are being
> thrown
> > > > > (keeping
> > > > > > > in
> > > > > > > > > mind that in storm, they may be in the storm logs
> > > (/var/log/storm
> > > > > on
> > > > > > > the
> > > > > > > > > supervisor nodes)?
> > > > > > > > >
> > > > > > > > > On Wed, Nov 2, 2016 at 10:47 PM, Tyler Moore <
> > > > [email protected]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey everyone,
> > > > > > > > > >
> > > > > > > > > > I've had a few sticking points that I encountered while
> > > trying
> > > > to
> > > > > > > > create
> > > > > > > > > > some custom solutions using the Metron platform and could
> > use
> > > > > some
> > > > > > > > > > guidance.
> > > > > > > > > >
> > > > > > > > > > 1) My custom enrichment config is not writing to
> > > elasticsearch
> > > > or
> > > > > > may
> > > > > > > > be
> > > > > > > > > > configured improperly.
> > > > > > > > > >
> > > > > > > > > > My extractor config:
> > > > > > > > > > {
> > > > > > > > > > "config" : {
> > > > > > > > > > "columns" : {
> > > > > > > > > > "ip" : 0,
> > > > > > > > > > "host" : 1
> > > > > > > > > > },
> > > > > > > > > > "indicator_column" : "ip",
> > > > > > > > > > "type" : "hostname",
> > > > > > > > > > "separator" : ","
> > > > > > > > > > },
> > > > > > > > > > "extractor" : "CSV"
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > My enrichment config:
> > > > > > > > > > {
> > > > > > > > > > "zkQuorum" : "node1:2181",
> > > > > > > > > > "sensorToFieldList" : {
> > > > > > > > > > "bro" : {
> > > > > > > > > > "type" : "ENRICHMENT",
> > > > > > > > > > "fieldToEnrichmentTypes" : {
> > > > > > > > > > "ip_src_addr" : ["hostname"],
> > > > > > > > > > "ip_dst_addr" : ["hostname"]
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > A sample of the data i'm uploading:
> > > > > > > > > > 0.0.0.0, "IGMP"
> > > > > > > > > > 10.113.145.135, "GLAZER"
> > > > > > > > > > 10.113.145.137, "GLAZER"
> > > > > > > > > > 10.113.145.138, "GLAZER"
> > > > > > > > > >
> > > > > > > > > > i'm uploading to zookeeper using the following command:
> > > > > > > > > > /usr/metron/0.2.1BETA/bin/flatfile_loader.sh -n
> > > > > > > > > > hostname_enrichment_config.json -i hostname_ref.csv -t
> > > > > enrichment
> > > > > > -c
> > > > > > > > > hosts
> > > > > > > > > > -e hostname_extractor_config.json
> > > > > > > > > >
> > > > > > > > > > 2) We eventually want to parse this data as a live stream
> > but
> > > > the
> > > > > > > > parser
> > > > > > > > > > errors out when I try sending data in. Here is the parser
> > > > config:
> > > > > > > > > > {
> > > > > > > > > > "parserClassName" : "org.apache.metron.parsers.
> > > > csv.CSVParser",
> > > > > > > > > > "writerClassName" :
> > > > > > > > > > "org.apache.metron.writer.hbase.
> > > SimpleHbaseEnrichmentWriter",
> > > > > > > > > > "sensorTopic":"hostname",
> > > > > > > > > > "parserConfig":
> > > > > > > > > > {
> > > > > > > > > > "shew.table" : "enrichment",
> > > > > > > > > > "shew.cf" : "hosts",
> > > > > > > > > > "shew.keyColumns" : "ip",
> > > > > > > > > > "shew.enrichmentType" : "hostname",
> > > > > > > > > > "columns" : {
> > > > > > > > > > "ip" : 0,
> > > > > > > > > > "host" : 1
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > 3) We will be moving from replay to using kafka-python
> for
> > > > > sending
> > > > > > > data
> > > > > > > > > > captures and I am able to send bytes to a new topic, but
> > > when I
> > > > > try
> > > > > > > > using
> > > > > > > > > > the json serializer via kafka producer my program exits
> > > without
> > > > > > error
> > > > > > > > and
> > > > > > > > > > no data is sent.
> > > > > > > > > > Here is the section of the python code i'm having trouble
> > > with:
> > > > > > > > > >
> > > > > > > > > > producer = KafkaProducer(bootstrap_servers='
> > > 50.253.243.17:6667
> > > > ',
> > > > > > > > > > value_serializer=lambda m: json.dumps(m).encode('ascii'),
> > > > > > > > api_version=(0,
> > > > > > > > > > 9))
> > > > > > > > > >
> > > > > > > > > > for _ in range(100):
> > > > > > > > > > producer.send('pcap', {'key': 'value'})
> > > > > > > > > > producer.flush()
> > > > > > > > > >
> > > > > > > > > > If anyone could point me in the right direction that
> would
> > be
> > > > > > great!!
> > > > > > > > I'm
> > > > > > > > > > not sure if the first 2 problems are related to indexing
> or
> > > > > maybe I
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > create a bolt to pass on the data in storm?
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Tyler Moore
> > > > > > > > > > Software Engineer
> > > > > > > > > > Flyball Labs
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
