Re: Metron correlation capabilities

[James Sirota]

Hi Carolyn,

The correlation capabilities are done via ES queries and are visualized in 
Kibana.  Metron's Stallar tranformation, enrichment, and threat intel 
correlation capabilities allow you to pull up all relevant data and context for 
all telemetries ingested with a single query.  Metron's PCAP services then 
allow you to tie it in with the underlying packet capture.  

With respect to ML analytics, Metron has Model as a Service that allows the 
creation of stand alone models, ensembles of models, or chaining of multiple 
models and provides model provisioning, discovery, and scoring.  If your 
customer has pre-existing analytics packs they wish to run on top of Metron 
please refer them to the boards and we will help them get the models to run on 
MaaS.  

Thanks,
James

05.10.2016, 14:41, "Carolyn Duby" :
> Does Metron have any correlation capabilities that we can demonstrate now?
>
> Are any analytics packs ready to show?
>
> We have a customer asking about these capabilities.
>
> Thanks
> Carolyn

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Metron correlation capabilities

[Carolyn Duby]

Does Metron have any correlation capabilities that we can demonstrate now?

Are any analytics packs ready to show?

We have a customer asking about these capabilities.

Thanks
Carolyn

Re: [DISCUSS] Storm worker processes not properly terminating on restart (METRON-485)

[Otto Fowler]

What you would have to do to try this is to edit
/usr/local/monit/stop_bro_topology.sh and add in a -w parameter.
 to read:

storm kill bro -w 60

save that and then try to stop if from monit again.

If that works, then we will know.

On October 5, 2016 at 15:18:43, zeo...@gmail.com (zeo...@gmail.com) wrote:

topology.message.timeout.secs is set to 30 according to ambari.

What you requested is essentially what I did before, except instead of
using `storm kill` I used monit to kill all of the topologies, and I waited
multiple minutes before stopping nimbus/supervisor/drpc, then waited a
couple more minutes before grabbing the prior metrics/processes. I've seen
this multiple times across the past week and each time I see it I shut
everything down, do a `kill -9` on the storm processes left behind, make
sure no processes are running as the storm user on my entire cluster, then
bring things up in ambari, then monit once ambari says
nimbus/supervisor/drpc is up.

Jon

On Wed, Oct 5, 2016 at 3:09 PM P. Taylor Goetz  wrote:

> To build on what Otto said:
>
> What’s your topology.message.timeout.secs configuration value? The
default
> is 30.
>
> If you issue the `storm kill` command without the -w option, Storm will
> first deactivate the topology, then wait topology.message.timeout.secs
> before actually shutting down the topology.
>
> Also, if you have topologies deployed, and stop ninmubs, supervisors,
etc.
> the workers may continue to run (depending on how you have monit setup).
>
> Can you try the following:
>
> 1. Kill the topology and wait topology.message.timeout.sec so Storm
> properly shuts down the topology.
> 2. Stop nimbus/supervisor/etc.
> 3. Check to see if you still have threads left behind.
>
>
> -Taylor
>
> > On Oct 5, 2016, at 2:31 PM, Otto Fowler 
wrote:
> >
> > The shutdown topology scripts only do a storm kill .
> > They probably need to -w WAITTIME.
> >
> >
> > --
> >
> > Sent with Airmail
> >
> > On October 5, 2016 at 13:53:33, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > Hi All,
> >
> > While tuning Metron I have been doing a large amount of storm topology
> > restarts, as I tune the parallelism and memory use of bolts. I have
> > noticed multiple times that some threads will be left behind on
restart,
> > which eventually causes all topologies to fail to start due to
> > "java.lang.OutOfMemoryError: unable to create new native thread". I am
> > only using the bro, indexing, and enrichments topologies right now, and
I
> > have seen all of them leave hanging processes behind. Has anybody else
> > seen anything like this?
> >
> > I'm exclusively using monit to do topology restarts, and Ambari to
> restart
> > supervisors/nimbus/UI/DRPC processes. When it is currently registering
as
> > stopped in both monit and Ambari, I see the following threads left
> behind:
> >> [jzeolla@node1 storm]$ ps -austorm | wc -l
> >> 6
> >> [jzeolla@node2 storm]$ ps -austorm | wc -l
> >> 12
> >> [jzeolla@node3 storm]$ ps -austorm | wc -l
> >> 9
> >> [jzeolla@node4 storm]$ ps -austorm | wc -l
> >> 11
> >> [jzeolla@node5 storm]$ ps -austorm | wc -l
> >> 9
> >
> > Some sample processes:
> >> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44
> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer
> > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm
> > -Dstorm.log.dir=/var/log/storm
> >
>
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib

> >
> > -Dstorm.conf.file= -cp
> >
>
/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/

> >
> > 2.4.0.
> >
>
0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/

> >
> > 2.4.0.
> >
>

Re: Name conventions for parsers

[James Sirota]

Well...a product can produce multiple types of telemetry.  There are also 
multiple versions of products that produce the same telemetry, but in different 
formats (string, json, xml, etc).  so maybe vendor_telemetry_version_format?

05.10.2016, 11:14, "Carolyn Duby" :
> It would be helpful to have the function as well as we get more storm 
> components. For example vendor_product_parse
>
> On 10/5/16, 1:43 PM, "James Sirota"  wrote:
>
>> I agree. Would you like to put forth a recommendation for a naming 
>> convention?
>>
>> 05.10.2016, 10:33, "Simon Elliston Ball" :
>>>  At present we do not have a formal convention. Many organizations will no 
>>> doubt want to create their own conventions to match existing naming 
>>> methodologies.
>>>
>>>  However, it seems like an excellent idea to at least produce some 
>>> community driven recommendations for a standard baseline those without 
>>> strong existing methods could adopt.
>>>
>>>  I like your vendor-product approach, but would consider adding something 
>>> around model / series / version to that. Does anyone have any thoughts on 
>>> how such a taxonomy would work best?
>>>
>>>  Simon
>>>
   On 5 Oct 2016, at 18:22, Vladimir Shlyakhtin 
  wrote:

   Hi

   Does Metron have any recommendation for name convention for parsers? 
 Like vendor-product.

   Thanks

   - Vladimir
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [DISCUSS] Storm worker processes not properly terminating on restart (METRON-485)

[Otto Fowler]

If you can reproduce, maybe you can modify the scripts and see if that
fixes it?

On October 5, 2016 at 14:31:30, Otto Fowler (ottobackwa...@gmail.com) wrote:

The shutdown topology scripts only do a storm kill .
They probably need to -w WAITTIME.


-- 

Sent with Airmail

On October 5, 2016 at 13:53:33, zeo...@gmail.com (zeo...@gmail.com) wrote:

Hi All,

While tuning Metron I have been doing a large amount of storm topology
restarts, as I tune the parallelism and memory use of bolts. I have
noticed multiple times that some threads will be left behind on restart,
which eventually causes all topologies to fail to start due to
"java.lang.OutOfMemoryError: unable to create new native thread". I am
only using the bro, indexing, and enrichments topologies right now, and I
have seen all of them leave hanging processes behind. Has anybody else
seen anything like this?

I'm exclusively using monit to do topology restarts, and Ambari to restart
supervisors/nimbus/UI/DRPC processes. When it is currently registering as
stopped in both monit and Ambari, I see the following threads left behind:
> [jzeolla@node1 storm]$ ps -austorm | wc -l
> 6
> [jzeolla@node2 storm]$ ps -austorm | wc -l
> 12
> [jzeolla@node3 storm]$ ps -austorm | wc -l
> 9
> [jzeolla@node4 storm]$ ps -austorm | wc -l
> 11
> [jzeolla@node5 storm]$ ps -austorm | wc -l
> 9

Some sample processes:
> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44
/usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer
-Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm
-Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp
/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/
2.4.0.
0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/
2.4.0.
0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/
2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf
-Xmx128m -Dlogfile.name=logviewer.log
-Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml
backtype.storm.daemon.logviewer
> storm 32072 0.0 0.1 37337056 177112 ? Sl Oct02 2:02
/usr/jdk64/jdk1.8.0_60/bin/java -cp

Re: [DISCUSS] Storm worker processes not properly terminating on restart (METRON-485)

[Otto Fowler]

The shutdown topology scripts only do a storm kill .
They probably need to -w WAITTIME.


-- 

Sent with Airmail

On October 5, 2016 at 13:53:33, zeo...@gmail.com (zeo...@gmail.com) wrote:

Hi All,

While tuning Metron I have been doing a large amount of storm topology
restarts, as I tune the parallelism and memory use of bolts. I have
noticed multiple times that some threads will be left behind on restart,
which eventually causes all topologies to fail to start due to
"java.lang.OutOfMemoryError: unable to create new native thread". I am
only using the bro, indexing, and enrichments topologies right now, and I
have seen all of them leave hanging processes behind. Has anybody else
seen anything like this?

I'm exclusively using monit to do topology restarts, and Ambari to restart
supervisors/nimbus/UI/DRPC processes. When it is currently registering as
stopped in both monit and Ambari, I see the following threads left behind:
> [jzeolla@node1 storm]$ ps -austorm | wc -l
> 6
> [jzeolla@node2 storm]$ ps -austorm | wc -l
> 12
> [jzeolla@node3 storm]$ ps -austorm | wc -l
> 9
> [jzeolla@node4 storm]$ ps -austorm | wc -l
> 11
> [jzeolla@node5 storm]$ ps -austorm | wc -l
> 9

Some sample processes:
> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44
/usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer
-Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm
-Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib

-Dstorm.conf.file= -cp
/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/

2.4.0.
0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/

2.4.0.
0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/

2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf

-Xmx128m -Dlogfile.name=logviewer.log
-Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml
backtype.storm.daemon.logviewer
> storm 32072 0.0 0.1 37337056 177112 ? Sl Oct02 2:02
/usr/jdk64/jdk1.8.0_60/bin/java -cp
/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/

2.4.0.
0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/

Re: Name conventions for parsers

[Carolyn Duby]

It would be helpful to have the function as well as we get more storm 
components.  For example vendor_product_parse



On 10/5/16, 1:43 PM, "James Sirota"  wrote:

>I agree. Would you like to put forth a recommendation for a naming convention?
>
>05.10.2016, 10:33, "Simon Elliston Ball" :
>> At present we do not have a formal convention. Many organizations will no 
>> doubt want to create their own conventions to match existing naming 
>> methodologies.
>>
>> However, it seems like an excellent idea to at least produce some community 
>> driven recommendations for a standard baseline those without strong existing 
>> methods could adopt.
>>
>> I like your vendor-product approach, but would consider adding something 
>> around model / series / version to that. Does anyone have any thoughts on 
>> how such a taxonomy would work best?
>>
>> Simon
>>
>>>  On 5 Oct 2016, at 18:22, Vladimir Shlyakhtin 
>>>  wrote:
>>>
>>>  Hi
>>>
>>>  Does Metron have any recommendation for name convention for parsers? Like 
>>> vendor-product.
>>>
>>>  Thanks
>>>
>>>  - Vladimir
>
>--- 
>Thank you,
>
>James Sirota
>PPMC- Apache Metron (Incubating)
>jsirota AT apache DOT org
>

[DISCUSS] Storm worker processes not properly terminating on restart (METRON-485)

[zeo...@gmail.com]

Hi All,

While tuning Metron I have been doing a large amount of storm topology
restarts, as I tune the parallelism and memory use of bolts.  I have
noticed multiple times that some threads will be left behind on restart,
which eventually causes all topologies to fail to start due to
"java.lang.OutOfMemoryError: unable to create new native thread".  I am
only using the bro, indexing, and enrichments topologies right now, and I
have seen all of them leave hanging processes behind.  Has anybody else
seen anything like this?

I'm exclusively using monit to do topology restarts, and Ambari to restart
supervisors/nimbus/UI/DRPC processes.  When it is currently registering as
stopped in both monit and Ambari, I see the following threads left behind:
> [jzeolla@node1 storm]$ ps -austorm | wc -l
> 6
> [jzeolla@node2 storm]$ ps -austorm | wc -l
> 12
> [jzeolla@node3 storm]$ ps -austorm | wc -l
> 9
> [jzeolla@node4 storm]$ ps -austorm | wc -l
> 11
> [jzeolla@node5 storm]$ ps -austorm | wc -l
> 9

Some sample processes:
> storm14257  0.1  0.2 6279184 293912 ?  Sl   Oct03   3:44
/usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer
-Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm
-Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp
/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/
2.4.0.
0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/
2.4.0.
0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/
2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf
-Xmx128m -Dlogfile.name=logviewer.log
-Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml
backtype.storm.daemon.logviewer
> storm32072  0.0  0.1 37337056 177112 ? Sl   Oct02   2:02
/usr/jdk64/jdk1.8.0_60/bin/java -cp
/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/
2.4.0.
0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/
2.4.0.

Re: Name conventions for parsers

[James Sirota]

I agree. Would you like to put forth a recommendation for a naming convention?

05.10.2016, 10:33, "Simon Elliston Ball" :
> At present we do not have a formal convention. Many organizations will no 
> doubt want to create their own conventions to match existing naming 
> methodologies.
>
> However, it seems like an excellent idea to at least produce some community 
> driven recommendations for a standard baseline those without strong existing 
> methods could adopt.
>
> I like your vendor-product approach, but would consider adding something 
> around model / series / version to that. Does anyone have any thoughts on how 
> such a taxonomy would work best?
>
> Simon
>
>>  On 5 Oct 2016, at 18:22, Vladimir Shlyakhtin 
>>  wrote:
>>
>>  Hi
>>
>>  Does Metron have any recommendation for name convention for parsers? Like 
>> vendor-product.
>>
>>  Thanks
>>
>>  - Vladimir

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [DISCUSS] Active Directory Parser for Metron

[James Sirota]

If you could make a pull request we can take a look at it together.  Be careful 
when posting test data, though.  Make sure your security team looks at it 
first.  Ideally we'd need a few sources of AD logs to make sure that the parser 
is generic enough.  

05.10.2016, 07:29, "zeo...@gmail.com" :
> How are you currently getting the logs to the parser? Are you adding any
> additional fields?
>
> We use NXLog to send windows logs as syslog and we do some minor transforms
> in order to clean it up, such as substituting tabs with spaces and adding
> the event ID at the end (which isn't there by default). I should be able
> to provide some cleaned samples from my environment.
>
> Jon
>
> On Wed, Oct 5, 2016 at 10:17 AM Tseytlin, Keren <
> keren.tseyt...@capitalone.com> wrote:
>
>>  Cool. I can work on it in my spare time. Additional log files would be
>>  incredibly useful, or else this parser will be very specific to our use
>>  case – which is unlikely to be particularly useful to the larger population.
>>
>>  Keren
>>
>>  On 10/5/16, 9:53 AM, "Nick Allen"  wrote:
>>
>>  That would be great, Keren. Let us know what you need to make that
>>  happen.
>>
>>  I think it would also be useful, if we could get anonymized test data
>>  from
>>  multiple organizations using Active Directory. That will help us
>>  ensure
>>  that the AD parser is broadly useful and not specific to one
>>  organization's
>>  AD installation. If anyone else has AD logs that they could anonymize
>>  and
>>  contribute, please chime in!
>>
>>  On Wed, Oct 5, 2016 at 9:39 AM, Tseytlin, Keren <
>>  keren.tseyt...@capitalone.com> wrote:
>>
>>  > Hi All,
>>  >
>>  > We have an active directory parser that is currently in production.
>>  We
>>  > would be happy to contribute it and work with whoever to make it
>>  generic ☺
>>  >
>>  > Best,
>>  > Keren
>>  >
>>  > On 10/3/16, 5:58 PM, "zeo...@gmail.com"  wrote:
>>  >
>>  > +1 in need of. No current effort because it is not our primary
>>  kerb
>>  > realm,
>>  > but we could use it.
>>  >
>>  > On Mon, Oct 3, 2016, 17:18 James Sirota 
>>  wrote:
>>  >
>>  > > I've seen traffic come through about multiple efforts for
>>  writing
>>  > the AD
>>  > > parser for Metron. I'd like to consolidate these efforts so
>>  that we
>>  > can
>>  > > come up with a generic parser that is suitable for everyone's
>>  needs
>>  > and
>>  > > that we don't duplicate effort. Please post to this thread if
>>  you
>>  > are
>>  > > working or are in need of the AD parser. We can then throw a
>>  > working group
>>  > > together and get the parser written and tested with everyone's
>>  > telemetry.
>>  > > Also, please indicate if you are able to provide sample
>>  (anonymized)
>>  > logs.
>>  > > If you are getting these logs from your corporate environment
>>  please
>>  > check
>>  > > with your security office first prior to posting them.
>>  > >
>>  > > ---
>>  > > Thank you,
>>  > >
>>  > > James Sirota
>>  > > PPMC- Apache Metron (Incubating)
>>  > > jsirota AT apache DOT org
>>  > >
>>  > --
>>  >
>>  > Jon
>>  >
>>  >
>>  > 
>>  >
>>  > The information contained in this e-mail is confidential and/or
>>  > proprietary to Capital One and/or its affiliates and may only be used
>>  > solely in performance of work or services for Capital One. The
>>  information
>>  > transmitted herewith is intended only for use by the individual or
>>  entity
>>  > to which it is addressed. If the reader of this message is not the
>>  intended
>>  > recipient, you are hereby notified that any review, retransmission,
>>  > dissemination, distribution, copying or other use of, or taking of
>>  any
>>  > action in reliance upon this information is strictly prohibited. If
>>  you
>>  > have received this communication in error, please contact the sender
>>  and
>>  > delete the material from your computer.
>>  >
>>
>>  --
>>  Nick Allen 
>>
>>  
>>
>>  The information contained in this e-mail is confidential and/or
>>  proprietary to Capital One and/or its affiliates and may only be used
>>  solely in performance of work or services for Capital One. The information
>>  transmitted herewith is intended only for use by the individual or entity
>>  to which it is addressed. If the reader of this message is not the intended
>>  recipient, you are hereby notified that any review, retransmission,
>>  dissemination, distribution, copying or other use of, or taking of any
>>  action in reliance upon this information is strictly prohibited. If 

Name conventions for parsers

[Vladimir Shlyakhtin]

Hi

Does Metron have any recommendation for name convention for parsers? Like 
vendor-product.

Thanks

- Vladimir

Re: [DISCUSS] Active Directory Parser for Metron

[zeo...@gmail.com]

How are you currently getting the logs to the parser?  Are you adding any
additional fields?

We use NXLog to send windows logs as syslog and we do some minor transforms
in order to clean it up, such as substituting tabs with spaces and adding
the event ID at the end (which isn't there by default).  I should be able
to provide some cleaned samples from my environment.

Jon

On Wed, Oct 5, 2016 at 10:17 AM Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Cool. I can work on it in my spare time. Additional log files would be
> incredibly useful, or else this parser will be very specific to our use
> case – which is unlikely to be particularly useful to the larger population.
>
> Keren
>
> On 10/5/16, 9:53 AM, "Nick Allen"  wrote:
>
> That would be great, Keren.  Let us know what you need to make that
> happen.
>
> I think it would also be useful, if we could get anonymized test data
> from
> multiple organizations using Active Directory.  That will help us
> ensure
> that the AD parser is broadly useful and not specific to one
> organization's
> AD installation.  If anyone else has AD logs that they could anonymize
> and
> contribute, please chime in!
>
> On Wed, Oct 5, 2016 at 9:39 AM, Tseytlin, Keren <
> keren.tseyt...@capitalone.com> wrote:
>
> > Hi All,
> >
> > We have an active directory parser that is currently in production.
> We
> > would be happy to contribute it and work with whoever to make it
> generic ☺
> >
> > Best,
> > Keren
> >
> > On 10/3/16, 5:58 PM, "zeo...@gmail.com"  wrote:
> >
> > +1 in need of.  No current effort because it is not our primary
> kerb
> > realm,
> > but we could use it.
> >
> > On Mon, Oct 3, 2016, 17:18 James Sirota 
> wrote:
> >
> > > I've seen traffic come through about multiple efforts for
> writing
> > the AD
> > > parser for Metron.  I'd like to consolidate these efforts so
> that we
> > can
> > > come up with a generic parser that is suitable for everyone's
> needs
> > and
> > > that we don't duplicate effort.  Please post to this thread if
> you
> > are
> > > working or are in need of the AD parser.  We can then throw a
> > working group
> > > together and get the parser written and tested with everyone's
> > telemetry.
> > > Also, please indicate if you are able to provide sample
> (anonymized)
> > logs.
> > > If you are getting these logs from your corporate environment
> please
> > check
> > > with your security office first prior to posting them.
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> > --
> >
> > Jon
> >
> >
> > 
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or
> entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of
> any
> > action in reliance upon this information is strictly prohibited. If
> you
> > have received this communication in error, please contact the sender
> and
> > delete the material from your computer.
> >
>
>
>
> --
> Nick Allen 
>
>
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
-- 

Jon

Re: [DISCUSS] Active Directory Parser for Metron

[Nick Allen]

There is probably also some general "meta-information" that we should start
to collect with each parser.  I don't know what all those questions should
be, but just keep that in mind.  For example, What version of AD was the
parser tested with.  How was AD configured?  How was the data transported
from AD? etc

On Wed, Oct 5, 2016 at 10:17 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Cool. I can work on it in my spare time. Additional log files would be
> incredibly useful, or else this parser will be very specific to our use
> case – which is unlikely to be particularly useful to the larger population.
>
> Keren
>
> On 10/5/16, 9:53 AM, "Nick Allen"  wrote:
>
> That would be great, Keren.  Let us know what you need to make that
> happen.
>
> I think it would also be useful, if we could get anonymized test data
> from
> multiple organizations using Active Directory.  That will help us
> ensure
> that the AD parser is broadly useful and not specific to one
> organization's
> AD installation.  If anyone else has AD logs that they could anonymize
> and
> contribute, please chime in!
>
> On Wed, Oct 5, 2016 at 9:39 AM, Tseytlin, Keren <
> keren.tseyt...@capitalone.com> wrote:
>
> > Hi All,
> >
> > We have an active directory parser that is currently in production.
> We
> > would be happy to contribute it and work with whoever to make it
> generic ☺
> >
> > Best,
> > Keren
> >
> > On 10/3/16, 5:58 PM, "zeo...@gmail.com"  wrote:
> >
> > +1 in need of.  No current effort because it is not our primary
> kerb
> > realm,
> > but we could use it.
> >
> > On Mon, Oct 3, 2016, 17:18 James Sirota 
> wrote:
> >
> > > I've seen traffic come through about multiple efforts for
> writing
> > the AD
> > > parser for Metron.  I'd like to consolidate these efforts so
> that we
> > can
> > > come up with a generic parser that is suitable for everyone's
> needs
> > and
> > > that we don't duplicate effort.  Please post to this thread if
> you
> > are
> > > working or are in need of the AD parser.  We can then throw a
> > working group
> > > together and get the parser written and tested with everyone's
> > telemetry.
> > > Also, please indicate if you are able to provide sample
> (anonymized)
> > logs.
> > > If you are getting these logs from your corporate environment
> please
> > check
> > > with your security office first prior to posting them.
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> > --
> >
> > Jon
> >
> >
> > 
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or
> entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of
> any
> > action in reliance upon this information is strictly prohibited. If
> you
> > have received this communication in error, please contact the sender
> and
> > delete the material from your computer.
> >
>
>
>
> --
> Nick Allen 
>
>
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>



-- 
Nick Allen 

[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/293

METRON-473 Add LENGTH() To Stellar

Add LENGTH to Stellar string functions and unit tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-473

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/293.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #293


commit 7aaa5d0cd08ae0861601ff2a763f45c71d70dbce
Author: Otto Fowler 
Date:   2016-10-05T14:23:50Z

METRON-473 Add LENGTH() function for getting the length of strings to 
Stellar dsl functions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #292: METRON-171 add .class to .gitignore

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/292

METRON-171 add .class to .gitignore

Add .class files to .gitignore

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/292.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #292


commit ab74b2cf223787d8d5f6a3c84be88fe815478b34
Author: Otto Fowler 
Date:   2016-10-05T13:42:04Z

METRON-171 add .class to .gitignore




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Complete steps to add a new parser

[Nick Allen]

Your changes to add the 'laa' sensor look "sensible."  Nice.

On Sat, Oct 1, 2016 at 11:59 PM, Otto Fowler 
wrote:

> I have been able to add a new parser to the the deployment, and have the
> cluster fully deploy successfully.  After I was able to push data to kafka
> from HDF and get it all indexed.
>
> Unlike quick dev and full, no problems getting the storm ports correct
> after deployment to my small cluster config.
>
> It looks to me that the steps I took to integrate the parser worked, but I
> still may have missed something.  One thing that I know I missed was
> modifying the dashboard - adding the saved searches and integrating them
> with the visualizations.
>
> Here is a gist off a patch of my changes.  The patch in the gist has been
> modified - so I don’t think it will apply for you.  I removed proprietary
> field names ( the stellar config, the enrichment hosts, es index template ).
>
> I hope what is there is enough for you to verify, correct what I have done.
>
> https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84
>
>
>
> On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Thanks Nick,
> That is some of the stuff that I have found trying to track down the
> deploy bits of the existing parsers, but I don’t want to miss anything, so
> I’d like some guidence.  Right now, I’m OK with doing it all inside the
> metron ansible base.  I expect that once I get it working and wrap my head
> around it I’ll have some ideas that I’ll float around improving this area,
> some use cases to propose that would possibly be external to the main
> deployment, or additive. First thing first is understanding all the points
> in doing it the hard way ;)
>
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 12:41:10, Nick Allen (n...@nickallen.org) wrote:
>
> Hi Otto -
>
> I would agree with you. We do not have documentation that describes how to
> 'permanently install' a new parser.. Your contribution would be highly
> appreciated in this area.
>
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code. An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser. But of course, if we can find
> ways to make this task easier, we will.
>
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
>
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of
> metron-deployment/roles/monit/templates/monit/parsers.monit to add your
> own
> parser definition to Monit.
>
> *Parsers* - The start script in
> `metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
> will give you good hooks into how each of the parsers are started.
>
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`. To understand
> that process, start at `tasks/main.yml`.
>
> I probably missed something, but let me know if you have questions.
>
>
>
>
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler 
> wrote:
>
> > My wish, is that when I do an ansible-playbook -v -i {my configuration}
> > metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> > my parser / topology is deployed, started and monitored the same way as
> the
> > current bro, snort, and yaf parsers are.
> >
> > I might be misunderstanding something however. I seems to me that all the
> > examples of adding other parsers are temporary and not permanent because
> > they do not have the full deployment, kind of push the config and run the
> > script and you are going. Am I missing something? Would the squid sample
> > steps result in a parser topology that would survive restarts / reboots
> > etc?
> >
> > On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org)
> > wrote:
> >
> > Just so I completely understand what you are asking for...you want to
> know
> > how to create a new parser topology with the JSON parser and plug it into
> > Monit so you can monitor and restart it on demand?
> >
> > 27.09.2016, 09:03, "Otto Fowler" :
> > > Thanks James,
> > >
> > > I want to deploy an instance of the JSONMapParser into my POC cluster
> and
> > vagrant. I’m trying to work out exactly how to add a new configured
> parser
> > instance to the deployment. I think these instructions would be a good
> > extension to the squid stuff that is already there. If I could get that
> > going and add a new parser all the way through, then maybe I can
> contribute
> > something in that area. The ability to do this will also enable some of
> > the other work you mentioned.
> > >
> > > On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org)
> > wrote:
> > 

Re: [DISCUSS] Active Directory Parser for Metron

[Tseytlin, Keren]

Hi All,

We have an active directory parser that is currently in production. We would be 
happy to contribute it and work with whoever to make it generic ☺

Best,
Keren

On 10/3/16, 5:58 PM, "zeo...@gmail.com"  wrote:

+1 in need of.  No current effort because it is not our primary kerb realm,
but we could use it.

On Mon, Oct 3, 2016, 17:18 James Sirota  wrote:

> I've seen traffic come through about multiple efforts for writing the AD
> parser for Metron.  I'd like to consolidate these efforts so that we can
> come up with a generic parser that is suitable for everyone's needs and
> that we don't duplicate effort.  Please post to this thread if you are
> working or are in need of the AD parser.  We can then throw a working 
group
> together and get the parser written and tested with everyone's telemetry.
> Also, please indicate if you are able to provide sample (anonymized) logs.
> If you are getting these logs from your corporate environment please check
> with your security office first prior to posting them.
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
-- 

Jon




The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.