[GitHub] metron pull request #606: METRON-980: Short circuit operations for Stellar

2017-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/606#discussion_r121049130 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java --- @@ -76,14 +93,80 @@ public Expression(Deque

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/609 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: performance benchmarks on the asa parser

2017-06-08 Thread Ali Nazemian
Hi Simon, We have noticed those issues as well. Can you share the changes you have made? so we can merge it with our version. We have implemented about 40-50 more ciscotags so far. It would be great if we can optimize it and contribute back to the community. However, we may end up reimplement it u

Re: performance benchmarks on the asa parser

2017-06-08 Thread Simon Elliston Ball
I thought about compile of first use and cache as an approach, but decided it would reduce the predictability of latency for a message, which is important in the metron enrichment context. As you say, we could end up growing a large number of Groks, but if the load of compile is all pushed to th

Re: performance benchmarks on the asa parser

2017-06-08 Thread Kyle Richardson
I like the pre-compile idea. One concern is I see the number of grok objects growing over time. This parser does not account for nearly all of the possible ASA message types, currently only the most common ones. Is there a middle ground implementation where we can compile on first use of a grok

[GitHub] metron issue #612: METRON-962: Configuration Based Unit Tests and Add integr...

2017-06-08 Thread kylerichardson
Github user kylerichardson commented on the issue: https://github.com/apache/metron/pull/612 'GrokAsaParser' is legacy OpenSOC I believe. It has been broken for some time and when I rewrote the ASA parser I did it as 'BasicAsaParser'. @justinleet Thanks for cleaning this up! I

Re: performance benchmarks on the asa parser

2017-06-08 Thread Simon Elliston Ball
The changes are pretty simple (pre-compile the grok, duh). Most other grok parser just use a single expression, which is already pre-compiled (/checks assumption in code) so really it’s just the ASA one because of it’s strange two stage grok. Shame, it would have been nice to find some more lo

Re: performance benchmarks on the asa parser

2017-06-08 Thread Otto Fowler
Are these changes that all grok parsers can benefit from? Are your changes to the base classes that they use or asa only? On June 8, 2017 at 20:49:49, Simon Elliston Ball ( si...@simonellistonball.com) wrote: I got mildly interested in parser performance as a result of some recent work on tuni

performance benchmarks on the asa parser

2017-06-08 Thread Simon Elliston Ball
I got mildly interested in parser performance as a result of some recent work on tuning, and did some very quick benchmarking with Predfix on the ASA parser (which I hadn’t really cared about enough due to relatively low volume previously). That said, it’s not exactly perf optimised. 3 runs of

[GitHub] metron issue #609: METRON-987: Allow stellar enrichments to be specified by ...

2017-06-08 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/metron/pull/609 @ottobackwards Yeah, that's why I went with backwards compatibility here, so we could defer that decision until we had a discussion. I think the enrichment topology should really change quite a bit

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r121001410 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview + +Th

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r121000712 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview + +Th

[GitHub] metron issue #610: METRON-877 Extract core implementation and UDF support, c...

2017-06-08 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/610 Don't forget - if you add an rpm, add the rpm to the ambari repo xml config --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If yo

[GitHub] metron issue #610: METRON-877 Extract core implementation and UDF support, c...

2017-06-08 Thread mattf-horton
Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/610 The Metron RPM build is broken with this patch, which interferes with testing. Will have it fixed shortly. --Matt --- If your project is set up for it, you can reply to this email and have you

[GitHub] metron issue #609: METRON-987: Allow stellar enrichments to be specified by ...

2017-06-08 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/609 Do we want to add a version number to these configuration now? Then we can say no version is 0, this is 1 etc. --- If your project is set up for it, you can reply to this email and have you

[GitHub] metron issue #609: METRON-987: Allow stellar enrichments to be specified by ...

2017-06-08 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/609 +1, I was able to run this up, both the smoke test and the adjusted variant --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread mmiklavc
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r120979346 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview + +This d

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread mmiklavc
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r120979059 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview + +This d

[GitHub] metron issue #609: METRON-987: Allow stellar enrichments to be specified by ...

2017-06-08 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/609 @cestella Perfect. Definitely didn't want to imply that belongs in this ticket at all. I'm in full support of it being backwards compatible for now, and keeping the discussion of how we prune ol

[GitHub] metron issue #609: METRON-987: Allow stellar enrichments to be specified by ...

2017-06-08 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/metron/pull/609 @justinleet yes, agreed. I wanted to get this functionality in and have it be backwards compatible for the short term, but ultimately, I think that it's a better approach and we should work toward

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread simonellistonball
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r120961228 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview +

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread simonellistonball
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r120961066 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview +

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r120964479 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide + +## Overview + +This

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/614#discussion_r120964036 --- Diff: metron-platform/Performance-tuning-guide.md --- @@ -0,0 +1,326 @@ +# Metron Performance Tunining Guide --- End diff -- "Tunining

[DISCUSS] Metron performance tuning guide

2017-06-08 Thread Michael Miklavcic
Hey all, I've created a PR for a performance tuning guide based on some recent experiences we had while tuning for 1 gbps traffic. I'd like to open this up for feedback from the community for a month so that we can hopefully get some insights about how others have tuned their platforms. Any and al

[GitHub] metron pull request #614: METRON-992: Create performance tuning guide

2017-06-08 Thread mmiklavc
GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/614 METRON-992: Create performance tuning guide https://issues.apache.org/jira/browse/METRON-992 This guide covers performance tuning the Metron topologies. I will be leaving this up for a mon

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-08 Thread merrimanr
GitHub user merrimanr reopened a pull request: https://github.com/apache/metron/pull/613 METRON-990: Clean up and organize flux properties ## Contributor Comments This PR is mainly a refactor of the enrichment and indexing flux files along with their matching property files. Th

[GitHub] metron pull request #613: METRON-990: Clean up and organize flux properties

2017-06-08 Thread merrimanr
Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/613 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120932742 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java --- @@ -40,10 +40,12 @@ public JSONFromPosit

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120932266 --- Diff: metron-platform/metron-enrichment/README.md --- @@ -71,40 +73,94 @@ The `fieldMap`contents are of interest because they contain the routing and co

[GitHub] metron pull request #606: METRON-980: Short circuit operations for Stellar

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/606#discussion_r120926071 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java --- @@ -76,14 +93,80 @@ public Expression(Deq

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120924836 --- Diff: metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java --- @@ -0,0 +1,134 @@ +

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120924708 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java --- @@ -40,10 +40,12 @@ public JSONFromPositio

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120919459 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java --- @@ -40,10 +40,12 @@ public JSONFromPosit

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120921756 --- Diff: metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java --- @@ -0,0 +1,134 @@

[GitHub] metron issue #609: METRON-987: Allow stellar enrichments to be specified by ...

2017-06-08 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/609 A lot of this brings up the discussions around deprecating / dropping functionality (both for Stellar and in general). It seems like ideally, we'd be deprecating the map functionality in favor of

[GitHub] metron pull request #609: METRON-987: Allow stellar enrichments to be specif...

2017-06-08 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/609#discussion_r120917140 --- Diff: metron-platform/metron-enrichment/README.md --- @@ -71,40 +73,94 @@ The `fieldMap`contents are of interest because they contain the routing and

[GitHub] metron pull request #602: METRON-906: Rest service storm configuration does ...

2017-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/602#discussion_r120890884 --- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml --- @@ -102,7 +102,7 @@ configurations: required_configurations: -

[GitHub] metron pull request #600: METRON-976 KafkaUtils doesn't handle SASL_PLAINTEX...

2017-06-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/600 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: Installation problem with Docker and processor that does not support virtualization

2017-06-08 Thread smlabs
ok > > Il 8 giugno 2017 alle 11.54 "zeo...@gmail.com" ha > scritto: > > Moving to user list > > Jon > > On Thu, Jun 8, 2017, 5:51 AM wrote: > > > > > > Hei Jon, > > > > thank you. > > > > I don't know which are the differences between 0.3.

Re: Installation problem with Docker and processor that does not support virtualization

2017-06-08 Thread zeo...@gmail.com
Moving to user list Jon On Thu, Jun 8, 2017, 5:51 AM wrote: > Hei Jon, > > thank you. > > I don't know which are the differences between 0.3.1 and 0.4.0. > > Unfortunately for me, my CPU does not support virtualization. That means > that I cannot use Docker. > > The only workaround that I foun

Re: Installation problem with Docker and processor that does not support virtualization

2017-06-08 Thread smlabs
Hei Jon, thank you. I don't know which are the differences between 0.3.1 and 0.4.0. Unfortunately for me, my CPU does not support virtualization. That means that I cannot use Docker. The only workaround that I found is to use AWS directly but for me that I have never used Mentor it could be

Re: Help needed in assign a Task

2017-06-08 Thread David Lyle
Added. -D... On Thu, Jun 8, 2017 at 3:33 AM, RaghuMitra Kandikonda < raghumitra@gmail.com> wrote: > Hi Matt, > > Thanks for pointing me in the right direction. > > Can any PMC help me with this?. > > username: raghumitra > email: raghumitra@gmail.com > > -Raghu Mitra > > On Wed, Jun 7,

Re: Installation problem with Docker and processor that does not support virtualization

2017-06-08 Thread zeo...@gmail.com
If I recall properly, 0.3.1 does not require docker yet. That will come with 0.4.0/master. It still does require virtualization, however, to spin up the dev environments (excluding if you were to run it on AWS). Jon On Thu, Jun 8, 2017, 2:03 AM wrote: > Hello, > > I didn't check the BIOS if I

Re: Help needed in assign a Task

2017-06-08 Thread RaghuMitra Kandikonda
Hi Matt, Thanks for pointing me in the right direction. Can any PMC help me with this?. username: raghumitra email: raghumitra@gmail.com -Raghu Mitra On Wed, Jun 7, 2017 at 11:40 PM, Matt Foley wrote: > Hi Raghu, this is appropriate for the ‘dev’ list rather than the ‘user’ > list, since