Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-27 Thread Ali Nazemian
Sorry, Michael. I am having some issues to share any code right now. It seems we need to go through an internal verification of anything we want to share. BTW, the curator script I mentioned is very simple and nothing special. It doesn't worth to waste any time on waiting for it. Cheers, Ali On

[GitHub] metron pull request #850: METRON-1335: Install metron-maas-service RPM as a ...

2017-11-27 Thread anandsubbu
GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/850 METRON-1335: Install metron-maas-service RPM as a part of the full-dev deployment ## Contributor Comments Modified mpack metainfo.xml to install metron-maas-service RPM as well.

[GitHub] metron-bro-plugin-kafka issue #4: METRON-1329: Simplify metron-bro-plugin-ka...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/4 +1 - review looks good - ran through test steps, everything works as described Great job! ---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

2017-11-27 Thread JonZeolla
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/836 @cestella I know you were out recently, just wanted to bring this one to the top of your inbox. Would like to have this in the upcoming release, but also want to get your input. ---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-27 Thread zeo...@gmail.com
In an attempt to keep this from becoming unbearably long, I will try to keep my responses short, but I would be happy to elaborate. That's a fairly good timeline and summary, but here are some clarifications in corresponding order: - The plugin history is quite short and you can probably get a

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

2017-11-27 Thread JonZeolla
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/848 Tested via ``` cd /Users/jzeolla/metron-pr848 docker run -it -v /Users/jzeolla/metron-pr848:/root/metron ansible-docker:2.0.0.2 bash cd /root/metron mvn clean package

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-27 Thread Otto Fowler
I am not sure that our use of the plugin necessarily equates to it being implicitly coupled to Metron. It seems like the Right Thing To Do™, esp. for an Apache project would be to make this available for use by the greater bro community. Unless we expect to do extensive iterative work on the

[GitHub] metron issue #847: METRON-1313: Update metron-deployment to use bro-pkg to i...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/847 +1, nice work ---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/849 I think that is valid ---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-27 Thread Matt Foley
[Please pardon me that the below is a little labored. I’m trying to understand the implications for both release and use, which requires some explanation as well as the two questions needed. Q1 and Q2 below are probably the same question, asked in slightly different contexts. Please consider

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-27 Thread zeo...@gmail.com
The reason we decided to do that was because it is the best way for it to be used (and thus improved on and quality tested) by the broader bro community. If it's any indication of it's popularity, there was just an email on the bro mailing list about the plugin a few days ago, and I've already

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-27 Thread James Sirota
I agree with Nick. Since the plugin is tightly coupled with Metron why not just pull it into the main repo and version it with the rest of the code? Do we really need the second repo for the plug-in? Thanks, James 16.11.2017, 08:06, "Nick Allen" : >>  I would suggest

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-27 Thread Michael Miklavcic
It's a worthy mention. Our existing pruner wouldn't be able to handle Solr without modification, so we'd either need something native to Solr or something custom. Mike On Mon, Nov 27, 2017 at 3:46 PM, James Sirota wrote: > One thing to keep in mind, as we will be

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-27 Thread James Sirota
One thing to keep in mind, as we will be introducing Solr shortly, is to find if something similar to curator exists for Solr. But we'll cross that bridge when we get there 22.11.2017, 22:58, "Ali Nazemian" : > Sure. I will have a chat internally and come back to you

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

2017-11-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/803 ---

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 Great. I will get this PR merged. I am glad to see that this one is ready to go. ---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/849 +1, Tested on OSX and on centos 6 with and without compliant g++ installed Thanks Nick! ---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153336117 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153334325 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r15358 --- Diff: metron-deployment/packaging/docker/ansible-docker/README.md --- @@ -1,17 +1,34 @@ # Overview -The Metron ansible-docker container is

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153332780 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -14,13 +14,18 @@ # See the License for the specific language

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/803 I agree. I'm fine with going ahead with this, but I'd like to see end to end stability being addressed as the next UI priority, which I believe @iraghumitra is already doing some work on.

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153328847 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread mraliagha
Github user mraliagha commented on the issue: https://github.com/apache/metron/pull/840 Yes, I agree. It completely makes sense to minimize the scope and work on stabilizing this version at this moment. ---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-27 Thread JonZeolla
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153327669 --- Diff: metron-deployment/packaging/docker/ansible-docker/README.md --- @@ -1,17 +1,34 @@ # Overview -The Metron ansible-docker container is

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-27 Thread JonZeolla
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153326885 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -14,13 +14,18 @@ # See the License for the specific language governing

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-27 Thread JonZeolla
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153326917 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -33,18 +38,28 @@ RUN tar xvf setuptools-11.3.tar.gz WORKDIR

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153325047 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread nickwallen
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153324714 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153322856 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

2017-11-27 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153322627 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler

[GitHub] metron pull request #849: METRON-1320 Add C++11 Compliance Check to platform...

2017-11-27 Thread nickwallen
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/849 METRON-1320 Add C++11 Compliance Check to platform-info Some of the module dependencies for the Management and Alerts UI must be built natively on the host. This requires a C/C++ compiler. In

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

2017-11-27 Thread ottobackwards
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/848 METRON-1333 Ensure that ansible-docker can be used to build metron ( including rpms ) The ansible-docker container could be used to build metron ( even the rpms ) and run the ansible

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread merrimanr
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/803 I agree with @nickwallen. I think we're good to merge this as long as e2e tests are being addressed in a separate PR. +1 ---

[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread justinleet
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r153289006 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template --- @@

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 For reference, here is a list of some of the follow-on work we should consider: - Improvements to Kibana dashboard - Add new timestamp field to parsers and index templates to take place

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread mmiklavc
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @mraliagha I do think we should consider revisiting the field name conventions, but I'd push for that as a follow-on task. As discussed in other points on this thread, e.g. going straight to ES

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 +1 I'd like to see sign-off from at least one other committer (if not more) before this gets merged. I previously outlined the manual functional testing that I performed. All

Re: [DISCUSS] Upcoming Release

2017-11-27 Thread Otto Fowler
Considering the problems we are having with people building the node stuff on centos, would it make sense to wait to take a potential PR that allows full metron builds in our ansible docker image? On November 26, 2017 at 21:26:27, Matt Foley (ma...@apache.org) wrote: Hope everyone (at least

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler
Also, since I changed the profiles to not run the rpm docker if you are in docker already ( and put the rpm tools into the ansible docker ) a while ago we may be able to build world in the ansible image, and point folks having issues to that…. On November 27, 2017 at 10:57:03, Otto Fowler

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler
OK, So I have >mvn clean package working in docker. I want to try a couple of things and maybe I can throw a pr together. On November 27, 2017 at 10:03:31, Otto Fowler (ottobackwa...@gmail.com) wrote: First issue is that we need c++ 11 on centos 6.8 On November 27, 2017 at 09:53:55, Simon

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread zeo...@gmail.com
Note that I cleaned up the ansible scripts that install C++ 11 in my latest PR , but it's not super relevant to this conversation. Jon On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com wrote: > That was also required for bro

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread zeo...@gmail.com
That was also required for bro 2.5.2, so I did that here . Feel free to reuse the approach elsewhere Jon On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler wrote: > First issue is that we need

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler
First issue is that we need c++ 11 on centos 6.8 On November 27, 2017 at 09:53:55, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Well, that’s good news on that issue. Reproducing the problem is half way to solving it, right? I would still say there are some systemic things going

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Simon Elliston Ball
Well, that’s good news on that issue. Reproducing the problem is half way to solving it, right? I would still say there are some systemic things going on that have manifested in a variety of ways on both the users and dev list, so it’s worth us having a good look at a more robust approach to

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-11-27 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 @iraghumitra Can you describe how you install Node/NPM on your development box? I want to install using the same mechanism (and versions) and see if I can get the e2e tests all working like you.

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Otto Fowler
I can reproduce the failure in out ansible docker build container, which is also centos. The issue is building our node on centos in all these cases. On November 27, 2017 at 07:02:51, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Thinking about this, doesn’t our build plugin

Re: [DISCUSS] NPM / Node Problems

2017-11-27 Thread Simon Elliston Ball
Thinking about this, doesn’t our build plugin explicitly install it’s own node? So actually all the node version things may be a red herring, since this is under our control through the pom. Not sure if we actually exercising this control. It seems that some of the errors people report are more

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

2017-11-27 Thread mraliagha
Github user mraliagha commented on the issue: https://github.com/apache/metron/pull/840 Is this the best time to ask for changing field name convention to avoid dot or colon? We are externally using Hive external tables on HDFS data, due to Hive limitations we need to change the