Re: [DISCUSS] Are/how are you using the ES data pruner?

Sorry, Michael. I am having some issues to share any code right now. It seems we need to go through an internal verification of anything we want to share. BTW, the curator script I mentioned is very simple and nothing special. It doesn't worth to waste any time on waiting for it. Cheers, Ali On T

[GitHub] metron pull request #850: METRON-1335: Install metron-maas-service RPM as a ...

GitHub user anandsubbu opened a pull request: https://github.com/apache/metron/pull/850 METRON-1335: Install metron-maas-service RPM as a part of the full-dev deployment ## Contributor Comments Modified mpack metainfo.xml to install metron-maas-service RPM as well. **Te

[GitHub] metron-bro-plugin-kafka issue #4: METRON-1329: Simplify metron-bro-plugin-ka...

Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/4 +1 - review looks good - ran through test steps, everything works as described Great job! ---

[GitHub] metron issue #836: METRON-1308: Fix Metron Documentation

Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/836 @cestella I know you were out recently, just wanted to bring this one to the top of your inbox. Would like to have this in the upcoming release, but also want to get your input. ---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

In an attempt to keep this from becoming unbearably long, I will try to keep my responses short, but I would be happy to elaborate. That's a fairly good timeline and summary, but here are some clarifications in corresponding order: - The plugin history is quite short and you can probably get a go

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/848 Tested via ``` cd /Users/jzeolla/metron-pr848 docker run -it -v /Users/jzeolla/metron-pr848:/root/metron ansible-docker:2.0.0.2 bash cd /root/metron mvn clean package -DskipTests

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

I am not sure that our use of the plugin necessarily equates to it being implicitly coupled to Metron. It seems like the Right Thing To Do™, esp. for an Apache project would be to make this available for use by the greater bro community. Unless we expect to do extensive iterative work on the plug

[GitHub] metron issue #847: METRON-1313: Update metron-deployment to use bro-pkg to i...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/847 +1, nice work ---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/849 I think that is valid ---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/849 Ran on a purposefully misconfigured macOS system. Is this the expected output? ``` -- g++ xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Please pardon me that the below is a little labored. I’m trying to understand the implications for both release and use, which requires some explanation as well as the two questions needed. Q1 and Q2 below are probably the same question, asked in slightly different contexts. Please consider

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

The reason we decided to do that was because it is the best way for it to be used (and thus improved on and quality tested) by the broader bro community. If it's any indication of it's popularity, there was just an email on the bro mailing list about the plugin a few days ago, and I've already rec

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

I agree with Nick. Since the plugin is tightly coupled with Metron why not just pull it into the main repo and version it with the rest of the code? Do we really need the second repo for the plug-in? Thanks, James 16.11.2017, 08:06, "Nick Allen" : >>  I would suggest that we institute a re

Re: [DISCUSS] Are/how are you using the ES data pruner?

It's a worthy mention. Our existing pruner wouldn't be able to handle Solr without modification, so we'd either need something native to Solr or something custom. Mike On Mon, Nov 27, 2017 at 3:46 PM, James Sirota wrote: > One thing to keep in mind, as we will be introducing Solr shortly, is to

Re: [DISCUSS] Are/how are you using the ES data pruner?

One thing to keep in mind, as we will be introducing Solr shortly, is to find if something similar to curator exists for Solr. But we'll cross that bridge when we get there 22.11.2017, 22:58, "Ali Nazemian" : > Sure. I will have a chat internally and come back to you shortly. It was a > quick a

[GitHub] metron pull request #803: Metron-1252: Build ui for grouping alerts into met...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/803 ---

[GitHub] metron issue #848: METRON-1333 Ensure that ansible-docker can be used to bui...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/848 Added above, adding as a comment to make sure it is seen - Also, after building with the rpms, exit docker and do full_dev with `vagrant --ansible-skip-tags="bui

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 Great. I will get this PR merged. I am glad to see that this one is ready to go. ---

[GitHub] metron issue #849: METRON-1334 Add C++11 Compliance Check to 'platform-info....

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/849 +1, Tested on OSX and on centos 6 with and without compliant g++ installed Thanks Nick! ---

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153336117 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +ech

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153334325 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r15358 --- Diff: metron-deployment/packaging/docker/ansible-docker/README.md --- @@ -1,17 +1,34 @@ # Overview -The Metron ansible-docker container is

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153332780 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -14,13 +14,18 @@ # See the License for the specific language governi

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/803 I agree. I'm fine with going ahead with this, but I'd like to see end to end stability being addressed as the next UI priority, which I believe @iraghumitra is already doing some work on.

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153328847 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user mraliagha commented on the issue: https://github.com/apache/metron/pull/840 Yes, I agree. It completely makes sense to minimize the scope and work on stabilizing this version at this moment. ---

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153327669 --- Diff: metron-deployment/packaging/docker/ansible-docker/README.md --- @@ -1,17 +1,34 @@ # Overview -The Metron ansible-docker container is prov

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153326885 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -14,13 +14,18 @@ # See the License for the specific language governing p

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/848#discussion_r153326917 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -33,18 +38,28 @@ RUN tar xvf setuptools-11.3.tar.gz WORKDIR /usr/src/setu

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153325047 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +ech

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153324714 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +ech

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153322856 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +

[GitHub] metron pull request #849: METRON-1334 Add C++11 Compliance Check to 'platfor...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/849#discussion_r153322627 --- Diff: metron-deployment/scripts/platform-info.sh --- @@ -73,6 +73,30 @@ echo "--" echo "npm" npm --version +# C++ compiler +

[GitHub] metron pull request #849: METRON-1320 Add C++11 Compliance Check to platform...

GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/849 METRON-1320 Add C++11 Compliance Check to platform-info Some of the module dependencies for the Management and Alerts UI must be built natively on the host. This requires a C/C++ compiler. In a

[GitHub] metron pull request #848: METRON-1333 Ensure that ansible-docker can be used...

GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/848 METRON-1333 Ensure that ansible-docker can be used to build metron ( including rpms ) The ansible-docker container could be used to build metron ( even the rpms ) and run the ansible scripts.

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/803 I agree with @nickwallen. I think we're good to merge this as long as e2e tests are being addressed in a separate PR. +1 ---

[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r153289006 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/snort_index.template --- @@ -10

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 For reference, here is a list of some of the follow-on work we should consider: - Improvements to Kibana dashboard - Add new timestamp field to parsers and index templates to take place

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @mraliagha I do think we should consider revisiting the field name conventions, but I'd push for that as a follow-on task. As discussed in other points on this thread, e.g. going straight to ES 6.x,

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 +1 I'd like to see sign-off from at least one other committer (if not more) before this gets merged. I previously outlined the manual functional testing that I performed. All

Re: [DISCUSS] Upcoming Release

Considering the problems we are having with people building the node stuff on centos, would it make sense to wait to take a potential PR that allows full metron builds in our ansible docker image? On November 26, 2017 at 21:26:27, Matt Foley (ma...@apache.org) wrote: Hope everyone (at least in

Re: [DISCUSS] NPM / Node Problems

Also, since I changed the profiles to not run the rpm docker if you are in docker already ( and put the rpm tools into the ansible docker ) a while ago we may be able to build world in the ansible image, and point folks having issues to that…. On November 27, 2017 at 10:57:03, Otto Fowler (ottoba

Re: [DISCUSS] NPM / Node Problems

OK, So I have >mvn clean package working in docker. I want to try a couple of things and maybe I can throw a pr together. On November 27, 2017 at 10:03:31, Otto Fowler (ottobackwa...@gmail.com) wrote: First issue is that we need c++ 11 on centos 6.8 On November 27, 2017 at 09:53:55, Simon El

Re: [DISCUSS] NPM / Node Problems

Note that I cleaned up the ansible scripts that install C++ 11 in my latest PR , but it's not super relevant to this conversation. Jon On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com wrote: > That was also required for bro 2.5.2, so I did that

Re: [DISCUSS] NPM / Node Problems

That was also required for bro 2.5.2, so I did that here . Feel free to reuse the approach elsewhere Jon On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler wrote: > First issue is that we need c++ 11 on centos 6.8 > > >

Re: [DISCUSS] NPM / Node Problems

First issue is that we need c++ 11 on centos 6.8 On November 27, 2017 at 09:53:55, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Well, that’s good news on that issue. Reproducing the problem is half way to solving it, right? I would still say there are some systemic things going on

Re: [DISCUSS] NPM / Node Problems

Well, that’s good news on that issue. Reproducing the problem is half way to solving it, right? I would still say there are some systemic things going on that have manifested in a variety of ways on both the users and dev list, so it’s worth us having a good look at a more robust approach to n

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/803 @iraghumitra Can you describe how you install Node/NPM on your development box? I want to install using the same mechanism (and versions) and see if I can get the e2e tests all working like you.

Re: [DISCUSS] NPM / Node Problems

I can reproduce the failure in out ansible docker build container, which is also centos. The issue is building our node on centos in all these cases. On November 27, 2017 at 07:02:51, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Thinking about this, doesn’t our build plugin explici

Re: [DISCUSS] NPM / Node Problems

Thinking about this, doesn’t our build plugin explicitly install it’s own node? So actually all the node version things may be a red herring, since this is under our control through the pom. Not sure if we actually exercising this control. It seems that some of the errors people report are more

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user mraliagha commented on the issue: https://github.com/apache/metron/pull/840 One of the issues that we had with the previous mpack was the lack of config segregation for Elasticsearch Master Nodes and Data Nodes. Hence, we have ended up hardcoding specific configuration out

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user mraliagha commented on the issue: https://github.com/apache/metron/pull/840 Is this the best time to ask for changing field name convention to avoid dot or colon? We are externally using Hive external tables on HDFS data, due to Hive limitations we need to change the Metro