[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155698109 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo':

[GitHub] metron issue #859: METRON-1345: Update EC2 README for custom Ansible tags

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/859 I just found another item when going through an AWS install. We don't setup users for the REST API. From what I can tell, we do this via blueprint when we install full-dev from a setting in

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155664814 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo': }

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155664254 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo': }

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155659293 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/861 Right, that should cover it. ---

[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...

Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/812 Sorry, I'm not at a computer right now but if anybody wants to push this before our release candidate is cut feel free ---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

Good, I’ll build the RC tonight. Thanks Jon. --Matt From: "zeo...@gmail.com" Date: Thursday, December 7, 2017 at 12:27 PM To: Matt Foley Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro' Otto, your understanding is correct,

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155648096 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155649024 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo':

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/858 This smells like it might be a bigger effort than one PR, I think I'd rather have smaller PRs on a feature branch so each is easier to review. I would expec that we will want to try it out and

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 As long as breakpoints are limited to code we maintain in Metron, I think it's just a matter of documenting how to set it up. For example, I run REST locally in Intellij and debugging works fine

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155646423 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo': }

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155646268 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/SelectTransformation.java --- @@ -0,0 +1,52

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/861 That is an excellent catch @simonellistonball, a test to go with it too ;) ---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155645745 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155645303 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo':

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155645013 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/861 It suddenly occurs to me that we should probably whitelist the original_string and timestamp fields, so that these are always kept by this transformation. Does that make sense? ---

[GitHub] metron pull request #847: METRON-1313: Update metron-deployment to use bro-p...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/847 ---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155644387 --- Diff: metron-platform/metron-parsers/README.md --- @@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal to 'foo': }

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/861 Somehow I knew you were going to say something about docs will add. ---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155641855 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0

Re: New PMC members

Congratulations, guys! Well deserved by all 3. Jon On Thu, Dec 7, 2017 at 10:48 AM Kyle Richardson wrote: > Congratulations guys! Well deserved. > > -Kyle > > On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen wrote: > > > Congrats to all 3 for joining

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

You and Matt should coordinate sending a mail to the dev list with a heads up, starting and done. I think you mean that: Between X and YYY, if you do a fetch apache && checkout -b foo apache/master and then do vagrant up with the sensors enabled it will fail. Right? On December 7,

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155630936 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/SelectTransformation.java --- @@ -0,0 +1,52 @@

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155631654 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

That means we can’t wait and release-vote the plugin before pushing its tag. I recommend that you go ahead and push the 0.1 tag as soon as you push the commit. This does not imply approval by the committee, nor does it constitute a release. It’s just a tag. If that tag turns out not to

[GitHub] metron-bro-plugin-kafka pull request #4: METRON-1329: Simplify metron-bro-pl...

Github user asfgit closed the pull request at: https://github.com/apache/metron-bro-plugin-kafka/pull/4 ---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

FYI to be uber clear about the effects of what I'm doing, spinning up full-dev only when including the sensors will be broken on the bro plugin install step between when I push the changes, and when mattf pushes the 0.1 tag to apache/metron-bro-plugin-kafka. Jon On Thu, Dec 7, 2017 at 3:05 PM

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

Sounds good. Yes Matt, I will handle my parts now. Thanks everyone Jon On Thu, Dec 7, 2017 at 2:32 PM Matt Foley wrote: > I can start the release process tonight. > > > > Jon, you mentioned you want to commit > > > https://github.com/apache/metron/pull/847 and > >

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/861 Thanks Simon. Before a code review, a couple of questions: I see a check next to steps to verify manually, but I don't see the steps. This is also missing documentation. ---

[GitHub] metron pull request #861: Implemented SELECT transformer to project fields f...

GitHub user simonellistonball opened a pull request: https://github.com/apache/metron/pull/861 Implemented SELECT transformer to project fields from parser ## Contributor Comments This is a simple PR to add FieldTransformation capabilities to the parsers allowing basic

[GitHub] metron pull request #860: METRON-1346 Add new PMC members to site

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/860 ---

[GitHub] metron issue #860: METRON-1346 Add new PMC members to site

Github user cestella commented on the issue: https://github.com/apache/metron/pull/860 +1 by inspection, lgtm ---

[GitHub] metron pull request #860: METRON-1346 Add new PMC members to site

GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/860 METRON-1346 Add new PMC members to site Add new PMC members @mmiklavc, @justinleet , @ottobackwards ##Testing Follow steps at

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/858 If we think that remote debugging would cause this to balloon, should we consider making this a feature branch so multiple PRs with iterative solutions can exist and be reviewed separately? Just a

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/858 We would want to remote debug any code that we maintain in Metron, so any services which run metron code would need to have remote debugging (e.g. storm, the REST app). Ironically, this is one of

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 Point taken on maintaining 2 different solutions. Which services would people want to enable remote debugging on? I've only ever debugged Metron code so I don't know which services are important

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

I agree with Justin. I suggest we move forward with option (a). I see it as simpler and more flexible moving forward. -Kyle On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet wrote: > Do we have any further discussion on this? Pardon me if I misstate > anyone's position, but

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

Do we have any further discussion on this? Pardon me if I misstate anyone's position, but it seems like we have a couple people (Otto and Jon and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a section of people like myself without a particular horse in the race. It seems

Re: New PMC members

Congratulations guys! Well deserved. -Kyle On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen wrote: > Congrats to all 3 for joining the PMC! > > On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella wrote: > > > Well, obviously, I meant Metron instead of Impala. To

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 @merrimanr would you imagine someone just leaving this running all the time? or starting on demand per use? ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 OK, After getting the new changes, everything works as advertised. I was able to run on mac os , and the end to end passed. I ran a few times in a row and there where no issues, so it

Re: New PMC members

Congrats to all 3 for joining the PMC! On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella wrote: > Well, obviously, I meant Metron instead of Impala. To this point, we > should have a wiki page around templates for this, similar to the impala > project. :) > > On Thu, Dec 7,

Re: New PMC members

Well, obviously, I meant Metron instead of Impala. To this point, we should have a wiki page around templates for this, similar to the impala project. :) On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella wrote: > The Project Management Committee (PMC) for Apache Impala has

Re: New PMC members

Boy are those Impala guys in for a surprise On December 7, 2017 at 10:06:19, Casey Stella (ceste...@gmail.com) wrote: The Project Management Committee (PMC) for Apache Impala has invited Otto Fowler, Michael Miklavcic and Justin Leet to become a PMC member and we are pleased to announce that

New PMC members

The Project Management Committee (PMC) for Apache Impala has invited Otto Fowler, Michael Miklavcic and Justin Leet to become a PMC member and we are pleased to announce that they have accepted. Congratulations and welcome!

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/858 I agree with @cestella (and this might spill over into a discuss thread as @ottobackwards mentioned). Maintaining both anything over other than short, short term is going to be a nightmare.

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 @ottobackwards I don't see anything obviously wrong. Maybe you could try pulling the latest commits or copying the one here:

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 Docker Compose provides a CLI just like vagrant does: https://docs.docker.com/compose/reference/overview/. I would be happy to wrap those commands in easy to understand scripts. ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/858 Regarding remote debugging, it seems that having both in memory components AND docker is the worst of all worlds here. I would prefer to not live in a world where we can't simplify this to one

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 We will, if this pr goes through need to have a 'writing new integration tests' readme about namespacing etc.. ---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 No matter what approach we take, my point is that we need to have some thing that says you need to run docker to debug the tests. Beyond that, we don't have to over back this. I think the

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 I would love to have a Debugging readme that is even more comprehensive than just integration tests. Being able to easily debug is important to me and I do it all the time. I had to do

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 ```bash metron-alerts configure table ✗ should select columns from table configuration - Error: connect ECONNREFUSED 127.0.0.1:9210 - Error: socket hang up

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 Ok, I'm going to try running through some things. One point, and I will admit this just may be me: I see the same problem with this as I had with the original docker stuff. The

Re: Heterogeneous indexing batch size for different Metron feeds

I’m trying to think of what information would be needed to look at this the next time it happens. If someone wanted to reproduce this. Examples of a set of configurations that do not work maybe. The metrics on the parser topologies involved ( how many parsers, message rate ). The platform_info.sh

Re: Wiki Docs links seem wrong

Awesome, many thanks! > On 7 Dec 2017, at 13:08, Kyle Richardson wrote: > > Fixed. > > -Kyle > > On Thu, Dec 7, 2017 at 7:20 AM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> https://cwiki.apache.org/confluence/display/METRON/ >>

Re: Wiki Docs links seem wrong

Fixed. -Kyle On Thu, Dec 7, 2017 at 7:20 AM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > https://cwiki.apache.org/confluence/display/METRON/ > Metron+User+Guide+-+per+release confluence/display/METRON/Metron+User+Guide+-+per+release> > > The links

Re: Heterogeneous indexing batch size for different Metron feeds

We use TreeCache . When the configuration is updated in zookeeper, the configuration object in the bolt is updated. This configuration is read on each message, so I think from what I see new

Wiki Docs links seem wrong

https://cwiki.apache.org/confluence/display/METRON/Metron+User+Guide+-+per+release The links don’t seem to correspond to the versions on this page. Would be happy to fix, but I don’t have wiki perms. Simon

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/858 I was more thinking about having to have a "Debugging Integration Tests" readme. I think having the in-memory stuff available is worth getting some feedback. I have in the past

Re: Heterogeneous indexing batch size for different Metron feeds

Thank you very much. Unfortunately, reproducing all the situations are very costly for us at this moment. We are kind of avoiding to hit that issue by using the same batch size for all the feeds. Hopefully, with the new PR Casey provided for the segregation of ES and HDFS, it will be very much