[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155698109
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

agreed, and added.


---

[GitHub] metron issue #859: METRON-1345: Update EC2 README for custom Ansible tags

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/859
  
I just found another item when going through an AWS install. We don't setup 
users for the REST API. From what I can tell, we do this via blueprint when we 
install full-dev from a setting in single_node_vm.yml

```
  - metron-rest-env:
  metron_spring_profiles_active: "dev"
```

I'm wondering if we can/should configure this same setting in 
small_cluster.yml, which is used by the Amazon EC2 deployment. Any thoughts on 
this? At the very least I think this PR should include a doc change to indicate 
that we aren't setting up users unless you explicitly set a profile via Ambari 
or create them manually as a separate task.


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155664814
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

We should document that the system fields are protected as well


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155664254
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

It should be documented then in the readme


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155659293
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java
 ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.stellar.dsl.Context;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
--- End diff --

It's a pretty broad reaching concern when you account for the impact of 
remove, stellar, and the other field transforms. I'm happy to look at it, but I 
expect it will just need to be reviewed again as part of a follow on.


---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/861
  
Right, that should cover it.


---

[GitHub] metron issue #812: METRON-1273: Website documentation link should point to t...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/812
  
Sorry, I'm not at a computer right now but if anybody wants to push this 
before our release candidate is cut feel free 


---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley]

Good, I’ll build the RC tonight.  Thanks Jon.
--Matt

From: "zeo...@gmail.com" 
Date: Thursday, December 7, 2017 at 12:27 PM
To: Matt Foley 
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

Otto, your understanding is correct, but given Mattf's comments and my recent 
actions, my initial comment is no longer true - it should continue to work 
throughout.

Mattf, agreed.  
Done.

Jon

On Thu, Dec 7, 2017 at 3:14 PM Matt Foley 
> wrote:
That means we can’t wait and release-vote the plugin before pushing its tag.
I recommend that you go ahead and push the 0.1 tag as soon as you push the 
commit.
This does not imply approval by the committee, nor does it constitute a release.
It’s just a tag.  If that tag turns out not to coincide with an approved 
release, that’s okay.
(But it probably will.)

Thanks,
--Matt

On 12/7/17, 12:09 PM, "zeo...@gmail.com" 
> wrote:

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com 
> wrote:

> Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley 
> wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release.  Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen >
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" 
>
>> Cc: Matt Foley >
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for 
Bro'
>>
>>
>>
>> I am more interested in getting a release cut.  If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
>
>> wrote:
>>
>> Do we have any further discussion on this?  Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and 
Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can 
get
>> the release bus moving again, and right now it seems like (a) is 
gathering
>> more explicit support.  Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest 
we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com 
>
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin 
first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation.  For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler 
> wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this.  It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley 
(ma...@apache.org) wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the 

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155648096
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java
 ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.stellar.dsl.Context;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
--- End diff --

That test would belong in this ticket



---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155649024
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

Yes, that would break. I'd say that was flat out user error, and not 
something to guard against, or indeed something it's possible to guard against. 
I guess the key thing here is that it requires maintaining order in field 
transformations.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/858
  
This smells like it might be a bigger effort than one PR, I think I'd 
rather have smaller PRs on a feature branch so each is easier to review.  I 
would expec that we will want to try it out and live with it a bit as it will 
impact day-to-day productivity fairly fundamentally.  I'd vote for a feature 
branch, but I'd like to hear other people's impressions.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
As long as breakpoints are limited to code we maintain in Metron, I think 
it's just a matter of documenting how to set it up.  For example, I run REST 
locally in Intellij and debugging works fine against the in memory components 
or full dev.  I think the Storm topologies should continue to be run locally 
with LocalCluster so we're covered there too as along as the topologies can 
access the services running in Docker.  The only difference I can think of 
would be that now you can't put a breakpoint in Kafka or Elasticsearch server 
side code without enabling remote debugging.

I have no problem with this being a feature branch.  There is a lot of work 
to be done and design issues to work through.  The tradeoff of having this in a 
feature branch is that it will delay the ability to run the UI e2e tests in our 
build.  The tradeoff of doing it incrementally is that our build times will 
increase until we switch the tests over because we will have to use both in 
memory components and Docker.  I'm happy to do it either way.  


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155646423
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

If a user has a stellar transform (existing ) that takes field X as input, 
but doesn't want X as output, and puts the SELECT in the wrong order, could 
they break their existing transform because X now missing?


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155646268
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/SelectTransformation.java
 ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import org.apache.metron.stellar.dsl.Context;
+
--- End diff --

I've added to relevant readme (in parsers docs) which would seem a more 
likely place than javadoc (if we want to add javadoc, which we don't publish 
anyway, we should also look at adding this to everything in this area, which 
feels out of scope). Not even the stellar version gets javadoc!


---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/861
  
That is an excellent catch @simonellistonball, a test to go with it too ;)


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155645745
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java
 ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.stellar.dsl.Context;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
--- End diff --

yeah, makes sense, I suspect on a different ticket.


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155645303
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

And order doesn't really matter, you might have this first, or last, or 
sandwiched between two stellar blocks. That said, I would expect the most 
common use case to be have this one at the end of a set.


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155645013
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java
 ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.stellar.dsl.Context;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
--- End diff --

or a FieldTransformer ( level up ) test more likely.



---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/861
  
It suddenly occurs to me that we should probably whitelist the 
original_string and timestamp fields, so that these are always kept by this 
transformation. Does that make sense?


---

[GitHub] metron pull request #847: METRON-1313: Update metron-deployment to use bro-p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/847


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155644387
  
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal 
to 'foo':
 }
 ```
 
+* `SELECT`: This transformation filters the fields in the message to 
include only the configured output fields, and drops any not explicitly 
included. 
+
+For example: 
+```
+{
+...
+"fieldTransformations" : [
+  {
+"output" : ["field1", "field2" ] 
+  , "transformation" : "SELECT"
+  }
+  ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will 
only output the first two.
+
--- End diff --

How does this relate to REMOVE?  If I have more than one transformation 
should this be last or does it not matter?


---

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/861
  
Somehow I knew you were going to say something about docs will add.


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[simonellistonball]

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155641855
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java
 ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.stellar.dsl.Context;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
--- End diff --

that feels a lot more like an integration test, or a broader test than 
belongs in this one piece. Otherwise every single test of transformations is 
going to end up with combinatorial complexity.


---

Re: New PMC members

[zeo...@gmail.com]

Congratulations, guys!  Well deserved by all 3.

Jon

On Thu, Dec 7, 2017 at 10:48 AM Kyle Richardson 
wrote:

> Congratulations guys! Well deserved.
>
> -Kyle
>
> On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen  wrote:
>
> > Congrats to all 3 for joining the PMC!
> >
> > On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella 
> wrote:
> >
> > > Well, obviously, I meant Metron instead of Impala.  To this point, we
> > > should have a wiki page around templates for this, similar to the
> impala
> > > project. :)
> > >
> > > On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella 
> > wrote:
> > >
> > > > The Project Management Committee (PMC) for Apache Impala has invited
> > Otto
> > > > Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and
> > we
> > > > are pleased to announce that they have accepted.
> > > >
> > > > Congratulations and welcome!
> > > >
> > > >
> > >
> >
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Otto Fowler]

You and Matt should coordinate sending a mail to the dev list with a heads
up, starting and done.

I think you mean that:

Between X  and YYY, if you do a fetch apache && checkout -b foo
apache/master and then
do vagrant up with the sensors enabled it will fail.  Right?



On December 7, 2017 at 15:09:52, zeo...@gmail.com (zeo...@gmail.com) wrote:

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com  wrote:

> Sounds good. Yes Matt, I will handle my parts now. Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley  wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release. Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen 
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" 
>> Cc: Matt Foley 
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
Bro'
>>
>>
>>
>> I am more interested in getting a release cut. If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
>> wrote:
>>
>> Do we have any further discussion on this? Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and
Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and
presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can
get
>> the release bus moving again, and right now it seems like (a) is
gathering
>> more explicit support. Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest
we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com 
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin
first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation. For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler 
wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this. It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org)
wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron. This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > > - open a jira,
>> > >
>> > > - add the submodule to the Metron code tree,
>> > >
>> > > - possibly (optionally) add build mechanism to the maven poms, and
>> > >
>> > > - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen 
>> > > Reply-To: "dev@metron.apache.org" 
>> > > Date: Tuesday, November 28, 2017 at 9:09 AM
>> > > To: "dev@metron.apache.org" 
>> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin
for
>> > Bro'
>> > >
>> > >
>> > >
>> > > I'll add a bit to Jon's technical comments.
>> > >
>> > >
>> > >
>> > > * We only created a separate repo because it was a technical
>> requirement
>> > to
>> > > leverage the bro-pkg mechanism.
>> > >
>> > > * Leveraging the new bro-pkg mechanism has many advantages as
>> outlined by
>> > > Jon.
>> > >
>> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
>> install
>> > the
>> > > plugin exactly how we use to.
>> > >
>> > >
>> > >
>> > > 

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155630936
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/SelectTransformation.java
 ---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import org.apache.metron.stellar.dsl.Context;
+
--- End diff --

Missing Javadoc.  The purpose and an example output of this transformation 
would be nice


---

[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/861#discussion_r155631654
  
--- Diff: 
metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java
 ---
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field.transformation;
+
+import java.util.HashMap;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.stellar.dsl.Context;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import com.google.common.collect.Iterables;
+
--- End diff --

Do we at a test where we have multiple transformers?
STELLAR , SELECT etc?


---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Matt Foley]

That means we can’t wait and release-vote the plugin before pushing its tag.
I recommend that you go ahead and push the 0.1 tag as soon as you push the 
commit.
This does not imply approval by the committee, nor does it constitute a 
release.  
It’s just a tag.  If that tag turns out not to coincide with an approved 
release, that’s okay. 
(But it probably will.)

Thanks,
--Matt 

On 12/7/17, 12:09 PM, "zeo...@gmail.com"  wrote:

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com  wrote:

> Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley  wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release.  Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen 
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" 
>> Cc: Matt Foley 
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for 
Bro'
>>
>>
>>
>> I am more interested in getting a release cut.  If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
>> wrote:
>>
>> Do we have any further discussion on this?  Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and 
Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can 
get
>> the release bus moving again, and right now it seems like (a) is 
gathering
>> more explicit support.  Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest 
we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com 
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin 
first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation.  For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this.  It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron.  This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > > - open a jira,
>> > >
>> > > - add the submodule to the Metron code tree,
>> > >
>> > > - possibly (optionally) add build mechanism to the maven poms, 
and
>> > >
>> > > - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen 
>> > > Reply-To: "dev@metron.apache.org" 
>> > > Date: Tuesday, November 28, 2017 at 

[GitHub] metron-bro-plugin-kafka pull request #4: METRON-1329: Simplify metron-bro-pl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron-bro-plugin-kafka/pull/4


---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com  wrote:

> Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley  wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release.  Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen 
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" 
>> Cc: Matt Foley 
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>>
>>
>>
>> I am more interested in getting a release cut.  If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
>> wrote:
>>
>> Do we have any further discussion on this?  Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can get
>> the release bus moving again, and right now it seems like (a) is gathering
>> more explicit support.  Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com 
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation.  For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this.  It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron.  This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > > - open a jira,
>> > >
>> > > - add the submodule to the Metron code tree,
>> > >
>> > > - possibly (optionally) add build mechanism to the maven poms, and
>> > >
>> > > - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen 
>> > > Reply-To: "dev@metron.apache.org" 
>> > > Date: Tuesday, November 28, 2017 at 9:09 AM
>> > > To: "dev@metron.apache.org" 
>> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
>> > Bro'
>> > >
>> > >
>> > >
>> > > I'll add a bit to Jon's technical comments.
>> > >
>> > >
>> > >
>> > > * We only created a separate repo because it was a technical
>> requirement
>> > to
>> > > leverage the bro-pkg mechanism.
>> > >
>> > > * Leveraging the new bro-pkg mechanism has many advantages as
>> outlined by
>> > > Jon.
>> > >
>> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
>> install
>> > the
>> > > plugin exactly how we use to.
>> > >
>> > >
>> > >
>> > > While I agree with Jon's technical comments, I disagree with the
>> > > non-technical ones.
>> > >
>> > >
>> > >
>> > > (1) I do not want to change our release management process. While we
>> > needed
>> > > to make a new repo (a technical change), I did not want that to change
>> > how
>> > > we operate as a community (our 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone

Jon

On Thu, Dec 7, 2017 at 2:32 PM Matt Foley  wrote:

> I can start the release process tonight.
>
>
>
> Jon, you mentioned you want to commit
>
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> before the release.  Is it convenient for you to do so today?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen 
> Date: Thursday, December 7, 2017 at 10:13 AM
> To: "dev@metron.apache.org" 
> Cc: Matt Foley 
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I am more interested in getting a release cut.  If me moving to the (a)
> camp gets us to consensus and cuts a release faster, then I'll do it.
> Let's get this release train moving.
>
>
>
> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
> wrote:
>
> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
>
> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com  wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > > - open a jira,
> > >
> > > - add the submodule to the Metron code tree,
> > >
> > > - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > > - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen 
> > > Reply-To: "dev@metron.apache.org" 
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" 
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technical ones.
> > >
> > >
> > >
> > > (1) I do not want to change our release management process. While we
> > needed
> > > to make a new repo (a technical change), I did not want that to change
> > how
> > > we operate as a community (our procedures, policies, versioning and
> > release
> > > cycles).
> > >
> > >
> > >
> > > (2) I see no value in adopting a separate release management process
> for
> > > the Bro plugin alone. Having a separate release process does not make
> the
> > > plugin *more* available to the Bro community.
> > >
> > >
> > >
> > > (3) I also see no value in positioning the plugin to be spun-out of the
> > > Metron project. It is a part of Metron and I want to see it benefit and
> > > evolve "the Apache-way".
> > >
> > >
> > >

[GitHub] metron issue #861: METRON-1341 Implemented SELECT transformer to project fie...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/861
  
Thanks Simon. Before a code review, a couple of questions:

I see a check next to steps to verify manually, but I don't see the steps.

This is also missing documentation.


---

[GitHub] metron pull request #861: Implemented SELECT transformer to project fields f...

[simonellistonball]

GitHub user simonellistonball opened a pull request:

https://github.com/apache/metron/pull/861

Implemented SELECT transformer to project fields from parser

## Contributor Comments
This is a simple PR to add FieldTransformation capabilities to the parsers 
allowing basic projection. There are definitely better ways to implement this 
as reflected in the comments, but those would need a wide ranging refactor of 
FieldTransformations, so I've decided to shelve that for now. 

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/metron METRON-1341

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/861.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #861


commit 7674c7283b5cefd2a40bb1329fc227e0b3a2227d
Author: Simon Elliston Ball 
Date:   2017-12-05T01:07:46Z

Implemented SELECT transformer to project fields from parser




---

[GitHub] metron pull request #860: METRON-1346 Add new PMC members to site

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/860


---

[GitHub] metron issue #860: METRON-1346 Add new PMC members to site

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/860
  
+1 by inspection, lgtm


---

[GitHub] metron pull request #860: METRON-1346 Add new PMC members to site

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/860

METRON-1346 Add new PMC members to site

Add new PMC members @mmiklavc, @justinleet , @ottobackwards 

##Testing
Follow steps at 
https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge


### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron new_pmcs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/860.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #860






---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/858
  
If we think that remote debugging would cause this to balloon, should we 
consider making this a feature branch so multiple PRs with iterative solutions 
can exist and be reviewed separately?  Just a thought.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/858
  
We would want to remote debug any code that we maintain in Metron, so any 
services which run metron code would need to have remote debugging (e.g. storm, 
the REST app).  Ironically, this is one of those situations where just running 
the in-memory components in a separate process would be easier, since it's just 
a JVM param to pass in to enable remote debugging.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
Point taken on maintaining 2 different solutions.  Which services would 
people want to enable remote debugging on?  I've only ever debugged Metron code 
so I don't know which services are important to people or why you would need to 
put breakpoints in core service code.  Enabling remote debugging will likely be 
a little bit different for each service since each project has their own 
mechanism for adding extra jvm params.  I'm happy to do a POC for this once I 
understand which services are important.

@ottobackwards I usually just leave mine running but spinning it up/down is 
really fast once all the images are loaded.



---

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Kyle Richardson]

I agree with Justin. I suggest we move forward with option (a). I see it as
simpler and more flexible moving forward.

-Kyle

On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet  wrote:

> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com  wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > > - open a jira,
> > >
> > > - add the submodule to the Metron code tree,
> > >
> > > - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > > - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen 
> > > Reply-To: "dev@metron.apache.org" 
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" 
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technical ones.
> > >
> > >
> > >
> > > (1) I do not want to change our release management process. While we
> > needed
> > > to make a new repo (a technical change), I did not want that to change
> > how
> > > we operate as a community (our procedures, policies, versioning and
> > release
> > > cycles).
> > >
> > >
> > >
> > > (2) I see no value in adopting a separate release management process
> for
> > > the Bro plugin alone. Having a separate release process does not make
> the
> > > plugin *more* available to the Bro community.
> > >
> > >
> > >
> > > (3) I also see no value in positioning the plugin to be spun-out of the
> > > Metron project. It is a part of Metron and I want to see it benefit and
> > > evolve "the Apache-way".
> > >
> > >
> > >
> > > In my mind, the best way to accommodate the additional repo, while
> > > minimizing changes to our release management process, is to treat the
> new
> > > repo as a submodule. I fail to see significant downsides to this
> > approach.
> > > A few extract command-line options do not seem overly onerous to me.
> > >
> > >
> > >
> > > Many thanks go to Jon for all the hard work he has put in enhancing the
> > > plugin.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > > In an attempt to keep this from becoming unbearably long, I will try to
> > > keep my responses short, but I would be happy to elaborate. That's a
> > fairly
> > > good timeline and summary, but here are some 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Justin Leet]

Do we have any further discussion on this?  Pardon me if I misstate
anyone's position, but it seems like we have a couple people (Otto and Jon
and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
section of people like myself without a particular horse in the race.

It seems like we need to come to some sort of consensus so that we can get
the release bus moving again, and right now it seems like (a) is gathering
more explicit support.  Do we have a compelling reason to not do (a)? To be
honest, my main worry is more "If we do (a) are we going to be miserable if
we need to iterate or adjust?" I'm not seeing anything that suggests
anything too terrible, so unless we see some more discussion, I suggest we
move forward with (a).

On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com  wrote:

> I would prefer a, but I was initially thinking of doing the plugin first
> and then get in the two PRs out to use this new tag, which are already +1'd
> and just waiting on this conversation.  For reference,
> https://github.com/apache/metron/pull/847 and
> https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> Jon
>
> On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
>
> > It seems to me, as I believe I have stated before that a) feels like the
> > proper way to handle this.  It is how I have seen other projects like
> NiFi
> > handle things as well.
> >
> >
> >
> > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> >
> > Okay, looking at this from the perspective of making a release:
> >
> >
> >
> > We have two choices:
> >
> > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > metron-bro-plugin-kafka, at the same time and using the same process
> > (modulo the necessary) as Metron.  This is dirt simple.
> >
> > b) I or someone needs to:
> >
> > - open a jira,
> >
> > - add the submodule to the Metron code tree,
> >
> > - possibly (optionally) add build mechanism to the maven poms, and
> >
> > - document as much as we think appropriate regarding what it is, how
> to
> > build it, and how to update it,
> >
> > and commit that before the 0.4.2 release.
> >
> >
> >
> > What is the will of the community?
> >
> > Thanks,
> >
> > --Matt
> >
> >
> >
> > From: Nick Allen 
> > Reply-To: "dev@metron.apache.org" 
> > Date: Tuesday, November 28, 2017 at 9:09 AM
> > To: "dev@metron.apache.org" 
> > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> Bro'
> >
> >
> >
> > I'll add a bit to Jon's technical comments.
> >
> >
> >
> > * We only created a separate repo because it was a technical requirement
> to
> > leverage the bro-pkg mechanism.
> >
> > * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> > Jon.
> >
> > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> the
> > plugin exactly how we use to.
> >
> >
> >
> > While I agree with Jon's technical comments, I disagree with the
> > non-technical ones.
> >
> >
> >
> > (1) I do not want to change our release management process. While we
> needed
> > to make a new repo (a technical change), I did not want that to change
> how
> > we operate as a community (our procedures, policies, versioning and
> release
> > cycles).
> >
> >
> >
> > (2) I see no value in adopting a separate release management process for
> > the Bro plugin alone. Having a separate release process does not make the
> > plugin *more* available to the Bro community.
> >
> >
> >
> > (3) I also see no value in positioning the plugin to be spun-out of the
> > Metron project. It is a part of Metron and I want to see it benefit and
> > evolve "the Apache-way".
> >
> >
> >
> > In my mind, the best way to accommodate the additional repo, while
> > minimizing changes to our release management process, is to treat the new
> > repo as a submodule. I fail to see significant downsides to this
> approach.
> > A few extract command-line options do not seem overly onerous to me.
> >
> >
> >
> > Many thanks go to Jon for all the hard work he has put in enhancing the
> > plugin.
> >
> >
> >
> >
> >
> > On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com 
> > wrote:
> >
> > In an attempt to keep this from becoming unbearably long, I will try to
> > keep my responses short, but I would be happy to elaborate. That's a
> fairly
> > good timeline and summary, but here are some clarifications in
> > corresponding order:
> >
> >
> >
> > - The plugin history is quite short and you can probably get a good bit
> of
> > context just by looking at the commits.
> >
> > - The plugin is only useful to the bro community, but it is rather
> popular.
> >
> > - The Bro team created the idea of bro packages, which can include bro
> > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > 'plugins' repo, they moved to have a 'packages' repo which is by default
> > referenced by 

Re: New PMC members

[Kyle Richardson]

Congratulations guys! Well deserved.

-Kyle

On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen  wrote:

> Congrats to all 3 for joining the PMC!
>
> On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella  wrote:
>
> > Well, obviously, I meant Metron instead of Impala.  To this point, we
> > should have a wiki page around templates for this, similar to the impala
> > project. :)
> >
> > On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella 
> wrote:
> >
> > > The Project Management Committee (PMC) for Apache Impala has invited
> Otto
> > > Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and
> we
> > > are pleased to announce that they have accepted.
> > >
> > > Congratulations and welcome!
> > >
> > >
> >
>

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
@merrimanr would you imagine someone just leaving this running all the 
time?  or starting on demand per use?


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
OK,
After getting the new changes, everything works as advertised.  I was able 
to run on mac os , and the end to end passed.  I ran a few times in a row and 
there where no issues, so it looks like setup/teardown works fine.

I think we you have done to this point is great @merrimanr, really great 
work.  I like the speed of getting compose up and everything. 

The idea of doing the namespaces for ES is great too.

I did not run the integration test

To summarize my feedback:

- I would like some scripts that wrap this up into high level usage
- I think we need doc around writing tests that use this framework ( will 
we be adding tests one by one + any new tests )?
- I think we should enable debugging on the services in docker on known 
ports.  Knowing that to @justinleet 's point, any java dev. can get their tool 
of choice to work.  We can do intellij specific examples on top of that as well
- I think we should have a print out statement, similar to what we have at 
the end of vagrant up/ansible that says the ip addresses and ports exposed + 
service name. this would include the debugging.  I know this helps with 
vagrant, it would help here.


---

Re: New PMC members

[Nick Allen]

Congrats to all 3 for joining the PMC!

On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella  wrote:

> Well, obviously, I meant Metron instead of Impala.  To this point, we
> should have a wiki page around templates for this, similar to the impala
> project. :)
>
> On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella  wrote:
>
> > The Project Management Committee (PMC) for Apache Impala has invited Otto
> > Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and we
> > are pleased to announce that they have accepted.
> >
> > Congratulations and welcome!
> >
> >
>

Re: New PMC members

[Casey Stella]

Well, obviously, I meant Metron instead of Impala.  To this point, we
should have a wiki page around templates for this, similar to the impala
project. :)

On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella  wrote:

> The Project Management Committee (PMC) for Apache Impala has invited Otto
> Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and we
> are pleased to announce that they have accepted.
>
> Congratulations and welcome!
>
>

Re: New PMC members

[Otto Fowler]

Boy are those Impala guys in for a surprise


On December 7, 2017 at 10:06:19, Casey Stella (ceste...@gmail.com) wrote:

The Project Management Committee (PMC) for Apache Impala has invited Otto
Fowler, Michael Miklavcic and Justin Leet to become a PMC member and we
are pleased to announce that they have accepted.

Congratulations and welcome!

New PMC members

[Casey Stella]

The Project Management Committee (PMC) for Apache Impala has invited Otto
Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and we
are pleased to announce that they have accepted.

Congratulations and welcome!

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/858
  
I agree with @cestella (and this might spill over into a discuss thread as 
@ottobackwards mentioned).  Maintaining both anything over other than short, 
short term is going to be a nightmare.  Invariably one or the other is going to 
break independently of the other.

Assuming we do go with remote debugging of Docker (and that's a total 
assumption, to be clear), we should make sure that the instructions aren't 
IntelliJ dependent.  Or at least, there are best effort instructions for people 
using something else.  I.e. if we end up using something like JetBrains' Docker 
Integration plugin, I personally think we should have at least some debugging 
instructions available outside that.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
@ottobackwards I don't see anything obviously wrong.  Maybe you could try 
pulling the latest commits or copying the one here:  
https://github.com/merrimanr/incubator-metron/blob/e2e-in-travis/metron-interface/metron-alerts/protractor.conf.js.
  Looks like your docker machine host is the same.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
Docker Compose provides a CLI just like vagrant does:  
https://docs.docker.com/compose/reference/overview/.  I would be happy to wrap 
those commands in easy to understand scripts.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/858
  
Regarding remote debugging, it seems that having both in memory components 
AND docker is the worst of all worlds here.  I would prefer to not live in a 
world where we can't simplify this to one solution rather than having to 
continue to maintain both.  I would vote for either enabling remote debugging 
in docker *xor* just migrating the in memory components out and documenting 
remote debugging for those processes.  Either way, documentation should be part 
of this PR.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
We will, if this pr goes through need to have a 'writing new integration 
tests' readme about namespacing etc..


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
No matter what approach we take, my point is that we need to have some 
thing that says you need to run docker to debug the tests.  Beyond that, we 
don't have to over back this.  I think the approach in this pr is preferable to 
the in-memory components in separate processes.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/858
  
I would love to have a Debugging readme that is even more comprehensive 
than just integration tests.  Being able to easily debug is important to me and 
I do it all the time.

I had to do some debugging to convert this test and it was the same process 
I have used before.  Just put a break point in your test and run the test in 
Debug in Intellij.  Debugging Storm is a different animal and I suspect we 
would continue to use LocalCluster.  We would need to ensure the topology 
running locally in an IDE would still be able to interact with services in 
Docker.  I have successfully done this in the past for each of the topologies.

The only other case I can think of would be putting breakpoints in code of 
services we use (Kafka code for example).  Is that something we want to be able 
to do?  I can see some value although I wouldn't think that's a normal thing.

We could always keep the in memory module and offer a way to start them up 
in a separate process.  That way you could still debug everything in Intellij.  
I think I would prefer that option over enabling remote debugging in Docker 
containers.

And again, we don't have to use Docker.  We could just move the in memory 
stuff outside of the integration tests and add a layer to manage them in 
separate processes.  But that means we now have to implement some features 
Docker provides us with custom code.


---

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
```bash
 metron-alerts configure table
✗ should select columns from table configuration
  - Error: connect ECONNREFUSED 127.0.0.1:9210
  - Error: socket hang up
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

**
*Failures*
**

1) metron-alerts configure table should select columns from table 
configuration
  - Error: connect ECONNREFUSED 127.0.0.1:9210
  - Error: socket hang up
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
  - Error: Timeout - Async callback was not invoked within timeout 
specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.

Executed 1 of 1 spec (1 FAILED) in 1 min 40 secs.
[09:14:23] I/launcher - 0 instance(s) of WebDriver still running
[09:14:23] I/launcher - chrome #01 failed 1 test(s)
[09:14:23] I/launcher - overall: 1 failed spec(s)
[09:14:23] E/launcher - Process exited with error code 1
```

I don't think that my protractor stuff is correct.Here is what I have:

```json
/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

// Protractor configuration file, see link for more information
// https://github.com/angular/protractor/blob/master/docs/referenceConf.js

/*global jasmine */
var SpecReporter = require('jasmine-spec-reporter').SpecReporter;

exports.config = {
  allScriptsTimeout: 25000,
  specs: [
// './e2e/login/login.e2e-spec.ts',
// './e2e/alerts-list/alerts-list.e2e-spec.ts',
'./e2e/alerts-list/configure-table/configure-table.e2e-spec.ts',
//'./e2e/alerts-list/save-search/save-search.e2e-spec.ts',
//'./e2e/alerts-list/tree-view/tree-view.e2e-spec.ts',
// './e2e/alerts-list/alert-filters/alert-filters.e2e-spec.ts',
// './e2e/alerts-list/alert-status/alerts-list-status.e2e-spec.ts',
// './e2e/alert-details/alert-status/alert-details-status.e2e-spec.ts',
//'./e2e/alerts-list/meta-alerts/meta-alert.e2e-spec.ts'
  ],
  capabilities: {
'browserName': 'chrome',
'chromeOptions': {
  args: [ "--headless", "--disable-gpu", "--window-size=1435,850" ],
  'prefs': {
'credentials_enable_service': false,
'profile': { 'password_manager_enabled': false}
  }
}
  },
  directConnect: true,
  baseUrl: 'http://192.168.99.100:4201/',
params: {
rest: {
url: '192.168.99.100:8082'
},
elasticsearch : {
url: '192.168.99.100:9210'
}
},
  framework: 'jasmine',
  jasmineNodeOpts: {
showColors: true,
defaultTimeoutInterval: 5,
print: function() {}
  },
  useAllAngular2AppRoots: true,
  rootElement: 'metron-alerts-root',
  beforeLaunch: function() {
require('ts-node').register({
  project: 'e2e'
});
  },
  onPrepare: function() {
var createMetaAlertsIndex =  
require('./e2e/utils/e2e_util').createMetaAlertsIndex;
createMetaAlertsIndex();
jasmine.getEnv().addReporter(new SpecReporter());
setTimeout(function() {
  browser.driver.executeScript(function() {
return {
  width: window.screen.availWidth,
  height: window.screen.availHeight
};
  }).then(function(result) {
browser.driver.manage().window().setSize(result.width, 
result.height);
  });
});
  },
  onComplete: function() {
var createMetaAlertsIndex =  
require('./e2e/utils/e2e_util').createMetaAlertsIndex;

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
Ok, I'm going to try running through some things.
One point, and I will admit this just may be me:

I see the same problem with this as I had with the original docker stuff.
The list of instructions for initial use is there, but I don't quite 
understand how after this initial use I would go back to this setup and use it 
day to day.  Do these steps need to happen every time?

Is there a way to script this such that there is an 
init_metron_testing_docker, start_metron_testing_docker  
shutdown_metron_testing_docker etc?




---

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

I’m trying to think of what information would be needed to look at this the
next time it happens.  If someone wanted to reproduce this.
Examples of a set of configurations that do not work maybe.
The metrics on the parser topologies involved ( how many parsers, message
rate ).
The platform_info.sh output for the machines running the indexing topology.
Any load information for those machines as well…..

Anyone think of anything else?



On December 7, 2017 at 07:45:26, Otto Fowler (ottobackwa...@gmail.com)
wrote:

We use TreeCache

.

When the configuration is updated in zookeeper, the configuration object in
the bolt is updated. This configuration is read on each message, so I think
from what I see new configurations should get picked up for the next
message.

I could be wrong though.



On December 7, 2017 at 06:47:15, Ali Nazemian (alinazem...@gmail.com) wrote:

Thank you very much. Unfortunately, reproducing all the situations are very
costly for us at this moment. We are kind of avoiding to hit that issue by
using the same batch size for all the feeds. Hopefully, with the new PR
Casey provided for the segregation of ES and HDFS, it will be very much
clear to tune them.

Do you know how the synchronization of indexing config will happen with the
topology? Does the topology gets synchronised by pulling the last configs
from ZK based on some background mechanism or it is based on an update
trigger? As I mentioned, based on our observation it looks like the
synchronization doesn't work until all the old messages in Kafka queue get
processed based on the old indexing configs.

Regards,
Ali

On Thu, Dec 7, 2017 at 12:33 AM, Otto Fowler 
wrote:

> Sorry,
> We flush for timeouts on every storm ‘tick’ message, not on every message.
>
>
>
> On December 6, 2017 at 08:29:51, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I have looked at it.
>
> We maintain batch lists for each sensor which gather messages to index.
> When we get a message that puts it over the batch size the messages are
> flushed and written to the target.
> There is also a timeout component, where the batch would be flushed based
> on timeout.
>
> While batch size checking occurs on a per sensor-message receipt basis,
> each message, regardless of sensor will trigger a check of the batch
> timeout for all the lists.
>
> At least that is what I think I see.
>
> Without understanding what the failures are for it is hard to see what the
> issue is.
>
> Do we have timing issues where all the lists are timing out all the time
> causing some kind of cascading failure for example?
> Does the number of sensors matter?  For example if only one sensor
> topology is running with batch setup X, is everything fine?  Do failures
> start after adding Nth additional sensor?
>
> Hopefully someone else on the list may have an idea.
> That code does not have any logging to speak of… well debug / trace
> logging that would help here either.
>
>
>
> On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Everything looks normal except the high number of failed tuples. Do you
> know how the indexing batch size works? Based on our observations it seems
> it doesn't update the messages that are in enrichments and indexing topics.
>
> On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler 
> wrote:
>
>> What do you see in the storm ui for the indexing topology?
>>
>>
>> On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Both hdfs and Elasticsearch batch sizes. There is no error in the logs.
>> It mpacts topology error rate and cause almost 90% error rate on indexing
>> tuples.
>>
>> On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:
>>
>> Where are you seeing the errors?  Screenshot?
>>
>>
>> On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> Which of the indexing options are you changing the batch size for?
>> HDFS?  Elasticsearch?  Both?
>>
>> Can you give an example?
>>
>>
>>
>> On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> No specific error in the logs. I haven't enabled debug/trace, though.
>>
>> On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
>> wrote:
>>
>>> My first thought is what are the errors when you get a high error rate?
>>>
>>>
>>> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
>>> wrote:
>>>
>>> Any thoughts?
>>>
>>> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > We have noticed recently that no matter what batch size we use for
>>> Metron
>>> > indexing feeds, as long as we start using different batch size for
>>> > different Metron feeds, indexing topology throughput will start
>>> dropping
>>> > due to the high error rate! So I was wondering whether based 

Re: Wiki Docs links seem wrong

[Simon Elliston Ball]

Awesome, many thanks!

> On 7 Dec 2017, at 13:08, Kyle Richardson  wrote:
> 
> Fixed.
> 
> -Kyle
> 
> On Thu, Dec 7, 2017 at 7:20 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> 
>> https://cwiki.apache.org/confluence/display/METRON/
>> Metron+User+Guide+-+per+release > confluence/display/METRON/Metron+User+Guide+-+per+release>
>> 
>> The links don’t seem to correspond to the versions on this page. Would be
>> happy to fix, but I don’t have wiki perms.
>> 
>> Simon

Re: Wiki Docs links seem wrong

[Kyle Richardson]

Fixed.

-Kyle

On Thu, Dec 7, 2017 at 7:20 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> https://cwiki.apache.org/confluence/display/METRON/
> Metron+User+Guide+-+per+release  confluence/display/METRON/Metron+User+Guide+-+per+release>
>
> The links don’t seem to correspond to the versions on this page. Would be
> happy to fix, but I don’t have wiki perms.
>
> Simon

Re: Heterogeneous indexing batch size for different Metron feeds

[Otto Fowler]

We use TreeCache

.

When the configuration is updated in zookeeper, the configuration object in
the bolt is updated. This configuration is read on each message, so I think
from what I see new configurations should get picked up for the next
message.

I could be wrong though.




On December 7, 2017 at 06:47:15, Ali Nazemian (alinazem...@gmail.com) wrote:

Thank you very much. Unfortunately, reproducing all the situations are very
costly for us at this moment. We are kind of avoiding to hit that issue by
using the same batch size for all the feeds. Hopefully, with the new PR
Casey provided for the segregation of ES and HDFS, it will be very much
clear to tune them.

Do you know how the synchronization of indexing config will happen with the
topology? Does the topology gets synchronised by pulling the last configs
from ZK based on some background mechanism or it is based on an update
trigger? As I mentioned, based on our observation it looks like the
synchronization doesn't work until all the old messages in Kafka queue get
processed based on the old indexing configs.

Regards,
Ali

On Thu, Dec 7, 2017 at 12:33 AM, Otto Fowler 
wrote:

> Sorry,
> We flush for timeouts on every storm ‘tick’ message, not on every message.
>
>
>
> On December 6, 2017 at 08:29:51, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I have looked at it.
>
> We maintain batch lists for each sensor which gather messages to index.
> When we get a message that puts it over the batch size the messages are
> flushed and written to the target.
> There is also a timeout component, where the batch would be flushed based
> on timeout.
>
> While batch size checking occurs on a per sensor-message receipt basis,
> each message, regardless of sensor will trigger a check of the batch
> timeout for all the lists.
>
> At least that is what I think I see.
>
> Without understanding what the failures are for it is hard to see what the
> issue is.
>
> Do we have timing issues where all the lists are timing out all the time
> causing some kind of cascading failure for example?
> Does the number of sensors matter?  For example if only one sensor
> topology is running with batch setup X, is everything fine?  Do failures
> start after adding Nth additional sensor?
>
> Hopefully someone else on the list may have an idea.
> That code does not have any logging to speak of… well debug / trace
> logging that would help here either.
>
>
>
> On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Everything looks normal except the high number of failed tuples. Do you
> know how the indexing batch size works? Based on our observations it seems
> it doesn't update the messages that are in enrichments and indexing topics.
>
> On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler 
> wrote:
>
>> What do you see in the storm ui for the indexing topology?
>>
>>
>> On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Both hdfs and Elasticsearch batch sizes. There is no error in the logs.
>> It mpacts topology error rate and cause almost 90% error rate on indexing
>> tuples.
>>
>> On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:
>>
>> Where are you seeing the errors?  Screenshot?
>>
>>
>> On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> Which of the indexing options are you changing the batch size for?
>> HDFS?  Elasticsearch?  Both?
>>
>> Can you give an example?
>>
>>
>>
>> On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> No specific error in the logs. I haven't enabled debug/trace, though.
>>
>> On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
>> wrote:
>>
>>> My first thought is what are the errors when you get a high error rate?
>>>
>>>
>>> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
>>> wrote:
>>>
>>> Any thoughts?
>>>
>>> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > We have noticed recently that no matter what batch size we use for
>>> Metron
>>> > indexing feeds, as long as we start using different batch size for
>>> > different Metron feeds, indexing topology throughput will start
>>> dropping
>>> > due to the high error rate! So I was wondering whether based on the
>>> current
>>> > indexing topology design, we have to choose the same batch size for
>>> all the
>>> > feeds or not. Otherwise, throughout will be dropped. I assume since it
>>> is
>>> > acceptable to use different batch sizes for different feeds, it is not
>>> > expected by design.
>>> >
>>> > Moreover, I have noticed in practice that even if we change the batch
>>> > size, it will not affect the messages that are already in enrichments
>>> or
>>> > indexing topics, and it will only affect the new 

Wiki Docs links seem wrong

[Simon Elliston Ball]

https://cwiki.apache.org/confluence/display/METRON/Metron+User+Guide+-+per+release
 


The links don’t seem to correspond to the versions on this page. Would be happy 
to fix, but I don’t have wiki perms. 

Simon

[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
I was more thinking about having to have a "Debugging Integration Tests" 
readme.
I think having the in-memory stuff available is worth getting some feedback.

I have in the past debugged through the kafka topic and storm stuff when 
looking to track down error messages and issues in the integration tests.  I 
don't know if that is something that happens often enough to warrant keeping 
the in-memory stuff.

But, if we don't keep them, then would we want the docker testing 
containers to be setup for debugging?  Would it work?  So the start.sh starts 
with the debugging on, known ports etc.



---

Re: Heterogeneous indexing batch size for different Metron feeds

[Ali Nazemian]

Thank you very much. Unfortunately, reproducing all the situations are very
costly for us at this moment. We are kind of avoiding to hit that issue by
using the same batch size for all the feeds. Hopefully, with the new PR
Casey provided for the segregation of ES and HDFS, it will be very much
clear to tune them.

Do you know how the synchronization of indexing config will happen with the
topology? Does the topology gets synchronised by pulling the last configs
from ZK based on some background mechanism or it is based on an update
trigger? As I mentioned, based on our observation it looks like the
synchronization doesn't work until all the old messages in Kafka queue get
processed based on the old indexing configs.

Regards,
Ali

On Thu, Dec 7, 2017 at 12:33 AM, Otto Fowler 
wrote:

> Sorry,
> We flush for timeouts on every storm ‘tick’ message, not on every message.
>
>
>
> On December 6, 2017 at 08:29:51, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I have looked at it.
>
> We maintain batch lists for each sensor which gather messages to index.
> When we get a message that puts it over the batch size the messages are
> flushed and written to the target.
> There is also a timeout component, where the batch would be flushed based
> on timeout.
>
> While batch size checking occurs on a per sensor-message receipt basis,
> each message, regardless of sensor will trigger a check of the batch
> timeout for all the lists.
>
> At least that is what I think I see.
>
> Without understanding what the failures are for it is hard to see what the
> issue is.
>
> Do we have timing issues where all the lists are timing out all the time
> causing some kind of cascading failure for example?
> Does the number of sensors matter?  For example if only one sensor
> topology is running with batch setup X, is everything fine?  Do failures
> start after adding Nth additional sensor?
>
> Hopefully someone else on the list may have an idea.
> That code does not have any logging to speak of… well debug / trace
> logging that would help here either.
>
>
>
> On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com)
> wrote:
>
> Everything looks normal except the high number of failed tuples. Do you
> know how the indexing batch size works? Based on our observations it seems
> it doesn't update the messages that are in enrichments and indexing topics.
>
> On Thu, Dec 7, 2017 at 12:13 AM, Otto Fowler 
> wrote:
>
>> What do you see in the storm ui for the indexing topology?
>>
>>
>> On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Both hdfs and Elasticsearch batch sizes. There is no error in the logs.
>> It mpacts topology error rate and cause almost 90% error rate on indexing
>> tuples.
>>
>> On 6 Dec. 2017 00:20, "Otto Fowler"  wrote:
>>
>> Where are you seeing the errors?  Screenshot?
>>
>>
>> On December 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> Which of the indexing options are you changing the batch size for?
>> HDFS?  Elasticsearch?  Both?
>>
>> Can you give an example?
>>
>>
>>
>> On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> No specific error in the logs. I haven't enabled debug/trace, though.
>>
>> On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler 
>> wrote:
>>
>>> My first thought is what are the errors when you get a high error rate?
>>>
>>>
>>> On December 4, 2017 at 19:34:29, Ali Nazemian (alinazem...@gmail.com)
>>> wrote:
>>>
>>> Any thoughts?
>>>
>>> On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian 
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > We have noticed recently that no matter what batch size we use for
>>> Metron
>>> > indexing feeds, as long as we start using different batch size for
>>> > different Metron feeds, indexing topology throughput will start
>>> dropping
>>> > due to the high error rate! So I was wondering whether based on the
>>> current
>>> > indexing topology design, we have to choose the same batch size for
>>> all the
>>> > feeds or not. Otherwise, throughout will be dropped. I assume since it
>>> is
>>> > acceptable to use different batch sizes for different feeds, it is not
>>> > expected by design.
>>> >
>>> > Moreover, I have noticed in practice that even if we change the batch
>>> > size, it will not affect the messages that are already in enrichments
>>> or
>>> > indexing topics, and it will only affect the new messages that are
>>> coming
>>> > to the parser. Therefore, we need to let all the messages pass the
>>> indexing
>>> > topology so that we can change the batch size!
>>> >
>>> > It would be great if we can have more details regarding the design of
>>> this
>>> > section so we can understand our observations are based on the design
>>> or
>>> > some kind of bug.
>>> >
>>> > Regards,
>>> > Ali
>>> >
>>>
>>>
>>>
>>> --
>>> A.Nazemian
>>>
>>>
>>
>>
>> --
>>