[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/861 ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155698109
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
agreed, and added.
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155664814
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
We should document that the system fields are protected as well
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155664254
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
It should be documented then in the readme
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155659293 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.stellar.dsl.Context; +import org.json.simple.JSONObject; +import org.junit.Assert; +import org.junit.Test; + +import com.google.common.collect.Iterables; + --- End diff -- It's a pretty broad reaching concern when you account for the impact of remove, stellar, and the other field transforms. I'm happy to look at it, but I expect it will just need to be reviewed again as part of a follow on. ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155648096 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.stellar.dsl.Context; +import org.json.simple.JSONObject; +import org.junit.Assert; +import org.junit.Test; + +import com.google.common.collect.Iterables; + --- End diff -- That test would belong in this ticket ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155649024
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
Yes, that would break. I'd say that was flat out user error, and not
something to guard against, or indeed something it's possible to guard against.
I guess the key thing here is that it requires maintaining order in field
transformations.
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155646423
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
If a user has a stellar transform (existing ) that takes field X as input,
but doesn't want X as output, and puts the SELECT in the wrong order, could
they break their existing transform because X now missing?
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155646268 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/SelectTransformation.java --- @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; + +import org.apache.metron.stellar.dsl.Context; + --- End diff -- I've added to relevant readme (in parsers docs) which would seem a more likely place than javadoc (if we want to add javadoc, which we don't publish anyway, we should also look at adding this to everything in this area, which feels out of scope). Not even the stellar version gets javadoc! ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155645745 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.stellar.dsl.Context; +import org.json.simple.JSONObject; +import org.junit.Assert; +import org.junit.Test; + +import com.google.common.collect.Iterables; + --- End diff -- yeah, makes sense, I suspect on a different ticket. ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155645303
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
And order doesn't really matter, you might have this first, or last, or
sandwiched between two stellar blocks. That said, I would expect the most
common use case to be have this one at the end of a set.
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155645129
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
This is the opposite of remove.
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155645013 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.stellar.dsl.Context; +import org.json.simple.JSONObject; +import org.junit.Assert; +import org.junit.Test; + +import com.google.common.collect.Iterables; + --- End diff -- or a FieldTransformer ( level up ) test more likely. ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/861#discussion_r155644387
--- Diff: metron-platform/metron-parsers/README.md ---
@@ -216,6 +216,23 @@ whenever `field2` exists and whose corresponding equal
to 'foo':
}
```
+* `SELECT`: This transformation filters the fields in the message to
include only the configured output fields, and drops any not explicitly
included.
+
+For example:
+```
+{
+...
+"fieldTransformations" : [
+ {
+"output" : ["field1", "field2" ]
+ , "transformation" : "SELECT"
+ }
+ ]
+}
+```
+
+when applied to a message containing keys field1, field2 and field3, will
only output the first two.
+
--- End diff --
How does this relate to REMOVE? If I have more than one transformation
should this be last or does it not matter?
---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155641855 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.stellar.dsl.Context; +import org.json.simple.JSONObject; +import org.junit.Assert; +import org.junit.Test; + +import com.google.common.collect.Iterables; + --- End diff -- that feels a lot more like an integration test, or a broader test than belongs in this one piece. Otherwise every single test of transformations is going to end up with combinatorial complexity. ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155630936 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/SelectTransformation.java --- @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; + +import org.apache.metron.stellar.dsl.Context; + --- End diff -- Missing Javadoc. The purpose and an example output of this transformation would be nice ---
[GitHub] metron pull request #861: METRON-1341 Implemented SELECT transformer to proj...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/861#discussion_r155631654 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/field/transformation/SelectTransformationTest.java --- @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.field.transformation; + +import java.util.HashMap; + +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.metron.common.configuration.FieldTransformer; +import org.apache.metron.common.configuration.SensorParserConfig; +import org.apache.metron.stellar.dsl.Context; +import org.json.simple.JSONObject; +import org.junit.Assert; +import org.junit.Test; + +import com.google.common.collect.Iterables; + --- End diff -- Do we at a test where we have multiple transformers? STELLAR , SELECT etc? ---
