Re: DISCUSS: Quick change to parser config

[Simon Elliston Ball]

Hmmm… Actually, I kinda like that. 

May want a little refactoring in the back for clarity. 

My question about whether we could ever imagine this ‘cleanup policy’ applying 
to other transforms would sway me to the field rather than transformation name 
approach though. 

Simon

> On 1 Dec 2017, at 01:17, Otto Fowler  wrote:
> 
> Or, we can create new transformation types
> STELLAR_COMPLETE, which may be more in line with the original design.
> 
> 
> 
> On November 30, 2017 at 20:14:46, Otto Fowler (ottobackwa...@gmail.com 
> ) wrote:
> 
>> I would suggest that instead of explicitly having “complete”, we have 
>> “operation”:”complete”
>> 
>> Such that we can have multiple transformations, each with a different 
>> “operation”.
>> No operation would be the status quo ante, if we can do it so that we don’t 
>> get errors with old configs and the keep same behavior.
>> 
>> { 
>> "fieldTransformations": [ 
>> { 
>> "transformation": "STELLAR", 
>> “operation": “complete", 
>> "output": ["ip_src_addr", "ip_dst_addr"], 
>> "config": { 
>> "ip_src_addr": "ipSrc", 
>> "ip_dest_addr": "ipDst" 
>> } ,
>> { 
>> "transformation": "STELLAR", 
>> “operation": “SomeOtherThing", 
>> "output": [“foo", “bar"], 
>> "config": { 
>> “foo": “TO_UPPER(foo)", 
>> “bar": “TO_LOWER(bar)" 
>> } 
>> } 
>> ] 
>> } 
>> 
>> 
>> Sorry for the junk examples, but hopefully it makes sense.
>> 
>> 
>> 
>> 
>> 
>> On November 30, 2017 at 20:00:06, Simon Elliston Ball 
>> (si...@simonellistonball.com ) wrote:
>> 
>>> I’m looking at the way parser config works, and transformation of field 
>>> from their native names in, for example the ASA or CEF parsers, into a 
>>> standard data model.
>>> 
>>> At the moment I would do something like this:
>>> 
>>> assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might 
>>> have:
>>> 
>>> {
>>> "fieldTransformations": [
>>> {
>>> "transformation": "STELLAR",
>>> "output": ["ip_src_addr", "ip_dst_addr", "message"],
>>> "config": {
>>> "ip_src_addr": "ipSrc",
>>> "ip_dest_addr": "ipDst"
>>> }
>>> }
>>> ]
>>> }
>>> 
>>> which leave me with the field set:
>>> [ipSrc, ipDst, pointlessExtraStuff, message, ip_src_addr, ip_dest_addr]
>>> 
>>> unless I go with:-
>>> 
>>> {
>>> "fieldTransformations": [
>>> {
>>> "transformation": "STELLAR",
>>> "output": ["ip_src_addr", "ip_dst_addr", "message"],
>>> "config": {
>>> "ip_src_addr": "ipSrc",
>>> "ip_dest_addr": "ipDst",
>>> "pointlessExtraStuff": null,
>>> "ipSrc": null,
>>> "ipDst": null
>>> }
>>> }
>>> ]
>>> }
>>> 
>>> which seems a little over verbose.
>>> 
>>> Do you think it would be valuable to add a switch of some sort on the 
>>> transformation to make it “complete”, i.e. to only preserve fields which 
>>> are explicitly set.
>>> 
>>> To my mind, this breaks a principal of mutability, but gives us much much 
>>> cleaner mapping of data.
>>> 
>>> I would propose something like:
>>> 
>>> {
>>> "fieldTransformations": [
>>> {
>>> "transformation": "STELLAR",
>>> "complete": true,
>>> "output": ["ip_src_addr", "ip_dst_addr", "message"],
>>> "config": {
>>> "ip_src_addr": "ipSrc",
>>> "ip_dest_addr": "ipDst"
>>> }
>>> }
>>> ]
>>> }
>>> 
>>> which would give me the set ["ip_src_addr", "ip_dst_addr", "message”] 
>>> effectively making the nulling in my previous example implicit.
>>> 
>>> Thoughts?
>>> 
>>> Also, in the second scenario, if ‘output' were to be empty would we assume 
>>> that the output field set should be ["ip_src_addr", “ip_dst_addr”]?
>>> 
>>> Simon

Re: DISCUSS: Quick change to parser config

[Simon Elliston Ball]

Do you have any thoughts on what these other operations might be? 

What I’m imagining is something that basically specifies a policy on how to 
handle things that the transformation block does not explicitly handle. Right 
now, we just leave them along and they flow through. 

Would “policy”: “explicit”, or “policy”: “onlyExplict” make sense and give the 
flex? 

To my mind “operation” implies further transformation, which would just be 
another block, no? 

Maybe it’s just semantic pedantry on my part… would we see this sort of policy 
logic applying to other transformations? It doesn’t really make sense for 
“remove”, and well… who cares about any of the other legacy transforms now we 
have Stellar :) 

Simon

> On 1 Dec 2017, at 01:14, Otto Fowler  wrote:
> 
> I would suggest that instead of explicitly having “complete”, we have 
> “operation”:”complete”
> 
> Such that we can have multiple transformations, each with a different 
> “operation”.
> No operation would be the status quo ante, if we can do it so that we don’t 
> get errors with old configs and the keep same behavior.
> 
> { 
> "fieldTransformations": [ 
> { 
> "transformation": "STELLAR", 
> “operation": “complete", 
> "output": ["ip_src_addr", "ip_dst_addr"], 
> "config": { 
> "ip_src_addr": "ipSrc", 
> "ip_dest_addr": "ipDst" 
> } ,
> { 
> "transformation": "STELLAR", 
> “operation": “SomeOtherThing", 
> "output": [“foo", “bar"], 
> "config": { 
> “foo": “TO_UPPER(foo)", 
> “bar": “TO_LOWER(bar)" 
> } 
> } 
> ] 
> } 
> 
> 
> Sorry for the junk examples, but hopefully it makes sense.
> 
> 
> 
> 
> 
> On November 30, 2017 at 20:00:06, Simon Elliston Ball 
> (si...@simonellistonball.com ) wrote:
> 
>> I’m looking at the way parser config works, and transformation of field from 
>> their native names in, for example the ASA or CEF parsers, into a standard 
>> data model.  
>> 
>> At the moment I would do something like this:  
>> 
>> assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might 
>> have: 
>> 
>> { 
>> "fieldTransformations": [ 
>> { 
>> "transformation": "STELLAR", 
>> "output": ["ip_src_addr", "ip_dst_addr", "message"], 
>> "config": { 
>> "ip_src_addr": "ipSrc", 
>> "ip_dest_addr": "ipDst" 
>> } 
>> } 
>> ] 
>> } 
>> 
>> which leave me with the field set:  
>> [ipSrc, ipDst, pointlessExtraStuff, message, ip_src_addr, ip_dest_addr] 
>> 
>> unless I go with:- 
>> 
>> { 
>> "fieldTransformations": [ 
>> { 
>> "transformation": "STELLAR", 
>> "output": ["ip_src_addr", "ip_dst_addr", "message"], 
>> "config": { 
>> "ip_src_addr": "ipSrc", 
>> "ip_dest_addr": "ipDst", 
>> "pointlessExtraStuff": null, 
>> "ipSrc": null, 
>> "ipDst": null 
>> } 
>> } 
>> ] 
>> } 
>> 
>> which seems a little over verbose.  
>> 
>> Do you think it would be valuable to add a switch of some sort on the 
>> transformation to make it “complete”, i.e. to only preserve fields which are 
>> explicitly set.  
>> 
>> To my mind, this breaks a principal of mutability, but gives us much much 
>> cleaner mapping of data.  
>> 
>> I would propose something like: 
>> 
>> { 
>> "fieldTransformations": [ 
>> { 
>> "transformation": "STELLAR", 
>> "complete": true, 
>> "output": ["ip_src_addr", "ip_dst_addr", "message"], 
>> "config": { 
>> "ip_src_addr": "ipSrc", 
>> "ip_dest_addr": "ipDst" 
>> } 
>> } 
>> ] 
>> } 
>> 
>> which would give me the set ["ip_src_addr", "ip_dst_addr", "message”] 
>> effectively making the nulling in my previous example implicit.  
>> 
>> Thoughts?  
>> 
>> Also, in the second scenario, if ‘output' were to be empty would we assume 
>> that the output field set should be ["ip_src_addr", “ip_dst_addr”]?  
>> 
>> Simon

Re: DISCUSS: Quick change to parser config

[Otto Fowler]

Or, we can create new transformation types
STELLAR_COMPLETE, which may be more in line with the original design.



On November 30, 2017 at 20:14:46, Otto Fowler (ottobackwa...@gmail.com)
wrote:

I would suggest that instead of explicitly having “complete”, we have
“operation”:”complete”

Such that we can have multiple transformations, each with a different
“operation”.
No operation would be the status quo ante, if we can do it so that we don’t
get errors with old configs and the keep same behavior.

{
"fieldTransformations": [
{
"transformation": "STELLAR",
“operation": “complete",
"output": ["ip_src_addr", "ip_dst_addr"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
} ,
{
"transformation": "STELLAR",
“operation": “SomeOtherThing",
"output": [“foo", “bar"],
"config": {
“foo": “TO_UPPER(foo)",
“bar": “TO_LOWER(bar)"
}
}
]
}


Sorry for the junk examples, but hopefully it makes sense.




On November 30, 2017 at 20:00:06, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

I’m looking at the way parser config works, and transformation of field
from their native names in, for example the ASA or CEF parsers, into a
standard data model.

At the moment I would do something like this:

assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might
have:

{
"fieldTransformations": [
{
"transformation": "STELLAR",
"output": ["ip_src_addr", "ip_dst_addr", "message"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
}
}
]
}

which leave me with the field set:
[ipSrc, ipDst, pointlessExtraStuff, message, ip_src_addr, ip_dest_addr]

unless I go with:-

{
"fieldTransformations": [
{
"transformation": "STELLAR",
"output": ["ip_src_addr", "ip_dst_addr", "message"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst",
"pointlessExtraStuff": null,
"ipSrc": null,
"ipDst": null
}
}
]
}

which seems a little over verbose.

Do you think it would be valuable to add a switch of some sort on the
transformation to make it “complete”, i.e. to only preserve fields which
are explicitly set.

To my mind, this breaks a principal of mutability, but gives us much much
cleaner mapping of data.

I would propose something like:

{
"fieldTransformations": [
{
"transformation": "STELLAR",
"complete": true,
"output": ["ip_src_addr", "ip_dst_addr", "message"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
}
}
]
}

which would give me the set ["ip_src_addr", "ip_dst_addr", "message”]
effectively making the nulling in my previous example implicit.

Thoughts?

Also, in the second scenario, if ‘output' were to be empty would we assume
that the output field set should be ["ip_src_addr", “ip_dst_addr”]?

Simon

Re: DISCUSS: Quick change to parser config

[Otto Fowler]

I would suggest that instead of explicitly having “complete”, we have
“operation”:”complete”

Such that we can have multiple transformations, each with a different
“operation”.
No operation would be the status quo ante, if we can do it so that we don’t
get errors with old configs and the keep same behavior.

{
"fieldTransformations": [
{
"transformation": "STELLAR",
“operation": “complete",
"output": ["ip_src_addr", "ip_dst_addr"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
} ,
{
"transformation": "STELLAR",
“operation": “SomeOtherThing",
"output": [“foo", “bar"],
"config": {
“foo": “TO_UPPER(foo)",
“bar": “TO_LOWER(bar)"
}
}
]
}


Sorry for the junk examples, but hopefully it makes sense.




On November 30, 2017 at 20:00:06, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

I’m looking at the way parser config works, and transformation of field
from their native names in, for example the ASA or CEF parsers, into a
standard data model.

At the moment I would do something like this:

assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might
have:

{
"fieldTransformations": [
{
"transformation": "STELLAR",
"output": ["ip_src_addr", "ip_dst_addr", "message"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
}
}
]
}

which leave me with the field set:
[ipSrc, ipDst, pointlessExtraStuff, message, ip_src_addr, ip_dest_addr]

unless I go with:-

{
"fieldTransformations": [
{
"transformation": "STELLAR",
"output": ["ip_src_addr", "ip_dst_addr", "message"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst",
"pointlessExtraStuff": null,
"ipSrc": null,
"ipDst": null
}
}
]
}

which seems a little over verbose.

Do you think it would be valuable to add a switch of some sort on the
transformation to make it “complete”, i.e. to only preserve fields which
are explicitly set.

To my mind, this breaks a principal of mutability, but gives us much much
cleaner mapping of data.

I would propose something like:

{
"fieldTransformations": [
{
"transformation": "STELLAR",
"complete": true,
"output": ["ip_src_addr", "ip_dst_addr", "message"],
"config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
}
}
]
}

which would give me the set ["ip_src_addr", "ip_dst_addr", "message”]
effectively making the nulling in my previous example implicit.

Thoughts?

Also, in the second scenario, if ‘output' were to be empty would we assume
that the output field set should be ["ip_src_addr", “ip_dst_addr”]?

Simon

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
Per the conversation above, i'm going to take a stab at the attributed 
approach.
I think the Stellar Functions should be a separate Jira.


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154245631
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

DISCUSS: Quick change to parser config

[Simon Elliston Ball]

I’m looking at the way parser config works, and transformation of field from 
their native names in, for example the ASA or CEF parsers, into a standard data 
model. 

At the moment I would do something like this: 

assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might 
have:

{
  "fieldTransformations": [
{
  "transformation": "STELLAR",
  "output": ["ip_src_addr", "ip_dst_addr", "message"],
  "config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
  }
}
  ]
}

which leave me with the field set: 
[ipSrc, ipDst, pointlessExtraStuff, message, ip_src_addr, ip_dest_addr]

unless I go with:-

{
  "fieldTransformations": [
{
  "transformation": "STELLAR",
  "output": ["ip_src_addr", "ip_dst_addr", "message"],
  "config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst",
"pointlessExtraStuff": null,
"ipSrc": null,
"ipDst": null
  }
}
  ]
}

which seems a little over verbose. 

Do you think it would be valuable to add a switch of some sort on the 
transformation to make it “complete”, i.e. to only preserve fields which are 
explicitly set. 

To my mind, this breaks a principal of mutability, but gives us much much 
cleaner mapping of data. 

I would propose something like:

{
  "fieldTransformations": [
{
  "transformation": "STELLAR",
  "complete": true,
  "output": ["ip_src_addr", "ip_dst_addr", "message"],
  "config": {
"ip_src_addr": "ipSrc",
"ip_dest_addr": "ipDst"
  }
}
  ]
}

which would give me the set ["ip_src_addr", "ip_dst_addr", "message”] 
effectively making the nulling in my previous example implicit. 

Thoughts? 

Also, in the second scenario, if ‘output' were to be empty would we assume that 
the output field set should be ["ip_src_addr", “ip_dst_addr”]? 

Simon

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154241894
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154231709
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

One more thought - the code for option 3 does add more code than the 
copy/paste, but in spite of that I think it puts us closer to being able to 
delete a lot of the redundant and/or dead code in config utils. So I think it's 
a net positive.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154215790
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

Sounds good.  Just so I'm clear about my preferred approach (option 3 
there), I went ahead and implemented it in a 
[PR](https://github.com/nickwallen/metron/pull/12) against your PR so you can 
see what I was talking about.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154209165
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

Your first suggestion with the callback just seems less obvious and clear 
to me (IMHO).

I'll try and think through your 2nd and 3rd suggestions, but (at least 
right now) I'm not seeing something that doesn't add more complexity to 
ConfigurationUtils.  

But clearly I could be totally wrong here.  Let me think on it.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154208493
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

Yes, I had the same thought and tried for a bit to refactor it.  I landed 
on this because the various other ways to do this either (1) seemed more 
complex and less obvious as to what we are actually doing here or (2) lead down 
a path of heavy refactoring of ConfigurationUtils.  Both of which i wanted to 
avoid


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154208318
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -156,7 +156,7 @@ public static void 
writeConfigToZookeeper(ConfigurationType configType,Optional<
   }
 
   private static String getConfigZKPath(ConfigurationType configType, 
Optional configName) {
-String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName : "";
+String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName.get() : "";
--- End diff --

Yeah, this is not one of those things that can be done as a side-effect of 
another PR.  I considered refactoring it as part of the zookeeper refactoring 
that I did earlier.  It needs more direct attention and refactoring.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154207865
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -156,7 +156,7 @@ public static void 
writeConfigToZookeeper(ConfigurationType configType,Optional<
   }
 
   private static String getConfigZKPath(ConfigurationType configType, 
Optional configName) {
-String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName : "";
+String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName.get() : "";
--- End diff --

I think that's because we handle global config in a special way. This whole 
class needs refactored, and I'd have done more with the original patch PR had 
it not blown up scope.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154207614
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -156,7 +156,7 @@ public static void 
writeConfigToZookeeper(ConfigurationType configType,Optional<
   }
 
   private static String getConfigZKPath(ConfigurationType configType, 
Optional configName) {
-String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName : "";
+String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName.get() : "";
--- End diff --

We need to refactor `ConfigurationUtils` so badly that it hurts.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154207266
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -156,7 +156,7 @@ public static void 
writeConfigToZookeeper(ConfigurationType configType,Optional<
   }
 
   private static String getConfigZKPath(ConfigurationType configType, 
Optional configName) {
-String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName : "";
+String pathSuffix = configName.isPresent() && configType != GLOBAL ? 
"/" + configName.get() : "";
--- End diff --

Yeah, this confused me too.  There are a lot of paths through 
ConfigurationUtils.  Some paths worked, others didn't.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154206090
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

Oh, an even better option (IMO) would be to add a 
`writeSensorConfigToZookeeper(String sensorType, byte[] configData, 
CuratorFramework client)` method to `ConfigurationType`.  Anyway, I'll stop ;)


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154204458
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

The other option here is to add a new method called 
`writeSensorConfigToZookeeper(ConfigurationType type, String sensorType, byte[] 
configData, CuratorFramework client)` that calls the appropriate 
`writeSensorXConfigToZookeeper` call.


---

[GitHub] metron pull request #851: METRON-1336 Patching Can Result in Bad Configurati...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/851#discussion_r154202685
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
 ---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String 
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
-  public static void uploadConfigsToZookeeper(String rootFilePath, 
CuratorFramework client,
-  ConfigurationType type, Optional configName) throws 
Exception {
+  public static void uploadConfigsToZookeeper(
+  String rootFilePath,
+  CuratorFramework client,
+  ConfigurationType type,
+  Optional configName) throws Exception {
+
 switch (type) {
+
   case GLOBAL:
 final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
 if (globalConfig.length > 0) {
   setupStellarStatically(client, Optional.of(new 
String(globalConfig)));
   writeGlobalConfigToZookeeper(globalConfig, client);
 }
 break;
-  case PARSER: // intentional pass-through
-  case ENRICHMENT: // intentional pass-through
-  case INDEXING:
-Map sensorIndexingConfigs = 
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
-  writeConfigToZookeeper(type, configName, 
sensorIndexingConfigs.get(sensorType), client);
+
+  case PARSER: {
+Map configs = 
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --

These cases look like they are cut and pasted which seems like code smell 
to me and might be a maintenance issue.  Can we extract the common code for 
Parser, Enrichment, and Indexing into a separate function that is called here?  
Perhaps something like:
```
void writeSensorConfigs(ConfigurationType type, Optional 
configName, BiFunction callback) {
   Map configs = readSensorConfigsFromFile(rootFilePath, 
type, configName);
  for(String sensorType : configs.keySet()) {
byte[] configData = configs.get(sensorType);
callback.apply(sensorType, configData);
  }
}
```
which could be called from this case via `writeSensorConfigs(PARSER, 
configName, (sensorType, configData) -> 
writeSensorParserConfigToZookeeper(sensorType, configData, client));`

Take the above as a very rough suggestion and you can feel to abstract it 
however you wish.


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
I am glad for the interest in this PR, and that it seems to have sparked 
some great ideas for continuing on.  

What I would like to do is line it up as follows

1. This PR with it's current scope and focus
2. New Jira and PR(s) for the Stellar Functions/Namespace that @cestella  
and @simonellistonball  mentioned
3. Some research and possible prototyping of the Attributed approach 
@nickwallen has suggested ( which I agree with )

Over the course of that work, and other work identified through working and 
reviewing it, I will iteratively refactor to a common code and reusable 
approach.




---

[GitHub] metron issue #855: METRON-1338 Excluding retry files from RAT check

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/855
  
+1 by inspection


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/814


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154189837
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/814
  
Ok, my +1 stands


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154185591
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154171805
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154155977
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154155446
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154135697
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154133974
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154133309
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154124236
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/856
  
@simonellistonball Agree to the namespace idea.  My bad :)


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154123254
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154118407
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154114714
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r15404
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154110765
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154110355
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154109552
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r154105765
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
 ---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements.  See the NOTICE file distributed with this work for 
additional information regarding
+ * copyright ownership.  The ASF licenses this file to you under the 
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with 
the License.  You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF 
ANY KIND, either express
+ * or implied. See the License for the specific language governing 
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import 
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured / 
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements 
StellarConfiguredStatementReporter {
+
+  public enum Type {
+ENRICHMENT, THREAT_INTEL;
+  }
+
+  public StellarStatementReporter() {
+  }
+
+  @Override
+  public String getName() {
+return "Apache Metron";
+  }
+
+  @Override
+  public void vist(CuratorFramework client, StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+  }
+
+  private void visitParserConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() + 
"/" + child);
+  try {
+SensorParserConfig parserConfig = 
SensorParserConfig.fromBytes(data);
+List transformations = 
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+  if 
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
 {
+FieldTransformation transformation = 
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+  List names = Arrays
+  .asList(getName(), PARSER.toString(), 
parserConfig.getSensorTopic(), k);
+  visitor.visit(names, v.toString());
+});
+  }
+});
+  } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(), 
child);
+errorConsumer.consume(names, e);
+  }
+}
+  }
+
+  @SuppressWarnings("unchecked")
+  private void visitEnrichmentConfigs(CuratorFramework client, 
StatementReportVisitor visitor,
+  ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children = 
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+  byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot() 
+ "/" + child);
+  try {
+final SensorEnrichmentConfig sensorEnrichmentConfig = 
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = 

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
Ok, I'll check it out after i do my full dev test on validate


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/814
  
sorry, it's not set, so it's null.


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
Ok,  what was is_alert in this test?


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/814
  
actually, hold on there, If ound one more bug:
```
[Stellar]>>> match { is_alert == null => null, is_alert => 'alert', default 
=> 'nah' }
[!] null
java.lang.NullPointerException
at 
org.apache.metron.stellar.common.StellarCompiler.lambda$exitMatchClauseAction$19(StellarCompiler.java:752)
at 
org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:190)
at 
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152)
at 
org.apache.metron.stellar.common.shell.StellarExecutor.execute(StellarExecutor.java:292)
at 
org.apache.metron.stellar.common.shell.StellarShell.handleStellar(StellarShell.java:282)
at 
org.apache.metron.stellar.common.shell.StellarShell.execute(StellarShell.java:514)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/814
  
Alright, +1 otto, you did great work here.  I'm very, very impressed.  
Thanks to @jjmeyer0 for the careful review and assistance.  Open source dev at 
its finest.


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
@simonellistonball, yes, the namespace should be part of the jira and 
interface design


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
So, the scenario here is checking things that *were* valid when uploaded, 
but have been invalidated by external changes ( language changes ).  I would 
like to keep the magic specific.

I think the functionality for the management functions is valid, but can we 
do that as a separate Jira/PR?  I'll do it, I just want to keep this tight.  If 
you create the jira and assign it to me that would be super.

I would do the files on disk using the management functions as well.

So we just need to think of the stellar interface for calling
`VALIDATE`  with a string, and with a file path.  Also saying what 
configuration type it is.

Does that make sense?


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/856
  
@cestella I would say that proposed validate function has to be very much 
in a namespace. It feels like a name that would be much more useful for a 
function replacing our current approach to global validation in the future than 
config validation, other than that it sounds like a good idea.


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/856
  
Also, it might be useful for `%validate_configured_expressions` to take a 
file path so you can validate a set of configs on disk (again, if it gets to 
zookeeper, zk_load_utils.sh should fail if it's invalid)


---

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/856
  
Any chance we can add a `VALIDATE(str, type)` function to the stellar 
management functions where str is the json blob string for the config and the 
type is the type of config?  Generally the goal is to disallow invalid stellar 
to get pushed to zookeeper via `zk_load_utils.sh`, so I suspect a function 
would be more useful in the situation where you're constructing a config in the 
REPL via management functions and want to validate it before pushing it.



---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
PR: https://github.com/apache/metron/pull/856  adds capability to use the 
stellar shell to validate stellar statements at rest out of ZK


---

[GitHub] metron issue #854: Experimental Improvements - Feedback Only - Do Not Merge

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/854
  
Yes, I would just get in the habit of running `./mvn clean install 
-DskipTests` rather than what we normally do.


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/814
  
Yeah, let me run this up once more in the REPL and take a final look at the 
docs, but I've been monitoring and I like what I see so far.


---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/856

METRON-1339 Stellar Shell functionality to verify stored stellar statements 

This will allow users to check their deployed statements, say after 
upgrade, when they are at rest ( and would fail on use ).
In other words, they were valid when stored, but are not now because of 
stellar changes, such as new keywords.

The interface `StellarConfiguredStatementReporter`, which is 
`@IndexSubclasses` ( ClassIndex) marked, allows the shell to discover reporters 
that can provide statements for validation.  This discovery allows de-coupling 
of stellar and 'hosts' that know about the location of the stored statements, 
and the configuration structure details.

> We do mention the configurations in the shell output at this time.

`metron-common` implements this interface, and can run through visiting all 
the configurations.

A new magic keyword was added ` %validate_configured_expressions`
When executed, the shell 

- discovers the reporters through class index 
- visits the reports, with callbacks for visits or errors
- per visit ( which is called for a specific stellar statement ) the 
statement is compiled and errors reported
- if the entire config fails ( threat triage stellar errors fail on 
deserialize so we don't get to do ANY enrichment visits in that case ) the 
error callback handles that

I'm getting this out there, still a couple of things todo:

1. full dev run. I have been testing with stellar external to full dev 
iteratively
2. readme
3. steps to test
4. unit test




### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron 
stellar_verify_deployed_shell

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #856


commit a5087f3a170eeda6ee778397c919d9eddd5597e2
Author: Otto Fowler 
Date:   2017-11-30T14:15:40Z

Stellar shell functionality to verify stellar statements.

This will allow users to check their deployed statements, say after 
upgrade, when they are at rest ( and would fail on use ).
In other words, they were valid when stored, but are not now because of 
stellar changes, such as new keywords.

The interface StellarConfiguredStatementReporter, which is @IndexSubclasses 
marked, allows the shell to discover
reporters that can provide statements for validation.  This discovery 
allows de-coupling of stellar and 'hosts' that
know about the location of the stored statements, and the configuration 
structure details.

We do mention the configurations in the shell output at this time.

metron-common implements this interface, and can run through visiting all 
the configurations.




---

[GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/823
  
I'm all set.  +1


---

[GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/823
  
@jasper-k thanks for the contribution.  Really great job.
+1

@cestella @justinleet  are you guys all set?


---

[GitHub] metron pull request #823: METRON-1286 Add MIN & MAX Stellar functions

[jasper-k]

Github user jasper-k closed the pull request at:

https://github.com/apache/metron/pull/823


---

[GitHub] metron pull request #823: METRON-1286 Add MIN & MAX Stellar functions

[jasper-k]

GitHub user jasper-k reopened a pull request:

https://github.com/apache/metron/pull/823

METRON-1286 Add MIN & MAX Stellar functions

## Contributor Comments
Currently Stellar lacks straightforward MAX & MIN functions that take just 
a list of values as input.
The functions STATS_MAX and STATS_MIN only take the internal Stellar 
statistics object as input.
Having MAX and MIN will be easier and understandable to most users


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasper-k/metron-apache master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #823


commit 50b3335b9699ed0bbed78e8ca37e3b8f639a7e0c
Author: jknulst 
Date:   2017-11-01T11:47:44Z

METRON-1286 Add MIN & MAX Stellar functions

commit eace1fec397a0297ee42fd59b8e824e1a34d9a90
Author: jknulst 
Date:   2017-11-14T07:37:36Z

METRON-1286 Min Max, added Apache header

commit f6e31c715caa91b71317b4338c98f3e6d37d3f2f
Author: jknulst 
Date:   2017-11-14T07:38:27Z

Merge branch 'master' of https://github.com/apache/metron

commit 9b596f45ee9b64adce29f32c0329f49efe5cf2c6
Author: jknulst 
Date:   2017-11-14T08:06:50Z

METRON-1286 Updated README.md

commit e606503e9a0d1c70b44e6098ab47e49b219c0410
Author: jknulst 
Date:   2017-11-29T19:38:12Z

Merge branch 'master' of https://github.com/apache/metron

commit 9410526274722f2ccdcded94a9ddadcf742defb4
Author: cstella 
Date:   2017-11-29T20:52:16Z

Updating functions and test.

commit a4c6d2b3599456c33a8dd2c228886b59af622868
Author: cstella 
Date:   2017-11-29T21:15:15Z

Whoops, fixed a subtle bug.

commit 08cc087daaa173c9ab7f3aaab9f64b34807b5c5a
Author: cstella 
Date:   2017-11-29T21:19:06Z

Missed an iterable

commit 6ef4ecfa035b68339e754c4d77fff743142cd66d
Author: jknulst 

[GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/823
  
That looks great!  Can you close and reopen the PR so we can get a green 
build?  Looks like there was a failure due to network issues.


---

[GitHub] metron issue #823: METRON-1286 Add MIN & MAX Stellar functions

[jasper-k]

Github user jasper-k commented on the issue:

https://github.com/apache/metron/pull/823
  
@ottobackwards 
Aligned annotation and README now. Added example and more clear function 
description


---

[GitHub] metron issue #852: METRON-1239 Drop extra dev environments

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/852
  
+1 (non-binding) @nickwallen . This is a much needed fix since it is now 
straight-forward to anyone  new and wanting to try Metron.


---