[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r165048687
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
@cestella ping
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r161240685
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
It is true that Stellar should exist, as much as possible, independent from
Metron; that was the aim of the effort to move stellar out of metron-common and
into its own top level component in the project. I'll look closer at this PR
and the (rather long, but seemingly coherent...so congrats ;) comment thread
and weigh in later.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r161239454
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
https://issues.apache.org/jira/browse/METRON-989
https://issues.apache.org/jira/browse/METRON-876
@mattf-horton @cestella
I have implemented this as to not increase the amount of tie-in between
stellar and metron, and support future non-metron configuration sources.
Thus, the sources of the configuration are discovered and not coupled. I
believe this is in the spirit if not the letter of the design discussions that
we have had.
Can you take a look?
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r161236049
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
> You mean couple it to metron? We don't want to do that anymore.
I don't see your justification. Maybe another reviewer will understand
this better. As is, I am a +0 on this.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160530273
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
Maybe that is where we are missing each other?
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160529767
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
You mean couple it to metron? We don't want to do that anymore.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160511065
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
But there is no need to discover those, right? Why have the extra
discovery logic when we don't need it?
It is also more confusing from a user's perspective. What I have on my
class path decides what is going to get validated. It is often very hard to
control what gets added to a classpath. That might lead to unexpected behavior.
If you just removed the discovery logic and instantiated the
`StellarConfigurationProvider`'s that you want to use directly...
1. the functionality would work the same way, all the time, no matter what
is on my class path.
1. the implementation would be simpler.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160042250
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
They would automatically be picked up.
I'm confused. This PR as stated is to help validation of stellar
configurations, where ever they may be. it is not for the user to hunt and
peck and pick a configuration. So that is kind of the point.
Although there have been comments about doing configuration validation with
specific objects or files passed in, that is not the thrust of this pr, even if
this pr enables it.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160030197
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
I think you're not understanding my point. More specifically... What
happens if another implementation gets added to the classpath?
> Based on your discovery logic here, just having a
FilesystemConfigurationProvider on the classpath (or any other implementation)
will cause the configuration in the file system to get validated. We don't want
that to happen. We want the user to control that behavior.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160026668
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
OK, I think I understand where you are coming from. In the end, there will
be more functions for validation. Those which take in a passed in object,
like CONFIG_GET() -> obj, EDIT -> obj, VALIDATE_CONFIG(obj). Those functions
will be user driven, ie. __**Validate this thing I gave you__**.
These functions are __**Validate everything the system knows about**. And
are born out of the introduction of possible errors through language changes in
already deployed stellar rules.
That is why we need discovery. The "job" of this function at the moment is
to find everything and make sure it is still OK.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r160002156 --- Diff: metron-platform/metron-management/README.md --- @@ -276,6 +278,13 @@ The functions are split roughly into a few sections: * aggregatorConfig - Optional config for aggregator * Returns: The String representation of the enrichment config +### Validation Functions +* `VALIDATE_STELLAR_RULE_CONFIGS` --- End diff -- There is reference to "rules" because that was the first use for Stellar. @cestella called them rules when he first implemented. But that is ancient history. We use Stellar everywhere now. Its long ago outgrown that. ---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r160001387
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
I will try to think of a different way to put it
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r15157
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
I am not following your explanation of why we need the discovery logic.
Can you try to explain it again? In the latest commits, I still see the
discovery logic in StellarSimpleValidator.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588712
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/annotations/StellarExpressionMap.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.common.utils.validation.annotations;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * {@code StellarExpressionMap} is applied to
+ * a {@code Map} which contains Stellar expressions as the values, such
+ * that calling .toString() on a value of this map yields a Stellar
expression.
+ *
+ * The key is used as the name the expression.
+ *
+ * It is possible for a {@code Map} to contain other maps or complex
objects,
+ * thus this annotation contains properties that give information on
evaluation of the {@code Map}
+ * should it be complex and contain nested maps.
+ *
+ */
+@Documented
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.FIELD,ElementType.TYPE})
+public @interface StellarExpressionMap {
+
+ /**
+ * The Name to give to the map
+ *
+ * @return String of the name
+ */
+ String name() default "default";
+
+ /**
+ * A map may be a StellarExpressionMap based on the type
+ * of another field, this is that field's name.
+ *
+ * {@code} qualify_with_field_type} is the type of that field
+ *
+ * @return Field Name or empty String
+ */
+ String qualify_with_field() default "";
--- End diff --
oops
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588686
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
The idea is that not only can we not know the details of the classes that
hold the rules, but also, that stellar may be hosted by other things than
metron, that 'know' how to provide those configurations.
The problem with this isn't the discovery per se, but in that it is not
correct given it's purpose and the implementation
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588556
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
--- End diff --
Sorry, I was going to take this out, this is from the prior shell based
integration
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588519
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarConfiguredStatementContainer.java
---
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+/**
+ * StellarConfiguredStatementProviders are used provide stellar statements
+ * and the context around those statements to the caller
+ */
+public interface StellarConfiguredStatementContainer {
--- End diff --
The issue is not using the interface, I'll address that.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588450
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
--- End diff --
I *think* you're right. Would suggest just a comment to clarify that point
and maybe help prompt us should that change
```
// indexing contains no stellar to validate
```
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r158588419 --- Diff: metron-platform/metron-management/README.md --- @@ -276,6 +278,13 @@ The functions are split roughly into a few sections: * aggregatorConfig - Optional config for aggregator * Returns: The String representation of the enrichment config +### Validation Functions +* `VALIDATE_STELLAR_RULE_CONFIGS` --- End diff -- Is you search the tree for 'rules' you will see that we call them rule or rules in various places. It 'seemed like the thing to do'. ---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588382
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
+return holders;
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children = client.getChildren().forPath(PARSER.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
+}
+for (String child : children) {
+ try {
+byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+ExpressionConfigurationHolder holder = new
ExpressionConfigurationHolder(
+String.format("%s/%s", getName(), PARSER.toString()),
parserConfig.getSensorTopic(),
+parserConfig);
+holders.add(holder);
+ } catch (Exception e) {
+errorConsumer.consume(String.format("%s/%s/%s", getName(),
PARSER.toString(), child), e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
--- End diff --
Will do
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588377
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
+return holders;
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children = client.getChildren().forPath(PARSER.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
--- End diff --
will do
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588357
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
+return holders;
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children = client.getChildren().forPath(PARSER.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
--- End diff --
Will do
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588342
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
--- End diff --
There is no stellar in indexing is there? It is not in the readme
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158588194
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
--- End diff --
That is a good idea
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158584449
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
If I understand this correctly, it seems that the
`StellarConfigurationProvider` interface allows us to extend where
configuration values get pulled in from. In your current default
implementation `ConfigurationProvider` you reach out to Zookeeper to pull in
the config.
If I wanted to validate configuration located on a file system, I would
just create a `FilesystemConfigurationProvider` implementation of this
interface.
The decision as to whether I want to validate the config in Zookeeper, on
the file system or both, needs to be user driven. A user should make that
decision based on how they call your new Stellar function.
Based on your discovery logic here, just having a
`FilesystemConfigurationProvider` on the classpath (or any other
implementation) will cause the configuration in the file system to get
validated. We don't want that to happen. We want the user to control that
behavior.
So I don't think we really need this discovery logic, which nicely
simplifies things. I think we could just alter the `StellarValidater`
interface to make this relationship simpler and more straight forward. The
`StellarConfigurationProvider` just gets passed in.
```
StellarValidator {
...
Iterable validate(StellarConfigurationProvider
provider);
...
}
```
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158584054
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarConfiguredStatementContainer.java
---
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+/**
+ * StellarConfiguredStatementProviders are used provide stellar statements
+ * and the context around those statements to the caller
+ */
+public interface StellarConfiguredStatementContainer {
--- End diff --
`ExpressionConfigurationHolder` is an implementation of this interface.
That being said, I don't understand the point of this interface.
In all the code that I see, you use the implementation class
`ExpressionConfigurationHolder` rather than this interface. For example, in
`StellarZookeeperBasedValidator` and other places.
We should either use the interface or get rid of it.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158583046
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
--- End diff --
What about indexing?
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r158584184 --- Diff: metron-platform/metron-management/README.md --- @@ -276,6 +278,13 @@ The functions are split roughly into a few sections: * aggregatorConfig - Optional config for aggregator * Returns: The String representation of the enrichment config +### Validation Functions +* `VALIDATE_STELLAR_RULE_CONFIGS` --- End diff -- This name confuses me, why Rules? `VALIDATE_STELLAR` seems simple and to the point to me. ---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158583012
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
+return holders;
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children = client.getChildren().forPath(PARSER.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
--- End diff --
Need to log and comment here. We are silently eating the exception. Seems
especially problematic because of the overly generic `Exception` declaration
that the Curator library gives us.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158582738
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/annotations/StellarExpressionMap.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.common.utils.validation.annotations;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * {@code StellarExpressionMap} is applied to
+ * a {@code Map} which contains Stellar expressions as the values, such
+ * that calling .toString() on a value of this map yields a Stellar
expression.
+ *
+ * The key is used as the name the expression.
+ *
+ * It is possible for a {@code Map} to contain other maps or complex
objects,
+ * thus this annotation contains properties that give information on
evaluation of the {@code Map}
+ * should it be complex and contain nested maps.
+ *
+ */
+@Documented
+@Retention(RetentionPolicy.RUNTIME)
+@Target({ElementType.FIELD,ElementType.TYPE})
+public @interface StellarExpressionMap {
+
+ /**
+ * The Name to give to the map
+ *
+ * @return String of the name
+ */
+ String name() default "default";
+
+ /**
+ * A map may be a StellarExpressionMap based on the type
+ * of another field, this is that field's name.
+ *
+ * {@code} qualify_with_field_type} is the type of that field
+ *
+ * @return Field Name or empty String
+ */
+ String qualify_with_field() default "";
--- End diff --
These variables_go_against_the_style convention, no? qualify_with_field,
qualify_with_field_type, inner_map_keys
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158583416
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c :
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --
What is the following code block doing? Why do we need to discover all of
the StellarConfigurationProvider classes?
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158583014
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
+
+ /**
+ * Default constructor.
+ */
+ public ConfigurationProvider() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public List
provideConfigurations(CuratorFramework client,
+ ErrorConsumer errorConsumer) {
+List holders = new LinkedList<>();
+visitParserConfigs(client, holders, errorConsumer);
+visitEnrichmentConfigs(client, holders, errorConsumer);
+visitProfilerConfigs(client, holders, errorConsumer);
+return holders;
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children = client.getChildren().forPath(PARSER.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
+}
+for (String child : children) {
+ try {
+byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+ExpressionConfigurationHolder holder = new
ExpressionConfigurationHolder(
+String.format("%s/%s", getName(), PARSER.toString()),
parserConfig.getSensorTopic(),
+parserConfig);
+holders.add(holder);
+ } catch (Exception e) {
+errorConsumer.consume(String.format("%s/%s/%s", getName(),
PARSER.toString(), child), e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
+ List holders, ErrorConsumer
errorConsumer) {
+List children = null;
+
+try {
+ children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+} catch (Exception nne) {
+ return;
--- End diff --
Need to log and comment here. We are silently eating the exception.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158583497
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationProvider.java
---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+import static
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.curator.framework.CuratorFramework;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.configuration.profiler.ProfileConfig;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.JSONUtils;
+import
org.apache.metron.stellar.common.utils.validation.ExpressionConfigurationHolder;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfigurationProvider;
+import
org.apache.metron.stellar.common.utils.validation.StellarConfiguredStatementContainer.ErrorConsumer;
+import org.apache.zookeeper.KeeperException.NoNodeException;
+
+/**
+ * {@code ConfigurationProvider} is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class ConfigurationProvider implements StellarConfigurationProvider
{
--- End diff --
This implementation of a StellarConfigurationProvider is one that talks to
Zookeeper, right? It retrieves Stellar configuration from Zookeeper.
Should we call it `ZookeeperConfigurationProvider` or something more
descriptive?
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r158584492
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String FAILED_COMPILE = "Failed to compile";
+ private CuratorFramework client;
+
+ public StellarZookeeperBasedValidator(CuratorFramework client) throws
NullArgumentException {
+if (client == null) {
+ throw new NullArgumentException("client");
+}
+this.client = client;
+ }
+
+
+ @Override
+ public Iterable validate(Optional writer) {
--- End diff --
Why do we need a 'writer'? It doesn't make sense to me why we need it nor
do we use it. I think it could be removed from the method signature completely
in `StellarValidator`.
---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
GitHub user ottobackwards reopened a pull request:
https://github.com/apache/metron/pull/856
METRON-1339 Stellar Shell functionality to verify stored stellar statements
This will allow users to check their deployed statements, say after
upgrade, when they are at rest ( and would fail on use ).
In other words, they were valid when stored, but are not now because of
stellar changes, such as new keywords.
The interface `StellarConfiguredStatementReporter`, which is
`@IndexSubclasses` ( ClassIndex) marked, allows the shell to discover reporters
that can provide statements for validation. This discovery allows de-coupling
of stellar and 'hosts' that know about the location of the stored statements,
and the configuration structure details.
> We do mention the configurations in the shell output at this time.
`metron-common` implements this interface, and can run through visiting all
the configurations.
A new magic keyword was added ` %validate_configured_expressions`
When executed, the shell
- discovers the reporters through class index
- visits the reports, with callbacks for visits or errors
- per visit ( which is called for a specific stellar statement ) the
statement is compiled and errors reported
- if the entire config fails ( threat triage stellar errors fail on
deserialize so we don't get to do ANY enrichment visits in that case ) the
error callback handles that
I'm getting this out there, still a couple of things todo:
[x] ~~full dev run. I have been testing with stellar external to full dev
iteratively~~
[x] ~~readme~~
[x] ~~steps to test~~
[x] ~~unit test~~
[x] ~~ThreatTriage Rule Reason~~
## Testing
- deploy full dev
- edit the squid parser transformation(s) such that the stellar would not
compile, such as adding a dangling `=` in zookeeper
```json
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "squid",
"parserConfig": {
"grokPath": "/patterns/squid",
"patternLabel": "SQUID_DELIMITED",
"timestampField": "timestamp"
},
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "full_hostname", "domain_without_subdomains" ]
,"config" : {
"full_hostname" : "URL_TO_HOST(url) ="
,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
}
}
]
}
```
- edit the snort threat triage rules in it's enrichment config in zookeeper
( here with an extra `)` )
```json
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel" : {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : [
{
"rule" : "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24')) )",
"score" : 10
}
],
"aggregator" : "MAX"
}
}
}
```
## Working with zookeeper
I am not a zk cli maestro, so I took the easy way out and used
[ZK-WEB](https://github.com/qiuxiafei/zk-web).
Following the readme instructions it was very simple to clone, edit the
config for full dev, and run from source. If you log in with the creds in the
config you can edit the nodes.
## Results
When you run the magic command, it will report the failed stellar
statements, and the failed enrichment config:
```bash
[Stellar]>>> %validate_configured_expressions
Discovered 1 reporters
Visiting all configurations. ThreatTriage rules are checked when loading
the configuration, thus an invalid ThreatTriage rule will fail the entire
Enrichement Configuration.
Apache Metron
Visiting Apache Metron
==
validating Apache Metron->PARSER->squid->full_hostname
[!] Error Visiting Apache Metron->PARSER->squid->full_hostname
Syntax error @ 1:17 token recognition error at: '='
--
[!] : URL_TO_HOST(url) =
==
==
validating Apache Metron->PARSER->squid->domain_without_subdomains
==
[!] Configuration Apache Metron->ENRICHMENT->snort is not valid, please
review
Done validation
[Stellar]>>>
```
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/856 ---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154375558
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154375781
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154245631
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154241894
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154200025
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154189837
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154185591
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154171805
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154171511
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154155977
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154155446
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154143988
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154135697
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154133974
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154133309
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154124236
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154123254
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154118407
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154116726
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154114714
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r15404
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154110765
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154110355
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154109552
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorE
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154102327
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r154105765
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/StellarStatementReporter.java
---
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.common.configuration;
+
+import static
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+import static
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import
org.apache.metron.common.configuration.enrichment.threatintel.RiskLevelRule;
+import
org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.StellarTransformation;
+import org.apache.metron.common.utils.StringUtils;
+import org.apache.metron.stellar.common.StellarConfiguredStatementReporter;
+
+/**
+ * StellarStatementReporter is used to report all of the configured /
deployed Stellar statements in
+ * the system.
+ */
+public class StellarStatementReporter implements
StellarConfiguredStatementReporter {
+
+ public enum Type {
+ENRICHMENT, THREAT_INTEL;
+ }
+
+ public StellarStatementReporter() {
+ }
+
+ @Override
+ public String getName() {
+return "Apache Metron";
+ }
+
+ @Override
+ public void vist(CuratorFramework client, StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+visitParserConfigs(client, visitor, errorConsumer);
+visitEnrichmentConfigs(client, visitor, errorConsumer);
+ }
+
+ private void visitParserConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(PARSER.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(PARSER.getZookeeperRoot() +
"/" + child);
+ try {
+SensorParserConfig parserConfig =
SensorParserConfig.fromBytes(data);
+List transformations =
parserConfig.getFieldTransformations();
+transformations.forEach((f) -> {
+ if
(StellarTransformation.class.isAssignableFrom(f.getFieldTransformation().getClass()))
{
+FieldTransformation transformation =
f.getFieldTransformation();
+f.getConfig().forEach((k, v) -> {
+ List names = Arrays
+ .asList(getName(), PARSER.toString(),
parserConfig.getSensorTopic(), k);
+ visitor.visit(names, v.toString());
+});
+ }
+});
+ } catch (Exception e) {
+List names = Arrays.asList(getName(), PARSER.toString(),
child);
+errorConsumer.consume(names, e);
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void visitEnrichmentConfigs(CuratorFramework client,
StatementReportVisitor visitor,
+ ConfigReportErrorConsumer errorConsumer) throws Exception {
+List children =
client.getChildren().forPath(ENRICHMENT.getZookeeperRoot());
+for (String child : children) {
+ byte[] data = client.getData().forPath(ENRICHMENT.getZookeeperRoot()
+ "/" + child);
+ try {
+final SensorEnrichmentConfig sensorEnrichmentConfig =
SensorEnrichmentConfig
+.fromBytes(data);
+
+EnrichmentConfig enrichmentConfig = null;
+enrichmentConfig = sensorEnri
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/856 METRON-1339 Stellar Shell functionality to verify stored stellar statements This will allow users to check their deployed statements, say after upgrade, when they are at rest ( and would fail on use ). In other words, they were valid when stored, but are not now because of stellar changes, such as new keywords. The interface `StellarConfiguredStatementReporter`, which is `@IndexSubclasses` ( ClassIndex) marked, allows the shell to discover reporters that can provide statements for validation. This discovery allows de-coupling of stellar and 'hosts' that know about the location of the stored statements, and the configuration structure details. > We do mention the configurations in the shell output at this time. `metron-common` implements this interface, and can run through visiting all the configurations. A new magic keyword was added ` %validate_configured_expressions` When executed, the shell - discovers the reporters through class index - visits the reports, with callbacks for visits or errors - per visit ( which is called for a specific stellar statement ) the statement is compiled and errors reported - if the entire config fails ( threat triage stellar errors fail on deserialize so we don't get to do ANY enrichment visits in that case ) the error callback handles that I'm getting this out there, still a couple of things todo: 1. full dev run. I have been testing with stellar external to full dev iteratively 2. readme 3. steps to test 4. unit test ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_verify_deployed_shell Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/856.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #856 commit a5087f3a170eeda6ee778397c919d9eddd5597e2 Author: Otto Fowler Date: 2017-11-30T14:15:40Z Stellar shell functionality to verify stellar statements. This will allow users to check their deployed statements, say after upgrade, when they are at rest ( and would fail on use ). In other words, they were valid when stored, but are not now because of stellar changes, such as new keywords. The interface StellarConfiguredStatementReporter, which is @IndexSubclasses marked, allows the shell to discover reporters that can provide statements for validation. This discovery allows de-coupling of stellar and 'hosts' that know about the location of the stored statements, and the configuration structure details. We do mention the configurations in the shell output at this time. metron-common implements this interface, and can run through visiting all the configurations. ---
