[jira] [Commented] (METRON-1335) Install metron-maas-service RPM as a part of the full-dev deployment
[ https://issues.apache.org/jira/browse/METRON-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278122#comment-16278122 ] ASF GitHub Bot commented on METRON-1335: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/850 bump. Any feedback on this pull request? > Install metron-maas-service RPM as a part of the full-dev deployment > > > Key: METRON-1335 > URL: https://issues.apache.org/jira/browse/METRON-1335 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian > > Currently, Metron MaaS is not installed by default on the full dev > deployments. We copy the metron-maas-service RPM onto the /localrepo folder, > but the RPM is not installed automatically. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1339) Stellar Shell: Should have a way to validate deployed functions
[ https://issues.apache.org/jira/browse/METRON-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277867#comment-16277867 ] ASF GitHub Bot commented on METRON-1339: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/856 @nickwallen @mattf-horton I think we can use the annotation approach to resolve [METRON-989](https://issues.apache.org/jira/browse/METRON-989) as well. Thoughts? > Stellar Shell: Should have a way to validate deployed functions > --- > > Key: METRON-1339 > URL: https://issues.apache.org/jira/browse/METRON-1339 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > It is possible that the Stellar Language is changed during a release, meaning > stellar expressions that are in zookeeper are now invalid, and will not run. > Currently users cannot verify this without just letting them fail and hunting > the errors down. > The Stellar Shell should expose a '%' function that will verify that all > stellar functions compile and are syntactically correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277865#comment-16277865
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154824244
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
METRON-1342
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize
[jira] [Created] (METRON-1342) Methods in Stellar Configuration that take paths as strings should use path
Otto Fowler created METRON-1342: --- Summary: Methods in Stellar Configuration that take paths as strings should use path Key: METRON-1342 URL: https://issues.apache.org/jira/browse/METRON-1342 Project: Metron Issue Type: Improvement Reporter: Otto Fowler Assignee: Otto Fowler It is more maintainable and a better practice to take a Path if you mean a Path, and not just take Strings. In things with many overload methods, it makes it more obvious. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277863#comment-16277863
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154824002
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
I am familiar with that ;)
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializ
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277864#comment-16277864
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154824031
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
I'll open a jira
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deseria
[jira] [Commented] (METRON-989) design and implement configuration interface for stand-alone Stellar
[ https://issues.apache.org/jira/browse/METRON-989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277859#comment-16277859 ] Otto Fowler commented on METRON-989: METRON-1339 could help with this I think. Because we can visit all stellar configuration items and fields, we should be able to 'find' any stellar embedded in any configuration. If we need more context, we can add that to the Annotations > design and implement configuration interface for stand-alone Stellar > > > Key: METRON-989 > URL: https://issues.apache.org/jira/browse/METRON-989 > Project: Metron > Issue Type: Sub-task >Reporter: Matt Foley > > In the first effort to separate Stellar from Metron, I left a truncated > version of Metron's configuration model in Stellar. This isn't very > satisfactory. The goals should be: > # User must be able to configure Stellar itself, regardless of the platform > it is plugged into. > # Stellar's configuration model should not be grossly Metron-specific; > ideally it should be storage-agnostic and possibly schema-driven. > # We would like to still be able to use Stellar to manipulate the > Metron-specific config when plugged into Metron. Reading and setting are > required, since Stellar has become a major tool of convenience for managing > Metron's config. > # We would like to be able to use Stellar to manipulate the > application-specific config of other platforms when Stellar is plugged into > them, if the other platform allows. Reading is more important than setting. > The last two items argue for a client interface layer where the platform can > provision Stellar's config model as the platform prefers. > Investigate whether Apache commons configuration would be appropriate, > perhaps with add-ins for JSON sources and Curator event support. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-879) design and implement split for Enrichment functionality of Stellar
[ https://issues.apache.org/jira/browse/METRON-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277857#comment-16277857 ] Otto Fowler commented on METRON-879: It is not entirely clear to me what you steps or goals where for this, at least not looking at the branch in the state it is in. Would it be possible to summarize here? > design and implement split for Enrichment functionality of Stellar > -- > > Key: METRON-879 > URL: https://issues.apache.org/jira/browse/METRON-879 > Project: Metron > Issue Type: Sub-task >Reporter: Matt Foley > > Step 3 from design document. This includes moving additional stand-alone > functionality into metron-stellar, such as the Field Validation operators, > etc., not just Enrichment stuff per se. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277840#comment-16277840
]
ASF GitHub Bot commented on METRON-1336:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154821946
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
Yeah I get it. ;). It wasn't my preference to include all of that here, but
I bend to make people happy.
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
>
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277730#comment-16277730
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154804149
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -572,15 +590,22 @@ public static void
applyConfigPatchToZookeeper(ConfigurationType configurationTy
* @param patchData a JSON patch in the format specified by RFC 6902
* @param client access to zookeeeper
*/
- public static void applyConfigPatchToZookeeper(ConfigurationType
configurationType,
- Optional configName,
- byte[] patchData, CuratorFramework client) throws Exception {
+ public static void applyConfigPatchToZookeeper(
+ ConfigurationType configurationType,
+ Optional configName,
+ byte[] patchData, CuratorFramework client) throws Exception {
+
byte[] configData = readConfigBytesFromZookeeper(configurationType,
configName, client);
JsonNode source = JSONUtils.INSTANCE.readTree(configData);
JsonNode patch = JSONUtils.INSTANCE.readTree(patchData);
JsonNode patchedConfig = JSONUtils.INSTANCE.applyPatch(patch, source);
-writeConfigToZookeeper(configurationType, configName,
-JSONUtils.INSTANCE.toJSONPretty(patchedConfig), client);
+byte[] prettyPatchedConfig =
JSONUtils.INSTANCE.toJSONPretty(patchedConfig);
+
+// ensure the patch produces a valid result; otherwise exception
thrown during deserialization
--- End diff --
Never mind Nick, I didn't read this correctly, and I have the
"serialization mixed with logical validation" stuff upfront in my head because
of my work over in https://github.com/apache/metron/pull/856.
Sorry
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sket
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277726#comment-16277726
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154803527
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -572,15 +590,22 @@ public static void
applyConfigPatchToZookeeper(ConfigurationType configurationTy
* @param patchData a JSON patch in the format specified by RFC 6902
* @param client access to zookeeeper
*/
- public static void applyConfigPatchToZookeeper(ConfigurationType
configurationType,
- Optional configName,
- byte[] patchData, CuratorFramework client) throws Exception {
+ public static void applyConfigPatchToZookeeper(
+ ConfigurationType configurationType,
+ Optional configName,
+ byte[] patchData, CuratorFramework client) throws Exception {
+
byte[] configData = readConfigBytesFromZookeeper(configurationType,
configName, client);
JsonNode source = JSONUtils.INSTANCE.readTree(configData);
JsonNode patch = JSONUtils.INSTANCE.readTree(patchData);
JsonNode patchedConfig = JSONUtils.INSTANCE.applyPatch(patch, source);
-writeConfigToZookeeper(configurationType, configName,
-JSONUtils.INSTANCE.toJSONPretty(patchedConfig), client);
+byte[] prettyPatchedConfig =
JSONUtils.INSTANCE.toJSONPretty(patchedConfig);
+
+// ensure the patch produces a valid result; otherwise exception
thrown during deserialization
--- End diff --
```java
//Some configurations have side effects to deserialization, and may throw
exceptions from
//validation that may happen during the deserialization. This does not
meant that the //deserialization has failed due to invalid serialization.
```
?
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Sou
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277722#comment-16277722
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154802693
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
No. I guess I miss read the ambition of this pr, given the meta-pr and all
;)
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at
> org.apache.m
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277692#comment-16277692
]
ASF GitHub Bot commented on METRON-1336:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154799152
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -572,15 +590,22 @@ public static void
applyConfigPatchToZookeeper(ConfigurationType configurationTy
* @param patchData a JSON patch in the format specified by RFC 6902
* @param client access to zookeeeper
*/
- public static void applyConfigPatchToZookeeper(ConfigurationType
configurationType,
- Optional configName,
- byte[] patchData, CuratorFramework client) throws Exception {
+ public static void applyConfigPatchToZookeeper(
+ ConfigurationType configurationType,
+ Optional configName,
+ byte[] patchData, CuratorFramework client) throws Exception {
+
byte[] configData = readConfigBytesFromZookeeper(configurationType,
configName, client);
JsonNode source = JSONUtils.INSTANCE.readTree(configData);
JsonNode patch = JSONUtils.INSTANCE.readTree(patchData);
JsonNode patchedConfig = JSONUtils.INSTANCE.applyPatch(patch, source);
-writeConfigToZookeeper(configurationType, configName,
-JSONUtils.INSTANCE.toJSONPretty(patchedConfig), client);
+byte[] prettyPatchedConfig =
JSONUtils.INSTANCE.toJSONPretty(patchedConfig);
+
+// ensure the patch produces a valid result; otherwise exception
thrown during deserialization
--- End diff --
I'm not sure exactly what you're looking for. Can you suggest some
specific text?
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OU
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277683#comment-16277683
]
ASF GitHub Bot commented on METRON-1336:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154798328
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
Sure, but this method signature already existed. Do I need to fix that on
this PR?
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at
> org.apache
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277579#comment-16277579 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Yeah, I'm trying to think of the right way to do it, without having to have it be "officially maintained". more like an informal /testing_stuff , with some descriptions, when it was valid/written against etc. Something like that. Then people could edit them and adapt them etc. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277562#comment-16277562 ] ASF GitHub Bot commented on METRON-939: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 I don't think that's a bad idea. I definitely like having test scripts associated with the PR's because we can explicitly see what was done and/or recommended at the time the PR was being ushered into master. But we could also start to consolidate these into manual testing scripts that we reference ongoing. Then you could say something like: ``` Testing Plan - es basic search test - meta alerts search test - kibana dashboard smoke test ``` That could be links, or we could copy-paste the current manual test plan from the source tree. I think the main thing I would want is to have the PR's test plan (here in the comments) be maintained statically for posterity based on the code at that time. For example, since we're removing the data pruner, the new test plan would not have a data pruner test, but I'd want to make sure any earlier PRs that did have a test plan for it can still be viewed statically. Basically, you'd either include the latest commit as part of the link or copy-paste the test plan. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277495#comment-16277495 ] ASF GitHub Bot commented on METRON-939: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 So, this has me thinking It is a shame to have these test materials ( and others like @JonZeolla creates ) embedded in these pr's. Maybe we should have someplace to put them in the code tree? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277487#comment-16277487
]
ASF GitHub Bot commented on METRON-939:
---
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/840
Ok, here is the remaining bit of the test plan for verifying ES. Thanks
@cestella for pulling together some of the public test scripts around this.
# Test Script
Testing Kibana dashboards - see
[here](https://github.com/apache/metron/pull/840#issuecomment-348085037)
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
# Preliminaries
Set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.4.2`
# Deploy the dummy parser
* Edit `$METRON_HOME/config/zookeeper/parsers/dummy.json`:
```
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"dummy"
}
```
* Create the dummy kafka topic:
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic dummy --partitions 1 --replication-factor 1`
* Persist config changes: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i
$METRON_HOME/config/zookeeper -z node1:2181`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z
node1:2181 -s dummy`
# Send dummy data through
* Edit `~/msg.json` with the following content:
```
{ "guid" : "guid0", "sensor.type" : "dummy", "timestamp" : 100 }
```
* Send `msg.json` through to kafka via `cat ~/msg.json |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic dummy`
* Validate data has been written to the index:
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty'
```
## Test Case: Update via patch
* Patch the message in ES and create a new field 'project' by executing
the following:
```
curl -u user:password -X PATCH --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"guid" : "guid0",
"sensorType" : "dummy",
"patch" : [
{
"op": "add"
, "path": "/project"
, "value": "metron"
}
]
}' 'http://node1:8082/api/v1/update/patch'
```
* Validate that the message has a field 'project':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "project" ]
}
'
```
## Test Case: Update via replace
* Replace the message in ES and create a couple of modifications:
* new field `new_field` == "brand new"
* modified `timestamp` == 7
Execute the following:
```
curl -u user:password -X POST --header 'Content-Type: application/json'
--header 'Accept: */*' -d '{
"guid" : "guid0",
"sensorType" : "dummy",
"replacement" : {
"source:type": "dummy",
"guid" : "guid0",
"new_field" : "brand new",
"timestamp" : 7
}
}' 'http://node1:8082/api/v1/update/replace'
```
* Validate that the message has a field 'new_field':
```
curl -XPOST 'http://localhost:9200/dummy*/_search?pretty' -d '
{
"_source" : [ "new_field", "timestamp" ]
}
'
```
## Meta Alerts Test
### Set Up Base Data
We're going to set up a bit of base data.
Retrieve the current list of indices so we know where to put our data
```
curl 'node1:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted
store.size pri.store.size
green open snort_index_2017.09.06.14 1 01300
180.9kb180.9kb
green open bro_index_2017.09.06.14 1 01600
564.3kb564.3kb
green open .kibana 1 0 520
71.2kb 71.2kb
green open metaalert_index 1 0 60
62.3kb 62.3kb
```
In this case, we care about `snort_index_2017.09.06.14` and
`metaalert_index`. To make our lives easier, we'll add a couple of stripped
down messages to the our snort index (Make sure to sub in the correct index
name):
```
curl -XPUT
'node1:9200/snort_index_2017.09.06.14/snort_doc/snort_test_1?pretty' -H
'Content-Type: application/json' -d'
{
"msg": "snort test alert",
"ip_dst_port": "8080",
"ethsrc": "0A:00:27:00:00:00",
"protocol": "TCP",
"source:type": "snort",
"ip_dst_addr": "192.168.66.121",
"ip_src_addr": "192.168.66.1",
"threat:triage:rules:0:score": 10,
"timestamp": 1504708744000,
"threat:triage:rules:0:reason": null,
"threat:triage:score":
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277418#comment-16277418
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154761355
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,42 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
--- End diff --
When reading from file, the API should explicitly require file Paths as
opposed to just strings.
It makes the api methods easier to understand when there are a bunch of
similar methods.
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackso
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277417#comment-16277417
]
ASF GitHub Bot commented on METRON-1336:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154763112
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -572,15 +590,22 @@ public static void
applyConfigPatchToZookeeper(ConfigurationType configurationTy
* @param patchData a JSON patch in the format specified by RFC 6902
* @param client access to zookeeeper
*/
- public static void applyConfigPatchToZookeeper(ConfigurationType
configurationType,
- Optional configName,
- byte[] patchData, CuratorFramework client) throws Exception {
+ public static void applyConfigPatchToZookeeper(
+ ConfigurationType configurationType,
+ Optional configName,
+ byte[] patchData, CuratorFramework client) throws Exception {
+
byte[] configData = readConfigBytesFromZookeeper(configurationType,
configName, client);
JsonNode source = JSONUtils.INSTANCE.readTree(configData);
JsonNode patch = JSONUtils.INSTANCE.readTree(patchData);
JsonNode patchedConfig = JSONUtils.INSTANCE.applyPatch(patch, source);
-writeConfigToZookeeper(configurationType, configName,
-JSONUtils.INSTANCE.toJSONPretty(patchedConfig), client);
+byte[] prettyPatchedConfig =
JSONUtils.INSTANCE.toJSONPretty(patchedConfig);
+
+// ensure the patch produces a valid result; otherwise exception
thrown during deserialization
--- End diff --
Can we reword this? It needs to be clear that some things cannot be
deserialized if they have an invalid configuration.
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
>
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277319#comment-16277319
]
ASF GitHub Bot commented on METRON-1336:
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/851
@mmiklavc @cestella The kick is complete. All PRs merged. Ready for review.
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT token
> at [Source: {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }; line: 2, column: 16] (through reference chain:
> org.apache.metron.common.configuration.profiler.ProfilerConfig["profiles"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:255)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:971)
> at
> org.apache.metron.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:967)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.handleNonArray(CollectionDeserializer.java:327)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:249)
> at
> org.apache.metron.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> at
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
>
[jira] [Commented] (METRON-1338) Rat Check Should Ignore Vagrant Retry Files
[ https://issues.apache.org/jira/browse/METRON-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277316#comment-16277316 ] ASF GitHub Bot commented on METRON-1338: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/855 > Rat Check Should Ignore Vagrant Retry Files > --- > > Key: METRON-1338 > URL: https://issues.apache.org/jira/browse/METRON-1338 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Trivial > Fix For: 0.4.1 > > > When Vagrant fails, it generates a *.retry file. These files are ignored by > Git, but will unnecessarily fail a build because of the Rat check. This has > annoyed me to no end. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-1341) Projection FieldTransformation
[
https://issues.apache.org/jira/browse/METRON-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Elliston Ball updated METRON-1341:
Summary: Projection FieldTransformation (was: Projection
FieldTansformation)
> Projection FieldTransformation
> --
>
> Key: METRON-1341
> URL: https://issues.apache.org/jira/browse/METRON-1341
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.4.2
>Reporter: Simon Elliston Ball
>Assignee: Simon Elliston Ball
> Labels: newbie
>
> It would be useful to have a projection transformation for Parsers which
> could use configured to limit the fields output from the parser.
> The configuration would look like this:
> {code:java}
> {
> "fieldTransformations": [
>{
> "transformation": "STELLAR",
> "config": [
>"ipSrc = TRIM(raw_ip_src)"
>"ip_src_addr := ipSrc"
> ]
>},
> {
> "transformation": "SELECT",
> "output" : [ "ip_src_addr", "ip_dst_addr", "message"]
>}
> ]
> }
> {code}
> This would lead to only the fields in the output definition of the SELECT
> transformation being put into the outbound message.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1341) Projection FieldTansformation
Simon Elliston Ball created METRON-1341:
---
Summary: Projection FieldTansformation
Key: METRON-1341
URL: https://issues.apache.org/jira/browse/METRON-1341
Project: Metron
Issue Type: Improvement
Affects Versions: 0.4.2
Reporter: Simon Elliston Ball
Assignee: Simon Elliston Ball
It would be useful to have a projection transformation for Parsers which could
use configured to limit the fields output from the parser.
The configuration would look like this:
{code:java}
{
"fieldTransformations": [
{
"transformation": "STELLAR",
"config": [
"ipSrc = TRIM(raw_ip_src)"
"ip_src_addr := ipSrc"
]
},
{
"transformation": "SELECT",
"output" : [ "ip_src_addr", "ip_dst_addr", "message"]
}
]
}
{code}
This would lead to only the fields in the output definition of the SELECT
transformation being put into the outbound message.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1286) Add MAX & MIN Stellar functions
[ https://issues.apache.org/jira/browse/METRON-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277027#comment-16277027 ] ASF GitHub Bot commented on METRON-1286: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/823 > Add MAX & MIN Stellar functions > > > Key: METRON-1286 > URL: https://issues.apache.org/jira/browse/METRON-1286 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Jasper Knulst >Priority: Minor > Labels: features > Fix For: 0.4.2 > > > Currently Stellar lacks straightforward MAX & MIN functions that take just a > list of values as input. > The functions STATS_MAX and STATS_MIN only take the internal Stellar > statistics object as input. > Having MAX and MIN will be easier and understandable to most -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1286) Add MAX & MIN Stellar functions
[ https://issues.apache.org/jira/browse/METRON-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276977#comment-16276977 ] ASF GitHub Bot commented on METRON-1286: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/823 +1, I'm set > Add MAX & MIN Stellar functions > > > Key: METRON-1286 > URL: https://issues.apache.org/jira/browse/METRON-1286 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.1 >Reporter: Jasper Knulst >Priority: Minor > Labels: features > Fix For: 0.4.2 > > > Currently Stellar lacks straightforward MAX & MIN functions that take just a > list of values as input. > The functions STATS_MAX and STATS_MIN only take the internal Stellar > statistics object as input. > Having MAX and MIN will be easier and understandable to most -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1336) Patching Can Result in Bad Configuration
[
https://issues.apache.org/jira/browse/METRON-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276853#comment-16276853
]
ASF GitHub Bot commented on METRON-1336:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/851#discussion_r154663544
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java
---
@@ -343,25 +343,57 @@ public static void uploadConfigsToZookeeper(String
rootFilePath, CuratorFramewor
* @param type config type to upload configs for
* @param configName specific config under the specified config type
*/
- public static void uploadConfigsToZookeeper(String rootFilePath,
CuratorFramework client,
- ConfigurationType type, Optional configName) throws
Exception {
+ public static void uploadConfigsToZookeeper(
+ String rootFilePath,
+ CuratorFramework client,
+ ConfigurationType type,
+ Optional configName) throws Exception {
+
switch (type) {
+
case GLOBAL:
final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath);
if (globalConfig.length > 0) {
setupStellarStatically(client, Optional.of(new
String(globalConfig)));
writeGlobalConfigToZookeeper(globalConfig, client);
}
break;
- case PARSER: // intentional pass-through
- case ENRICHMENT: // intentional pass-through
- case INDEXING:
-Map sensorIndexingConfigs =
readSensorConfigsFromFile(rootFilePath, type,
-configName);
-for (String sensorType : sensorIndexingConfigs.keySet()) {
- writeConfigToZookeeper(type, configName,
sensorIndexingConfigs.get(sensorType), client);
+
+ case PARSER: {
+Map configs =
readSensorConfigsFromFile(rootFilePath, PARSER, configName);
--- End diff --
The meta-PR is merged
> Patching Can Result in Bad Configuration
>
>
> Key: METRON-1336
> URL: https://issues.apache.org/jira/browse/METRON-1336
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: 0.4.1
>
>
> When applying a patch with `zk_load_configs` the resulting configuration can
> be invalid. The resulting configuration should be validated so that a patch
> can never result in an invalid configuration.
> For example, applying the following patch with `zk_load_config` to the
> Profiler yields a broken Profiler configuration.
> {code}
> [
>{
> "path":"profiles",
> "value":{
> "profile":"sketchy_mad",
> "onlyif":"true",
> "update":{
> "s":"OUTLIER_MAD_ADD(s, value)"
> },
> "init":{
> "s":"OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach":"'global'",
> "result":"s"
> },
> "op":"add"
>}
> ]
> {code}
> The broken configuration is only discovered after dumping the configuration.
> {code}
> $ bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
> "profiles" : {
> "profile" : "sketchy_mad",
> "onlyif" : "true",
> "update" : {
> "s" : "OUTLIER_MAD_ADD(s, value)"
> },
> "init" : {
> "s" : "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad','global',
> PROFILE_FIXED(5, 'MINUTES')))"
> },
> "foreach" : "'global'",
> "result" : "s"
> }
> }
> at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
> at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$6(ConfigurationsUtils.java:621)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:575)
> at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:619)
> at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:189)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:268)
> at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:243)
> at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:355)
> Caused by: org.apache.metron.jackson.databind.JsonMappingException: Can not
> deserialize instance of java.util.ArrayList out of START_OBJECT tok
[jira] [Commented] (METRON-1340) Improve e2e tests for metron alerts
[ https://issues.apache.org/jira/browse/METRON-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276852#comment-16276852 ] ASF GitHub Bot commented on METRON-1340: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/857 I haven't looked at the code yet, but I was able to pull this down and run the e2e tests repeatedly without failures, so this is definitely great stuff. > Improve e2e tests for metron alerts > --- > > Key: METRON-1340 > URL: https://issues.apache.org/jira/browse/METRON-1340 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra > > Need to improve e2e tests in the following areas: > - Tests should not be flaky > - Remove the sleep ( This should implicitly make the tests run faster) > - Truncate HBase table 'metron_update' before starting the tests > - Improve the tests descriptions > - Run the tests headless if possible > - Check the node version and browser version before launching the tests > The expected behavior is that there are no intermittent failures. Acceptance > criteria: 5 consecutive runs without failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1339) Stellar Shell: Should have a way to validate deployed functions
[ https://issues.apache.org/jira/browse/METRON-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276804#comment-16276804 ] ASF GitHub Bot commented on METRON-1339: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/856 @cestella @simonellistonball With the new implementation, doing the blob or file check should be a piece of cake would you prefer it as part of this or as a new issue? > Stellar Shell: Should have a way to validate deployed functions > --- > > Key: METRON-1339 > URL: https://issues.apache.org/jira/browse/METRON-1339 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > It is possible that the Stellar Language is changed during a release, meaning > stellar expressions that are in zookeeper are now invalid, and will not run. > Currently users cannot verify this without just letting them fail and hunting > the errors down. > The Stellar Shell should expose a '%' function that will verify that all > stellar functions compile and are syntactically correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
