GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/851
METRON-1336 Patching Can Result in Bad Configuration
The following problems are addressed in this PR.
* A patch can be constructed that when applied creates an invalid
configuration. The invalid configuration is not discovered until attempting to
"pull" it back out of Zookeeper. In all cases, the result of applying a patch
needs to be validated before pushing it to Zookeeper.
* The Profiler configuration can only be pushed when pushing all
configurations at once. Attempting to push just the Profiler configuration
alone, fails silently.
* Added unit tests to validate that a 'bad' patch cannot be applied.
* Added a good number of unit tests to ensure invalid configurations cannot
be pushed under all circumstances (parser, enrichment, indexing, profiler,
"push all").
* Added a test to ensure that the Profiler configuration can be pushed
independently.
## Manual Testing
1. Launch Full Dev.
1. Environment definitions that will be used later.
```
export METRON_HOME=/usr/metron/0.4.2
export ZOOKEEPER=node1:2181
```
1. Dump all existing configurations.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP
```
1. Dump just the Profiler configuration. There should be none.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
```
1. Create a Profiler configuration on disk.
```
$ cat $METRON_HOME/config/profiler.json
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
```
1. Push the Profiler configuration.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c PROFILER
-i $METRON_HOME/config
```
1. Dump just the Profiler configuration. It should be defined now.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PROFILER
```
1. Create a patch for the Profiler configuration.
```
$ cat profile.patch
[
{
"op": "add",
"path": "/profiles/0/profile",
"value": "changed-profile-name"
}
]
```
1. Apply the patch.
```
$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c PROFILER
-pf profile.patch
```
1. Dump the Profiler configuration and ensure that the name of the Profile
was changed.
```
[root@node1 ~]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m
DUMP -c PROFILER
PROFILER Config: profiler
{
"profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
"count" : "0"
},
"update" : {
"count" : "count + 1"
},
"result" : "count"
} ]
}
```
1. Create a patch that would make the Profiler configuration invalid.
```
[root@node1 ~]# cat bad.patch
[
{
"op": "add",
"path": "/profiles/0/invalid",
"value": "22"
}
]
```
1. Apply the bad patch. The script should terminate with an error.
```
[root@node1 ~]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m
PATCH -c PROFILER -pf bad.patch
2017-11-28 17:37:17 ERROR ConfigurationManager:314 - Unable to apply
patch to Zookeeper config
java.lang.RuntimeException: Unable to load {
"profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
"count" : "0"
},
"update" : {
"count" : "count + 1"
},
"result" : "count",
"invalid" : "22"
} ]
}
at
org.apache.metron.common.configuration.ConfigurationType.lambda$static$4(ConfigurationType.java:68)
at
org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:93)
at
org.apache.metron.common.configuration.ConfigurationsUtils.applyConfigPatchToZookeeper(ConfigurationsUtils.java:621)
at
org.apache.metron.common.cli.ConfigurationManager.patch(ConfigurationManager.java:307)
at
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:285)
at
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:244)
at
org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:360)
Caused by:
org.apache.metron.jackson.databind.exc.UnrecognizedPropertyException:
Unrecognized field "invalid" (class
org.apache.metron.common.configuration.profiler.ProfileConfig), not marked as
ignorable (8 known properties: "init", "expires", "groupBy", "profile",
"foreach", "result", "onlyif", "update"])
at [Source: {
"profiles" : [ {
"profile" : "changed-profile-name",
"onlyif" : "exists(ip_src_addr)",
"foreach" : "ip_src_addr",
"init" : {
"count" : "0"
},
"update" : {
"count" : "count + 1"
},
"result" : "count",
"invalid" : "22"
} ]
...
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1336
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/851.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #851
----
----
---
