[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174693#comment-16174693 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/760 Thanks for making the adjustments, @mmiklavc. I'm +1 on this. It's a great step forward and gives a good foundation for future improvements we want to make. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173363#comment-16173363 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 @justinleet One of my earlier commits erroneously triggered the Github "outdated comment" feature on some of your requests for javadocs. I've gone back through and added them. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172107#comment-16172107 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 A @JonZeolla fixing it now. Sorry about that - I missed one of the "patch_path -> patch_file" arg changes in the mpack. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172104#comment-16172104 ] ASF GitHub Bot commented on METRON-1188: Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/760 This is failing to spin up for me. As a part of Metron Enrichment Start, during: ``` 2017-09-19 18:05:54,844 - Execute['/usr/metron/0.4.1/bin/zk_load_configs.sh --zk_quorum node1:2181 --mode PATCH --config_type GLOBAL --patch_path /tmp/metron-global-config-patch.json'] {'path': ['/usr/jdk64/jdk1.8.0_77/bin']} ``` I get ``` Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1/package/scripts/enrichment_master.py", line 117, in Enrichment().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1/package/scripts/enrichment_master.py", line 58, in start self.configure(env) File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1/package/scripts/enrichment_master.py", line 50, in configure metron_service.refresh_configs(params) File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1/package/scripts/metron_service.py", line 129, in refresh_configs patch_global_config(params) File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.1/package/scripts/metron_service.py", line 111, in patch_global_config path=ambari_format("{java_home}/bin") File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 293, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/metron/0.4.1/bin/zk_load_configs.sh --zk_quorum node1:2181 --mode PATCH --config_type GLOBAL --patch_path /tmp/metron-global-config-patch.json' returned 255. Unable to parse args: --zk_quorum node1:2181 --mode PATCH --config_type GLOBAL --patch_path /tmp/metron-global-config-patch.json org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: --patch_path at org.apache.commons.cli.Parser.processOption(Parser.java:363) at org.apache.commons.cli.Parser.parse(Parser.java:199) at org.apache.commons.cli.Parser.parse(Parser.java:85) at org.apache.metron.common.cli.ConfigurationManager$ConfigurationOptions.parse(ConfigurationManager.java:140) at org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:352) usage: configuration_manager -c,--config_type The configuration type: GLOBAL, PARSER, ENRICHMENT, INDEXING, PROFILER -f,--forceForce operation -h,--help Generate Help screen -i,--input_dir The input directory containing the configuration files named like "$source.json" -m,--mode The mode of operation: DUMP, PULL, PUSH, PATCH -n,--config_name The configuration name: bro, yaf, snort, squid, etc. -o,--output_dir The output directory which will store the JSON configuration from Zookeeper -pf,--patch_file Path to the patch file. -pk,--patch_keyThe key to modify
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170784#comment-16170784 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139550653 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/cli/ConfigurationManager.java --- @@ -130,29 +201,39 @@ public void pull(CuratorFramework client, String outFileStr, final boolean force public void visit(ConfigurationType configurationType, String name, String data) { File out = getFile(outputDir, configurationType, name); if (!out.exists() || force) { - if(!out.exists()) { + if (!out.exists()) { out.getParentFile().mkdirs(); } try { Files.write(data, out, Charset.defaultCharset()); } catch (IOException e) { throw new RuntimeException("Sorry, something bad happened writing the config to " + out.getAbsolutePath() + ": " + e.getMessage(), e); } -} -else if(out.exists() && !force) { +} else if (out.exists() && !force) { throw new IllegalStateException("Unable to overwrite existing file (" + out.getAbsolutePath() + ") without the force flag (-f or --force) being set."); } } }); } public void push(String inputDirStr, CuratorFramework client) throws Exception { - final File inputDir = new File(inputDirStr); +final File inputDir = new File(inputDirStr); --- End diff -- Latest commit addresses the input dir null check. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170775#comment-16170775 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139547949 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/cli/ConfigurationManager.java --- @@ -130,29 +201,39 @@ public void pull(CuratorFramework client, String outFileStr, final boolean force public void visit(ConfigurationType configurationType, String name, String data) { File out = getFile(outputDir, configurationType, name); if (!out.exists() || force) { - if(!out.exists()) { + if (!out.exists()) { out.getParentFile().mkdirs(); } try { Files.write(data, out, Charset.defaultCharset()); } catch (IOException e) { throw new RuntimeException("Sorry, something bad happened writing the config to " + out.getAbsolutePath() + ": " + e.getMessage(), e); } -} -else if(out.exists() && !force) { +} else if (out.exists() && !force) { throw new IllegalStateException("Unable to overwrite existing file (" + out.getAbsolutePath() + ") without the force flag (-f or --force) being set."); } } }); } public void push(String inputDirStr, CuratorFramework client) throws Exception { - final File inputDir = new File(inputDirStr); +final File inputDir = new File(inputDirStr); --- End diff -- Yeah, I think we could definitely handle the scripts better in general. I'm not opposed to just leaving this as-is, but it seems like a decent / small opportunity to make it more usable. Up to you. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170650#comment-16170650 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 @justinleet Looks like you found the missing -i option. Also, it's funny that I confirmed "pp" and still ended up typing "pf." While I'm in here, what do you think is better, pp or pf? I'm happy to change it. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170624#comment-16170624 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/760 I spun this up in full dev and it worked really well. I think it's a great step towards cleaning up our config management. I'm still looking over some of the review updates and discussion, but this is a great. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170615#comment-16170615 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/760 Couple notes on the test instructions for anyone trying this out. * Missing the input dir flag on the PUSH instructions, just need -i: `$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c GLOBAL -i $METRON_HOME/config/zookeeper` `$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c PARSER -n bro -i $METRON_HOME/config/zookeeper` * The flag for files isn't `-pf`, it's `-pp` for patch path > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170600#comment-16170600 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139526927 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/cli/ConfigurationManager.java --- @@ -130,29 +201,39 @@ public void pull(CuratorFramework client, String outFileStr, final boolean force public void visit(ConfigurationType configurationType, String name, String data) { File out = getFile(outputDir, configurationType, name); if (!out.exists() || force) { - if(!out.exists()) { + if (!out.exists()) { out.getParentFile().mkdirs(); } try { Files.write(data, out, Charset.defaultCharset()); } catch (IOException e) { throw new RuntimeException("Sorry, something bad happened writing the config to " + out.getAbsolutePath() + ": " + e.getMessage(), e); } -} -else if(out.exists() && !force) { +} else if (out.exists() && !force) { throw new IllegalStateException("Unable to overwrite existing file (" + out.getAbsolutePath() + ") without the force flag (-f or --force) being set."); } } }); } public void push(String inputDirStr, CuratorFramework client) throws Exception { - final File inputDir = new File(inputDirStr); +final File inputDir = new File(inputDirStr); --- End diff -- The examples in the test plan don't work directly because they NPE here. I assume it's because inputDirStr is null here and we only check for validity after wrapping it in a `File`. While you're in here, could you add a check (and meaningful exception) for the case where inputDirStr is null? Error is: ``` [root@node1 zookeeper]# $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c GLOBAL Exception in thread "main" java.lang.NullPointerException at java.io.File.(File.java:277) at org.apache.metron.common.cli.ConfigurationManager.push(ConfigurationManager.java:230) at org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:256) at org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:242) at org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:350) ``` > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170483#comment-16170483 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 I also ran this through testing with Kerberos enabled in full dev. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170424#comment-16170424 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 ### Test Plan 1. Run up full dev and verify you have data flowing through to the indexes 2. Change global config options. 1. Go into the Ambari management UI and modify any of the following config options. e.g. set the es_clustername property to "ivebeenchanged". These are values that end up in global.json and will prompt for a component restart. Restart the components. - es.clustername - es.ip - es.date.format - parser.error.topic - update.hbase.table - update.hbase.cf - profiler.client.period.duration - profiler.client.period.duration.units 2. Verify $METRON_HOME/config/zookeeper/global.json now has the new value. For the example above, you would expect to see es.clustername=ivebeenchanged. 3. Verify Zookeeper contains the new config. `$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c GLOBAL` 3. Test changing configs from the command line. The zk config utils has been updated with new backwards compatible features, so we'll test that here. 1. Change one of the zookeeper configs locally and then try a vanilla config PUSH using the original method. Verify the values made it into zookeeper. `${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PUSH -z $ZOOKEEPER` 2. Change a config locally and push JUST that config to Zookeeper by using one of the new config loader options. You can then perform a DUMP in a similar manner. ``` # global config only $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c GLOBAL # bro config only $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PUSH -c PARSER -n bro # take a dump of config $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c GLOBAL $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m DUMP -c PARSER -n bro ``` 3. Try the new JSON patch mechanism with key/value from the command line. Below are some examples but please feel free to explore other combinations. **Note** - commons cli strips the first outermost pair of quotes from all arguments, so you have to provide escaped quotes twice. Neither \""bar"\" nor "\"bar\"" will work. You must provide both escaped quotes for string values. \"\"bar\"\". Complex objects are a bit easier, but you still have to escape quotes inside the string to get them to parse properly. ``` # global config only - add a new key "foo" with simple string value "bar" $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pm ADD -pk foo -pv \"\"bar\"\" # global config only - add a new key "foo" to the JSON doc root with complex value "{ \"bar\" : { \"baz\" : [ \"bazval1\", \"bazval2\" ] } }" $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pm ADD -pk "/foo" -pv "{ \"bar\" : { \"baz\" : [ \"bazval1\", \"bazval2\" ] } }" # global config only - remove a key altogether $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pm REMOVE -pk "/foo" ``` 4. Try the new JSON patch mechanism with a patch file. Below are some examples but please feel free to explore other combinations. Note that a patch file is a JSON array of individual patches. 1. Try a single patch from file ``` # Create a single patch that adds a new node. Create file "/tmp/mypatch.txt" and add the following JSON array. [ { "op": "add", "path": "/you", "value": { "cannot" : { "handle" : [ "the", "flow" ] } } } ] # Perform the patch $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pf /tmp/mypatch.txt # Should result in adding the following to global.json "you" : { "cannot" : { "handle" : [ "the", "flow" ] } } ``` 2. Try multiple patches in a patch file ``` Create a single patch file that adds multiple nodes. Create file "/tmp/mygrimespatch.txt" and add the following JSON array. # 2 patches in a single patch file [ { "op": "add", "path": "/la", "value": { "jiggy" : { "jar" : { "jar" : "do" } } }
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168835#comment-16168835 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc closed the pull request at: https://github.com/apache/metron/pull/760 > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168789#comment-16168789 ] ASF GitHub Bot commented on METRON-1188: Github user cestella commented on the issue: https://github.com/apache/metron/pull/760 Why would those .j2 files be in gitignore? > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168784#comment-16168784 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 Found a possible problem. I was wondering why my changes aren't making it into my commits for the enrichment properties. There's this https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/.gitignore but also this https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py#L41 > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168643#comment-16168643 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 Merge conflicts, of course. Unfortunately, I have an appointment and will be back in a couple hours to fix. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168642#comment-16168642 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 Just pushed out a commit to address comments. I added documentation around the config management lifecycle to the Ambari README. I'm continuing with testing and will verify Kerberos works as expected also. More testing details to follow. @justinleet I'm looking also at options for added logging per your comments above. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168479#comment-16168479 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 You're never going to believe this, but I found our Ambari docs under, wait for it, the ambari project in packaging. https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari. Thanks @justinleet and @merrimanr for writing that doc. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168196#comment-16168196 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 @ottobackwards Added the mailing list ref > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168188#comment-16168188 ] ASF GitHub Bot commented on METRON-1188: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/760 So, it is not uncommon for a PR that has a discussion thread backing up it's approach to reference that thread to point out the public discussion when it has occurred. Since this thread exists, it should be referenced in the jira or the PR. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168149#comment-16168149 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 Just pushed a unit test fix to get the build working again. The other code changes and doc requests from @ottobackwards and @justinleet soon to follow. Thanks for the feedback guys! @ottobackwards let me know how I can help wrt the feature branch and these changes. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168134#comment-16168134 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139194807 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -226,6 +276,44 @@ public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramewor uploadConfigsToZookeeper(rootFilePath, rootFilePath, rootFilePath, rootFilePath, rootFilePath, client); } + public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramework client, + ConfigurationType type) throws Exception { +uploadConfigsToZookeeper(rootFilePath, client, type, Optional.empty()); + } + + public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramework client, + ConfigurationType type, Optional configName) throws Exception { +switch (type) { + case GLOBAL: +final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath); +if (globalConfig.length > 0) { + setupStellarStatically(client, Optional.of(new String(globalConfig))); + writeGlobalConfigToZookeeper(readGlobalConfigFromFile(rootFilePath), client); +} +break; + case PARSER: --- End diff -- Ha, yes. Good catch, I missed refactoring that piece. Red, green, refactor Mike! Red, green, refactor! > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168140#comment-16168140 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139194942 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -226,6 +276,44 @@ public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramewor uploadConfigsToZookeeper(rootFilePath, rootFilePath, rootFilePath, rootFilePath, rootFilePath, client); } + public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramework client, + ConfigurationType type) throws Exception { +uploadConfigsToZookeeper(rootFilePath, client, type, Optional.empty()); + } + + public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramework client, + ConfigurationType type, Optional configName) throws Exception { +switch (type) { + case GLOBAL: +final byte[] globalConfig = readGlobalConfigFromFile(rootFilePath); +if (globalConfig.length > 0) { + setupStellarStatically(client, Optional.of(new String(globalConfig))); + writeGlobalConfigToZookeeper(readGlobalConfigFromFile(rootFilePath), client); +} +break; + case PARSER: +MapsensorParserConfigs = readSensorConfigsFromFile(rootFilePath, PARSER, configName); +for (String sensorType : sensorParserConfigs.keySet()) { + writeConfigToZookeeper(type, configName, sensorParserConfigs.get(sensorType), client); +} +break; + case ENRICHMENT: +Map sensorEnrichmentConfigs = readSensorConfigsFromFile(rootFilePath, ENRICHMENT, configName); --- End diff -- Will fix this also. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168128#comment-16168128 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139194190 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py --- @@ -55,14 +56,14 @@ def is_kafka_acl_configured(self): def set_kafka_configured(self): Logger.info("Setting Kafka Configured to True") File(self.__params.enrichment_kafka_configured_flag_file, - content="", + content="This file created on: " + datetime.now().strftime('%Y-%m-%d %H:%M:%S'), --- End diff -- Yeah, probably worthwhile. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168120#comment-16168120 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 > Can you reference : http://mail-archives.apache.org/mod_mbox/metron-dev/201701.mbox/%3cCAPpQHK1svDAx7R-s7x2Q0kJR6=5d+KDmR6a+ZmSo-5=wcmo...@mail.gmail.com%3e > Which I think is the discuss list? Not sure I follow - you want me to reference it in the docs? This doesn't change the current paradigm of Zookeeper being the source of truth at all times. > Is this a standard way of working between ambari and zookeeper or a new take? There isn't really a standard way of managing configs in Zookeeper in Ambari that I'm aware of. This is mostly fixing a gap in how we handle materializing configs from the properties that are managed via the Ambari interface. > I have not found it this morning, but my recollection is we did some documentation around ambari and what we do there, do you plan on putting in something about this for maintainers etc? Yes, completely agreed. Per my initial PR comments this workflow should definitely be explained so the implications of making config changes from local disk, Ambari, Zookeeper, and the management UI are well understood. I have to look for where we put that. While I'm addressing some other code review comments, if anyone happens to know where off the top of their head, that would be helpful. > Do you plan on integrating this into the feature branch yourself? Or were you planning on me doing it? This is backwards compatible with existing config management. The only major change would be around Ambari: 1. actually writing the global config from properties held in the Ambari UI (as opposed to the hard-coded values) and 2. using a patching (additive in nature, overwrite in the event of property conflict) mechanism 3. doing a zk config pull to local disk after any service starts/restarts. What problems with integration are you expecting? There aren't any new classes introduced. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168096#comment-16168096 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139190738 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -76,7 +80,7 @@ public static void writeProfilerConfigToZookeeper(byte[] config, CuratorFramewor } public static void writeSensorParserConfigToZookeeper(String sensorType, SensorParserConfig sensorParserConfig, String zookeeperUrl) throws Exception { -writeSensorParserConfigToZookeeper(sensorType, JSONUtils.INSTANCE.toJSON(sensorParserConfig), zookeeperUrl); +writeSensorParserConfigToZookeeper(sensorType, JSONUtils.INSTANCE.toJSONPretty(sensorParserConfig), zookeeperUrl); --- End diff -- Yeah, I hastily read that stacktrace. This was indeed the problem. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167994#comment-16167994 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139172028 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -346,19 +434,56 @@ public static void setupStellarStatically(CuratorFramework client, Optional readSensorConfigsFromFile(String rootPath, ConfigurationType configType) throws IOException { +return readSensorConfigsFromFile(rootPath, configType, Optional.empty()); + } + + public static MapreadSensorConfigsFromFile(String rootPath, + ConfigurationType configType, Optional configName) throws IOException { Map sensorConfigs = new HashMap<>(); File configPath = new File(rootPath, configType.getDirectory()); -if (configPath.exists()) { +if (configPath.exists() && configPath.isDirectory()) { File[] children = configPath.listFiles(); - if (children != null) { + if (!configName.isPresent()) { for (File file : children) { - sensorConfigs.put(FilenameUtils.removeExtension(file.getName()), Files.readAllBytes(file.toPath())); + sensorConfigs.put(FilenameUtils.removeExtension(file.getName()), + Files.readAllBytes(file.toPath())); +} + } else { +for (File file : children) { + if (FilenameUtils.removeExtension(file.getName()).equals(configName.get())) { + sensorConfigs.put(FilenameUtils.removeExtension(file.getName()), +Files.readAllBytes(file.toPath())); + } +} +if (sensorConfigs.isEmpty()) { + throw new RuntimeException("Unable to find configuration for " + configName.get()); } } } return sensorConfigs; } + public static void applyConfigPatchToZookeeper(ConfigurationType configurationType, byte[] patchData, String zookeeperUrl) throws Exception { --- End diff -- Javadocs again. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167991#comment-16167991 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139160248 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py --- @@ -55,14 +56,14 @@ def is_kafka_acl_configured(self): def set_kafka_configured(self): Logger.info("Setting Kafka Configured to True") File(self.__params.enrichment_kafka_configured_flag_file, - content="", + content="This file created on: " + datetime.now().strftime('%Y-%m-%d %H:%M:%S'), --- End diff -- Mostly thinking out loud about a follow on, would it be worth it to just pull these file creations (and probably a variant of the Logger statement) into a `createConfiguredFlagFile` function in the common area, and just pass it the flag file? They all have the same content, owner, and perms, right? > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167990#comment-16167990 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139171727 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -346,19 +434,56 @@ public static void setupStellarStatically(CuratorFramework client, Optional readSensorConfigsFromFile(String rootPath, ConfigurationType configType) throws IOException { +return readSensorConfigsFromFile(rootPath, configType, Optional.empty()); + } + + public static MapreadSensorConfigsFromFile(String rootPath, --- End diff -- Would you mind adding Javadocs while you're already in the functions? > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167989#comment-16167989 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139172458 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -401,7 +530,16 @@ else if (configType.equals(PARSER) || configType.equals(ENRICHMENT) || configTyp public static void dumpConfigs(PrintStream out, CuratorFramework client) throws Exception { ConfigurationsUtils.visitConfigs(client, (type, name, data) -> { type.deserialize(data); - out.println(type + " Config: " + name + "\n" + data); + out.println(type + " Config: " + name + System.lineSeparator() + data); }); } + + public static void dumpConfigs(PrintStream out, CuratorFramework client, --- End diff -- Javadocs while you're already in here, if you don't mind. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167988#comment-16167988 ] ASF GitHub Bot commented on METRON-1188: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139170122 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -226,6 +276,44 @@ public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramewor uploadConfigsToZookeeper(rootFilePath, rootFilePath, rootFilePath, rootFilePath, rootFilePath, client); } + public static void uploadConfigsToZookeeper(String rootFilePath, CuratorFramework client, --- End diff -- Can you add Javadocs to these methods? > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167865#comment-16167865 ] ASF GitHub Bot commented on METRON-1188: Github user cestella commented on the issue: https://github.com/apache/metron/pull/760 A bit on the second point: I would argue this is a new (more correct) take if, for no other reason, that there isn't to my my knowledge other patterns to emulate here (other projects don't let ambari have control over their zk configs). The problem was letting ambari own the whole global config on disk in the first place via the templating system. This moves to a more rational approach of just having ambari apply patches to the config within the global config that it owns, rather than controlling the whole file/global config. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167716#comment-16167716 ] ASF GitHub Bot commented on METRON-1188: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/760 Thanks for the contribution! A couple of questions before I try to review: - Can you reference : http://mail-archives.apache.org/mod_mbox/metron-dev/201701.mbox/%3cCAPpQHK1svDAx7R-s7x2Q0kJR6=5d+KDmR6a+ZmSo-5=wcmo...@mail.gmail.com%3e Which I think is the discuss list? - Is this a standard way of working between ambari and zookeeper or a new take? - I have not found it this morning, but my recollection is we did some documentation around ambari and what we do there, do you plan on putting in something about this for maintainers etc? - Do you plan on integrating this into the feature branch yourself? Or were you planning on me doing it? > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167500#comment-16167500 ] ASF GitHub Bot commented on METRON-1188: Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/760#discussion_r139086639 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigurationsUtils.java --- @@ -76,7 +80,7 @@ public static void writeProfilerConfigToZookeeper(byte[] config, CuratorFramewor } public static void writeSensorParserConfigToZookeeper(String sensorType, SensorParserConfig sensorParserConfig, String zookeeperUrl) throws Exception { -writeSensorParserConfigToZookeeper(sensorType, JSONUtils.INSTANCE.toJSON(sensorParserConfig), zookeeperUrl); +writeSensorParserConfigToZookeeper(sensorType, JSONUtils.INSTANCE.toJSONPretty(sensorParserConfig), zookeeperUrl); --- End diff -- So, I think the test is failing because you are pretty printing here. `GlobalConfigServiceImplTest` functions are expecting to write out `{}` and retrieve back literally `{}` and, yet, you are pretty printing, which would yield `{ }`. I think what you're doing is fine, but you probably want to adjust the failing `GlobalConfigServiceImplTest` cases to expect `{ }` rather than `{}`. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167236#comment-16167236 ] ASF GitHub Bot commented on METRON-1188: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/760 Hm, I'll have to look into that build failure. I don't think I touched anything in the profiler, but that's where the test failures are. > Ambari global configuration management broken > - > > Key: METRON-1188 > URL: https://issues.apache.org/jira/browse/METRON-1188 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > Ambari currently ignores values from the UI when managing the global.json > file. These values are hard-coded in the status_params.py file as part of the > mpack. In addition, Ambari restarts are clobbering configuration changes in > Zookeeper. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1188) Ambari global configuration management broken
[ https://issues.apache.org/jira/browse/METRON-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167208#comment-16167208 ] ASF GitHub Bot commented on METRON-1188: GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/760 METRON-1188: Ambari global configuration management broken ## Contributor Comments This PR addresses https://issues.apache.org/jira/browse/METRON-1188 There are a number of issues with Ambari configuration management with the Metron configuration stored in Zookeeper. I wanted to get this PR out for discussion while I finish testing/tweaking docs around this because this is a fairly major defect. This is one part new features, one part defect fixes. Here's the quick gist: - Added ability to patch JSON configs per RFC-6902 - Added ability to patch/push/pull/dump individual config types, e.g. GLOBAL, PARSERS, ENRICHMENTS, etc. - Added ability to patch/push/pull/dump individual configuration names given a specific config type, e.g. PARSERS + 'bro' - Fixed the order of operations around configuration refresh - On first service start, these steps are performed - push zookeeper config (initialize zookeeper configs) - run the start/restart commands - set "configured" flag so this initial setup doesn't run again - On service start or restart - create a patch file for the global config in /tmp - read global config from zookeeper, apply patch via new CLI zk_load_configs.sh option (note: I am pretty print formatting the JSON on writing to Zookeeper. ConfigurationManager is a bit of a monstrosity right now, and we should probably clean it up at some point. There are still some modes (PUSH) that miss the pretty print code and I'll work to fix those up also) - pull zookeeper config locally - Added timestamps in the "configured" flag files. Yes, I know you can stat a file, but I like explicitly stating in the file when Ambari suggests it wrote to disk :) - Ambari now patches configuration in an additive way. Note: Any specific configs managed by Ambari will overwrite user changes to those parameters. Here's the patch excerpt from metron_service.py ``` patch_template = """ [ { "op": "add", "path": "/es.clustername", "value": "{{ es_cluster_name }}" }, { "op": "add", "path": "/es.ip", "value": "{{ es_url }}" }, { "op": "add", "path": "/es.date.format", "value": "{{es_date_format}}" }, { "op": "add", "path": "/parser.error.topic", "value": "{{parser_error_topic}}" }, { "op": "add", "path": "/update.hbase.table", "value": "{{update_hbase_table}}" }, { "op": "add", "path": "/update.hbase.cf", "value": "{{update_hbase_cf}}" }, { "op": "add", "path": "/profiler.client.period.duration", "value": "{{profiler_period_duration}}" }, { "op": "add", "path": "/profiler.client.period.duration.units", "value": "{{profiler_period_units}}" } ] """ ``` - removed the global.json.j2 template file and moved a default global.json to metron-common. Modified the assembly.xml and the RPM metron.spec to included this accordingly also. - Fixed global config not updating based on Ambari parameters - look at the Metron status_params.py file to see the changes. Also updated the parameter names to be consistent and modified metron-env.xml and metron-enrichment.xml to properly include these params as well. A friendlier, detailed testing plan for this is coming very soon. Minimally, you should be able to spin up full dev without it failing. A green deployment is a major indicator we're looking good. I'm currently working through Kerberos testing and various restart combinations. Any of the parameters in that patch list above are good candidates to mess around with in Ambari to ensure they are cascaded to both Zookeeper and the local file system. Also check that any new parameters pushed from a local global.json change to Zookeeper, e.g. "blah" : "bloop", are not blown away by restarting Metron services. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for