----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35607/#review88379 -----------------------------------------------------------
Ship it! Ship It! - John Speidel On June 18, 2015, 2:49 p.m., Robert Nettleton wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35607/ > ----------------------------------------------------------- > > (Updated June 18, 2015, 2:49 p.m.) > > > Review request for Ambari, John Speidel, Mahadev Konar, and Robert Levas. > > > Bugs: AMBARI-12000 > https://issues.apache.org/jira/browse/AMBARI-12000 > > > Repository: ambari > > > Description > ------- > > This patch resolves AMBARI-12000. > > An export of a Blueprint from a running, Kerberized cluster would include the > following properties with hostnames: > > "admin_server_host" in "kerberos-env" > "kdc_host" in "kerberos-env" > "hadoop.proxyuser.yarn.hosts" in "core-site" > > Exported Blueprints should not contain hostnames in general, since this makes > the Blueprint less portable over different cluster configurations. > > This patch implements the following changes: > > 1. Adds a filter to remove the "admin_server_host" and "kdc_host" properties > from an exported Blueprint. These hostname properties refer to the KDC > server of a Kerberized cluster, and so are not useful to an exported > Blueprint at this time. Since these properties generally refer to an > external KDC server, it is likely that a new cluster would point to a new KDC > install, which is typically an external server. > 2. Adds a property updaters to mask the hostname information in the > "hadoop.proxyuser.yarn.hosts" property in "core-site". This property > includes a comma-separated list of hostnames that include Yarn > ResourceManager instances. While not immediately useful outside of Kerberos, > having this property include the %HOSTGROUP% syntax for hostname substitution > is in keeping with the handling of other "*proxyuser*" properties, and could > eventually be helpful in creating Kerberized clusters in future versions of > Ambari. > 3. Adds a unit test to verify the changes to the handling for these three > properties. > > > Diffs > ----- > > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java > af3b331 > > ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java > abf5675 > > Diff: https://reviews.apache.org/r/35607/diff/ > > > Testing > ------- > > 1. Ran the ambari-server unit test suite. Results: > > Java Tests: > > "Results : > > Tests run: 3088, Failures: 0, Errors: 0, Skipped: 28" > > Python Tests: > > "Failed tests: > FAIL: test_recommendTezConfigurations > (test_stack_advisor.TestHDP23StackAdvisor) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/Users/rnettleton/dev/workspaces/ambari/ambari-common/src/test/python/mock/mock.py", > line 1199, in patched > return func(*args, **keywargs) > File > "/Users/rnettleton/dev/workspaces/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py", > line 661, in test_recommendTezConfigurations > self.assertEquals(configurations, expected) > AssertionError: {'capacity-scheduler': {'properties': > {'yarn.scheduler.capacity.root.queues': 'q [truncated]... != > {'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues': > 'q [truncated]... > {'capacity-scheduler': {'properties': > {'yarn.scheduler.capacity.root.queues': 'queue1,queue2'}}, > 'tez-site': {'properties': {'tez.am.launch.cmd-opts': '-XX:+PrintGCDetails > -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC', > 'tez.am.resource.memory.mb': '4000', > 'tez.runtime.io.sort.mb': '307', > 'tez.runtime.unordered.output.buffer.size-mb': > '57', > 'tez.session.am.dag.submit.timeout.secs': > '600', > 'tez.task.launch.cmd-opts': > '-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA > -XX:+UseParallelGC', > 'tez.task.resource.memory.mb': '768', > - 'tez.tez-ui.history-url.base': > 'http://HW10578.local:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}}, > ? ^^ ^ -- > ^ ^^^ > > + 'tez.tez-ui.history-url.base': > 'http://hw11465.home:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}}, > ? ^^ ^^^ > ^ ^^ > > 'yarn-site': {'properties': {'yarn.scheduler.maximum-allocation-mb': > '8192', > 'yarn.scheduler.minimum-allocation-mb': > '256'}}} > > ---------------------------------------------------------------------- > Total run:762 > Total errors:0 > Total failures:1" > > > The failure seen here also occurs in trunk without my patch applied, and so > is an un-related failure. > > 2. Deployed a single-node HDFS/Yarn/Zookeeper/Metrics cluster with the Ambari > UI, with my patch applied. Exported a Blueprint from this running cluster, > and verified that these properties are no longer include in the exported > Blueprint, or in the case of the proxyuser property are properly masked with > %HOSTGROUP% tokens. Also verified that no hostname information is included > in the exported Blueprint for this cluster. > > > Thanks, > > Robert Nettleton > >
