-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35607/
-----------------------------------------------------------
Review request for Ambari, John Speidel, Mahadev Konar, and Robert Levas.
Bugs: AMBARI-12000
https://issues.apache.org/jira/browse/AMBARI-12000
Repository: ambari
Description
-------
This patch resolves AMBARI-12000.
An export of a Blueprint from a running, Kerberized cluster would include the
following properties with hostnames:
"admin_server_host" in "kerberos-env"
"kdc_host" in "kerberos-env"
"hadoop.proxyuser.yarn.hosts" in "core-site"
Exported Blueprints should not contain hostnames in general, since this makes
the Blueprint less portable over different cluster configurations.
This patch implements the following changes:
1. Adds a filter to remove the "admin_server_host" and "kdc_host" properties
from an exported Blueprint. These hostname properties refer to the KDC server
of a Kerberized cluster, and so are not useful to an exported Blueprint at this
time. Since these properties generally refer to an external KDC server, it is
likely that a new cluster would point to a new KDC install, which is typically
an external server.
2. Adds a property updaters to mask the hostname information in the
"hadoop.proxyuser.yarn.hosts" property in "core-site". This property includes
a comma-separated list of hostnames that include Yarn ResourceManager
instances. While not immediately useful outside of Kerberos, having this
property include the %HOSTGROUP% syntax for hostname substitution is in keeping
with the handling of other "*proxyuser*" properties, and could eventually be
helpful in creating Kerberized clusters in future versions of Ambari.
3. Adds a unit test to verify the changes to the handling for these three
properties.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
af3b331
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
abf5675
Diff: https://reviews.apache.org/r/35607/diff/
Testing
-------
1. Ran the ambari-server unit test suite. Results:
Java Tests:
"Results :
Tests run: 3088, Failures: 0, Errors: 0, Skipped: 28"
Python Tests:
"Failed tests:
FAIL: test_recommendTezConfigurations (test_stack_advisor.TestHDP23StackAdvisor)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/Users/rnettleton/dev/workspaces/ambari/ambari-common/src/test/python/mock/mock.py",
line 1199, in patched
return func(*args, **keywargs)
File
"/Users/rnettleton/dev/workspaces/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py",
line 661, in test_recommendTezConfigurations
self.assertEquals(configurations, expected)
AssertionError: {'capacity-scheduler': {'properties':
{'yarn.scheduler.capacity.root.queues': 'q [truncated]... !=
{'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues':
'q [truncated]...
{'capacity-scheduler': {'properties': {'yarn.scheduler.capacity.root.queues':
'queue1,queue2'}},
'tez-site': {'properties': {'tez.am.launch.cmd-opts': '-XX:+PrintGCDetails
-verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
'tez.am.resource.memory.mb': '4000',
'tez.runtime.io.sort.mb': '307',
'tez.runtime.unordered.output.buffer.size-mb':
'57',
'tez.session.am.dag.submit.timeout.secs': '600',
'tez.task.launch.cmd-opts': '-XX:+PrintGCDetails
-verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC',
'tez.task.resource.memory.mb': '768',
- 'tez.tez-ui.history-url.base':
'http://HW10578.local:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
? ^^ ^ -- ^
^^^
+ 'tez.tez-ui.history-url.base':
'http://hw11465.home:8080/#/main/views/TEZ/0.7.0.2.3.0.0-2155/TEZ_CLUSTER_INSTANCE'}},
? ^^ ^^^ ^
^^
'yarn-site': {'properties': {'yarn.scheduler.maximum-allocation-mb': '8192',
'yarn.scheduler.minimum-allocation-mb': '256'}}}
----------------------------------------------------------------------
Total run:762
Total errors:0
Total failures:1"
The failure seen here also occurs in trunk without my patch applied, and so is
an un-related failure.
2. Deployed a single-node HDFS/Yarn/Zookeeper/Metrics cluster with the Ambari
UI, with my patch applied. Exported a Blueprint from this running cluster, and
verified that these properties are no longer include in the exported Blueprint,
or in the case of the proxyuser property are properly masked with %HOSTGROUP%
tokens. Also verified that no hostname information is included in the exported
Blueprint for this cluster.
Thanks,
Robert Nettleton
