-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50350/
-----------------------------------------------------------
(Updated July 26, 2016, 6:48 p.m.)
Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Madhan
Neethiraj, Nate Cole, Sumit Mohanty, Swapan Shridhar, Suma Shivaprasad, Sid
Wagle, and Tim Thorpe.
Changes
-------
Addressed comments.
Bugs: AMBARI-17782
https://issues.apache.org/jira/browse/AMBARI-17782
Repository: ambari
Description
-------
To support Atlas in HDP 2.5, make several config changes:
/etc/hive/conf/atlas-application.properties:
* atlas.rest.address (add this property)
/etc/hive/conf/hive-site.xml:
* atlas.cluster.name (remove this property)
* atlas.hook.hive.maxThreads (remove this property)
* atlas.hook.hive.minThreads (remove this property)
* atlas.rest.address (remove this property)
/etc/storm/conf/atlas-application.properties:
* atlas.rest.address (add this property)
/etc/storm/conf/storm.yaml:
* atlas.cluster.name (remove this property)
/etc/falcon/conf/atlas-application.properties:
* atlas.rest.address (add this property)
* atlas.cluster.name (this value is empty; need to set to correct value)
/etc/sqoop/conf/atlas-application.properties:
* atlas.jaas.KafkaClient.option.keyTab (remove this property)
* atlas.jaas.KafkaClient.option.principal (remove this property)
* atlas.jaas.KafkaClient.option.storeKey (remove this property)
* atlas.jaas.KafkaClient.option.useKeyTab (remove this property)
* atlas.jaas.KafkaClient.option.useTicketCache=true (add this property)
* atlas.jaas.KafkaClient.option.renewTicket=true (add this property)
* atlas.rest.address (add this property)
/etc/sqoop/conf/sqoop-site.xml:
* atlas.cluster.name (remove this property)
Also, there is no 'Custom sqoop-atlas-application.properties' section in Sqoop
Diffs (updated)
-----
ambari-common/src/main/python/ambari_commons/constants.py 525e024
ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
a384f69
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
f0a3f32
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/application-properties.xml
4305965
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
6111c34
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
0590244
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml
c53dd39
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml
ec14979
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
f53625c
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
6b05134
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py
4f38055
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml
d78c5be
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py
963d169
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py
d765ca3
ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml
3a3879b
ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py
8d5cdc9
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml
bfdb3d3
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
a1b93e3
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml
96b1400
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml
b2cc1c4
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
86e0964
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml
2099958
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-atlas-application.properties.xml
PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml
88c2aae
ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/configuration/sqoop-site.xml
PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json
PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
4972972
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
7aa4a4e
ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 34bebe8
ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py ebd2f2a
ambari-server/src/test/python/stacks/2.5/configs/default.json a7e380a
Diff: https://reviews.apache.org/r/50350/diff/
Testing
-------
Verified the following test cases.
1. Install Ambari 2.4, HDP 2.5 along with Hive, Storm, Kafka, Sqoop and all of
their dependencies.
Verify that none of the atlas.* configs exist nor the
$HOOK-atlas-application.properties file
Ensure that hive-site and storm.yaml don't have any Atlas properties.
2. After #1, add Atlas and verify that all of the hooks have atlas.rest.address
and atlas.cluster.name
3. After #2, kerberize the cluster and ensure Sqoop has the right configs in
its sqoop-atlas-application.properties file.
4. After #1, kerberize the cluster. This should add the application configs to
sqoop-atlas-application.properties in the DB, but the file will not be saved to
the local file system until Atlas is added.
After this, add Atlas service (and its dependencies), and ensure that all of
the configs in /etc/sqoop/conf/sqoop-atlas-application.properties are correct.
5. Install Ambari 2.2.2 with HDP 2.4 along with Hive, Storm, Kafka, Sqoop, and
Atlas. Upgrade Ambari to 2.4.0 and ensure Atlas still works.
Install bits for HDP 2.5, remove Atlas, and perform either an EU/RU to HDP 2.5.
This should remove atlas.cluster.name and atlas.rest.address from hive-site,
plus several other security-related properties from sqoop-site if the cluster
was kerberized.
Thanks,
Alejandro Fernandez
