-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50350/
-----------------------------------------------------------

(Updated July 26, 2016, 6:48 p.m.)


Review request for Ambari, Di Li, Jonathan Hurley, Jayush Luniya, Madhan 
Neethiraj, Nate Cole, Sumit Mohanty, Swapan Shridhar, Suma Shivaprasad, Sid 
Wagle, and Tim Thorpe.


Changes
-------

Addressed comments.


Bugs: AMBARI-17782
    https://issues.apache.org/jira/browse/AMBARI-17782


Repository: ambari


Description
-------

To support Atlas in HDP 2.5, make several config changes:

/etc/hive/conf/atlas-application.properties:

* atlas.rest.address (add this property)
 
/etc/hive/conf/hive-site.xml:

* atlas.cluster.name (remove this property)
* atlas.hook.hive.maxThreads (remove this property)
* atlas.hook.hive.minThreads (remove this property)
* atlas.rest.address (remove this property)

/etc/storm/conf/atlas-application.properties:

* atlas.rest.address (add this property)
 
/etc/storm/conf/storm.yaml:

* atlas.cluster.name (remove this property)
 
/etc/falcon/conf/atlas-application.properties:

* atlas.rest.address (add this property)
* atlas.cluster.name (this value is empty; need to set to correct value)
 
 
/etc/sqoop/conf/atlas-application.properties:

* atlas.jaas.KafkaClient.option.keyTab (remove this property)
* atlas.jaas.KafkaClient.option.principal (remove this property)
* atlas.jaas.KafkaClient.option.storeKey (remove this property)
* atlas.jaas.KafkaClient.option.useKeyTab (remove this property)
* atlas.jaas.KafkaClient.option.useTicketCache=true (add this property)
* atlas.jaas.KafkaClient.option.renewTicket=true (add this property)
* atlas.rest.address (add this property)
 
/etc/sqoop/conf/sqoop-site.xml:

* atlas.cluster.name (remove this property)

Also, there is no 'Custom sqoop-atlas-application.properties' section in Sqoop


Diffs (updated)
-----

  ambari-common/src/main/python/ambari_commons/constants.py 525e024 
  
ambari-common/src/main/python/resource_management/libraries/functions/setup_atlas_hook.py
 a384f69 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 f0a3f32 
  
ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/application-properties.xml
 4305965 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
 6111c34 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-atlas-application.properties.xml
 0590244 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-site.xml
 c53dd39 
  ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 
ec14979 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat.py
 f53625c 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive.py
 6b05134 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat.py
 4f38055 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml
 d78c5be 
  
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py
 963d169 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/storm.py
 d765ca3 
  
ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml
 3a3879b 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
8d5cdc9 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 
bfdb3d3 
  
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml
 a1b93e3 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
96b1400 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 
b2cc1c4 
  
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml
 86e0964 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
2099958 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-atlas-application.properties.xml
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/metainfo.xml 
88c2aae 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/configuration/sqoop-site.xml
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/SQOOP/kerberos.json 
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
4972972 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
 7aa4a4e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 34bebe8 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py ebd2f2a 
  ambari-server/src/test/python/stacks/2.5/configs/default.json a7e380a 

Diff: https://reviews.apache.org/r/50350/diff/


Testing
-------

Verified the following test cases.


1. Install Ambari 2.4, HDP 2.5 along with Hive, Storm, Kafka, Sqoop and all of 
their dependencies.
Verify that none of the atlas.* configs exist nor the 
$HOOK-atlas-application.properties file
Ensure that hive-site and storm.yaml don't have any Atlas properties.

2. After #1, add Atlas and verify that all of the hooks have atlas.rest.address 
and atlas.cluster.name


3. After #2, kerberize the cluster and ensure Sqoop has the right configs in 
its sqoop-atlas-application.properties file.


4. After #1, kerberize the cluster. This should add the application configs to 
sqoop-atlas-application.properties in the DB, but the file will not be saved to 
the local file system until Atlas is added.
After this, add Atlas service (and its dependencies), and ensure that all of 
the configs in /etc/sqoop/conf/sqoop-atlas-application.properties are correct.


5. Install Ambari 2.2.2 with HDP 2.4 along with Hive, Storm, Kafka, Sqoop, and 
Atlas. Upgrade Ambari to 2.4.0 and ensure Atlas still works.
Install bits for HDP 2.5, remove Atlas, and perform either an EU/RU to HDP 2.5.
This should remove atlas.cluster.name and atlas.rest.address from hive-site, 
plus several other security-related properties from sqoop-site if the cluster 
was kerberized.


Thanks,

Alejandro Fernandez

Reply via email to