-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
-----------------------------------------------------------
(Updated March 18, 2016, 8:57 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo
Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420
Repository: ambari
Description
-------
Refactor resource_management library, to generalize the library and remove
HDP-specific hardcodings
Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding
hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of
hardcoding the "/usr/hdp"
In addition, also added a feature in the stack processing engine to support
properties values to be defined a external property file (See tarball_map.json,
stack_tools.json in patch)
Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(),
get_tarball_map() are added in script.py, which will set the defaults if these
config properties are not defined (ambari-server upgrade scenario needs to be
addressed to add these config properties on upgrade). These helper functions
are used to remove hardcodings in resource_management library.
Remaining HDP-specific logic in resource_management library
1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()
Refactoring this HDP-specific logic would require "Stack Featurization"
(AMBARI-13364) to be in place.
Diffs
-----
ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py
ee7db0a
ambari-common/src/main/python/resource_management/core/utils.py 71d0008
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
770595f
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
647b8b6
ambari-common/src/main/python/resource_management/libraries/functions/default.py
23383dc
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
a20b03c
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
f2e6567
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py
f1cd9cb
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py
d585dea
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py
4d9d8a4
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
2ccc0c6
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
4a8eeb9
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py
95c5cba
ambari-common/src/main/python/resource_management/libraries/script/script.py
1b0116e
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
4be4049
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java
7f21aaa
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java
0c7faea
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java
c739211
ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java
7d934bb
ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java
e7c9c27
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java
3f7f756
ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java
PRE-CREATION
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
12ca388
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py
45dbb24
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
d000846
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py
2c06c40
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py
535c821
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py
85e7012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_upgrade.py
c3d15e5
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
215e012
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
7aeac24
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py
30b9a41
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
81a227e
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py
030fb2d
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
27e2766
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/status_params.py
954bb80
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
e5b54cd
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
906b198
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
e110c0c
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/status_params.py
984a4ba
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
eb80ad6
ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_windows.py
dd732f5
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
db4f32e
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py
e05ed60
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
d18e4d7
ambari-server/src/main/resources/custom_actions/scripts/install_packages.py
e0f3752
ambari-server/src/main/resources/custom_actions/scripts/remove_bits.py
e69a2e4
ambari-server/src/main/resources/custom_actions/scripts/ru_set_all.py b8bf176
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml
6197d87
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py
96dc104
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py
07faae4
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_tools.json
PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/tarball_map.json
PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/stack/ServiceModuleTest.java
92c1200
ambari-server/src/test/java/org/apache/ambari/server/utils/TestJsonUtils.java
PRE-CREATION
ambari-server/src/test/python/custom_actions/test_ru_set_all.py 3090f6b
ambari-server/src/test/python/host_scripts/TestAlertDiskSpace.py 2608050
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_hdfs_client.py b5b43d6
ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_client.py
efe6038
ambari-server/src/test/python/stacks/2.0.6/YARN/test_yarn_client.py 4601092
ambari-server/src/test/python/stacks/2.0.6/hooks/after-INSTALL/test_after_install.py
31dac56
ambari-server/src/test/python/stacks/2.1/TEZ/test_tez_client.py ab08776
Diff: https://reviews.apache.org/r/44835/diff/
Testing (updated)
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [3.906s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.029s]
[INFO] Ambari Web ........................................ SUCCESS [50.786s]
[INFO] Ambari Views ...................................... SUCCESS [3.417s]
[INFO] Ambari Admin View ................................. SUCCESS [6.596s]
[INFO] ambari-metrics .................................... SUCCESS [0.215s]
[INFO] Ambari Metrics Common ............................. SUCCESS [3.073s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.126s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [3.080s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [4.413s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [1.547s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [9:53.524s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.886s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [39.843s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [3:25.123s]
[INFO] Ambari Server ..................................... SUCCESS
[1:07:30.219s]
[INFO] Ambari Functional Tests ........................... SUCCESS [3.081s]
[INFO] Ambari Agent ...................................... SUCCESS [14.506s]
[INFO] Ambari Client ..................................... SUCCESS [0.063s]
[INFO] Ambari Python Client .............................. SUCCESS [0.381s]
[INFO] Ambari Groovy Client .............................. SUCCESS [13.112s]
[INFO] Ambari Shell ...................................... SUCCESS [0.034s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.044s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [9.936s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:23:34.755s
[INFO] Finished at: Fri Mar 18 13:49:52 PDT 2016
[INFO] Final Memory: 103M/1003M
[INFO] ------------------------------------------------------------------------
Thanks,
Jayush Luniya
