-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26858/
-----------------------------------------------------------
(Updated Oct. 18, 2014, 1:17 a.m.)
Review request for Ambari, Sumit Mohanty and Sid Wagle.
Bugs: AMBARI-7825
https://issues.apache.org/jira/browse/AMBARI-7825
Repository: ambari
Description
-------
HDP 2.2 using versioned RPMs
When Ambari installs Hive, it copies any file that matches
/usr/hdp/current/tez-client/lib/tez*.tar.gz file to hdfs:///apps/tez/tez.tar.gz
(Aside: it also ends up copying other tez-* files that it shouldn't because
they are not needed, and already zipped in the tar.gz file)
The problem is that during a rolling upgrade, currently running jobs still
needs to reference the same jar files as when they were created.
Today, Hive's install_jars.py and params.py route
/usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz
Instead, it should preserve the version number when copying to HDFS.
Further, when tez.py generates the tez-site.xml file, it should replace the
value of the tez.lib.uris parameter to include the rpm version number in the
value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz
It does 2 things:
1. Copies files from the local file system to HDFS using a predefined directory
structure
2. Provides a way of doing dynamic variable interpretation in *-site.xml files
This will break the Download Client Configs, which calculates the dynamic
properties on the agent-side.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
PRE-CREATION
ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py
453c546
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
34f2d96
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
08a0a50
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
5834b89
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py
b034cbc
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
78d2ea2
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
fe4324c
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml
7370dd2
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
6a793bf
Diff: https://reviews.apache.org/r/26858/diff/
Testing
-------
Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps,
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env tez_tar_source "/usr/hdp/current/tez-client/lib/tez-{{
component_version }}.{{ hdp_stack_version }}.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version
}}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev tez-site
tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version
}}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz"
2. Verified properties were saved
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env
http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site
3. Copy changed files
yes | cp
/vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py
/usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py
yes | cp
/vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
/usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py
rm
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py
4. Check that tarballs are not already in HDFS. If they are, delete them.
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls
/hdp/apps/2.2.0.0-913/tez/
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r
/hdp/apps/2.2.0.0-913/tez/
5. Restart Hive service and verify file is copied to HDFS,
[hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls
/hdp/apps/2.2.0.0-913/tez/
6. Before running Tez script, check that it still has the dynamic property
without the actual values,
less /etc/tez/conf/tez-site.xml
/ tez.lib.uris
7. Run tez script manually, which generates tez-site.xml during its install
Need to first find the latest command file.
ls -ltr /var/lib/ambari-agent/data/command-*.json
Get the digits from the last command-*.json file
python
/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py
CONFIGURE /var/lib/ambari-agent/data/command-69.json
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS
/var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp
Check the /etc/tez/conf/tez-site.xml file again. This time it should have,
<name>tez.lib.uris</name>
<value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value>
8. Ensure Hive runs with tez-execution engine
[root@c6404 ~]# su - hive
[hive@c6404 ~]$ hive
hive> select name FROM school order by name asc limit 10;
*Unit tests currently fail*
Thanks,
Alejandro Fernandez
