----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26858/#review57460 -----------------------------------------------------------
Ship it! Ship It! - Sid Wagle On Oct. 20, 2014, 10:26 p.m., Alejandro Fernandez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26858/ > ----------------------------------------------------------- > > (Updated Oct. 20, 2014, 10:26 p.m.) > > > Review request for Ambari, Dmytro Sen, Jonathan Hurley, Mahadev Konar, Nate > Cole, Sumit Mohanty, Srimanth Gunturi, and Sid Wagle. > > > Bugs: AMBARI-7825 > https://issues.apache.org/jira/browse/AMBARI-7825 > > > Repository: ambari > > > Description > ------- > > HDP 2.2 using versioned RPMs > When Ambari installs Hive, it copies any file that matches > /usr/hdp/current/tez-client/lib/tez*.tar.gz file to > hdfs:///apps/tez/tez.tar.gz > (Aside: it also ends up copying other tez-* files that it shouldn't because > they are not needed, and already zipped in the tar.gz file) > The problem is that during a rolling upgrade, currently running jobs still > needs to reference the same jar files as when they were created. > Today, Hive's install_jars.py and params.py route > /usr/hdp/current/tez-client/lib/tez*.tar.gz => tez.tar.gz > > Instead, it should preserve the version number when copying to HDFS. > Further, when tez.py generates the tez-site.xml file, it should replace the > value of the tez.lib.uris parameter to include the rpm version number in the > value, e.g., hdfs:////apps/tez/tez-0.6.0.2.2.0.0-887.tar.gz > > It does 2 things: > 1. Copies files from the local file system to HDFS using a predefined > directory structure > 2. Provides a way of doing dynamic variable interpretation in *-site.xml files > > This will break the Download Client Configs, which calculates the dynamic > properties on the agent-side. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py > PRE-CREATION > > ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py > 453c546 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py > 34f2d96 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py > 08a0a50 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py > 5834b89 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py > b034cbc > > ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py > 78d2ea2 > > ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py > fe4324c > > ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml > 7370dd2 > > ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml > 6a793bf > ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py ed759ac > ambari-server/src/test/python/stacks/2.2/SLIDER/test_slider_client.py > 746f346 > ambari-server/src/test/python/stacks/2.2/configs/default.json 4c2ada9 > > Diff: https://reviews.apache.org/r/26858/diff/ > > > Testing > ------- > > Deployed an Ambari 1.7.0 cluster with HDP, then ran the following steps, > > > 1. Set properties > /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev > cluster-env tez_tar_source > "/usr/hdp/current/tez-client/lib/tez-{{ component_version }}.{{ > hdp_stack_version }}.tar.gz" > /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev > cluster-env tez_tar_destination_folder "hdfs:///hdp/apps/{{ hdp_stack_version > }}/tez/" > /var/lib/ambari-server/resources/scripts/configs.sh set localhost dev > tez-site tez.lib.uris "hdfs:///hdp/apps/{{ hdp_stack_version > }}/tez/tez-{{ component_version }}.{{ hdp_stack_version }}.tar.gz" > > 2. Verified properties were saved > http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env > http://c6404.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=tez-site > > 3. Copy changed files > yes | cp > /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/script/config_dictionary.py > > /usr/lib/ambari-server/lib/resource_management/libraries/script/config_dictionary.py > yes | cp > /vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py > > /usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py > yes | cp > /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py > > /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py > yes | cp > /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py > > /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/params.py > yes | cp > /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py > > /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez.py > yes | cp > /vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/package/scripts/params.py > > /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/params.py > rm > /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/install_jars.py > > 4. Check that tarballs are not already in HDFS. If they are, delete them. > [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls > /hdp/apps/2.2.0.0-913/tez/ > [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r > /hdp/apps/2.2.0.0-913/tez/ > > 5. Restart Hive service and verify file is copied to HDFS, > [hdfs@c6404 ~]$ hdfs --config /etc/hadoop/conf dfs -ls > /hdp/apps/2.2.0.0-913/tez/ > > 6. Before running Tez script, check that it still has the dynamic property > without the actual values, > less /etc/tez/conf/tez-site.xml > / tez.lib.uris > > 7. Run tez script manually, which generates tez-site.xml during its install > Need to first find the latest command file. > ls -ltr /var/lib/ambari-agent/data/command-*.json > Get the digits from the last command-*.json file > python > /var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py > CONFIGURE /var/lib/ambari-agent/data/command-69.json > /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS > /var/lib/ambari-agent/data/output-69.txt DEBUG /var/lib/ambari-agent/data/tmp > > Check the /etc/tez/conf/tez-site.xml file again. This time it should have, > <name>tez.lib.uris</name> > <value>hdfs:///hdp/apps/2.2.0.0-913/tez/tez-0.6.0.2.2.0.0-913.tar.gz</value> > > 8. Ensure Hive runs with tez-execution engine > [root@c6404 ~]# su - hive > [hive@c6404 ~]$ hive > hive> select name FROM school order by name asc limit 10; > > *Unit tests passed* > > > Thanks, > > Alejandro Fernandez > >
