-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27311/
-----------------------------------------------------------
(Updated Oct. 28, 2014, 11:56 p.m.)
Review request for Ambari, Dmytro Sen, Mahadev Konar, Raja Aluri, Sumit
Mohanty, Srimanth Gunturi, Sid Wagle, Vinod Kumar Vavilapalli, and Yusaku Sako.
Bugs: AMBARI-7842
https://issues.apache.org/jira/browse/AMBARI-7842
Repository: ambari
Description
-------
With HDP 2.2, Ambari needs to copy the tarballs/jars from the local file system
to a certain location in HDFS.
The tarballs/jars no longer have a version number (either component version or
HDP stack version + build) in the name), but the destination folder in HDFS
does contain the HDP Version (e.g., 2.2.0.0-999).
```
/hdp/apps/$(hdp-stack-version)
|---- mapreduce/mapreduce.tar.gz
|---- mapreduce/hadoop-streaming.jar (which is needed by WebHcat. In the file
system, it is a symlink to a versioned file, so HDFS needs to follow the link)
|---- tez/tez.tar.gz
|---- pig/pig.tar.gz
|---- hive/hive.tar.gz
|---- sqoop/sqoop.tar.gz
```
Furthermore, the folders created in HDFS need to have a permission of 0555,
while files need 0444.
The owner should be hdfs, and the group should be hadoop.
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
728620e
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
5e2000d
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py
62d37a8
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/package/scripts/historyserver.py
29692fc
ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml
da15055
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/configuration/webhcat-site.xml
5182d82
ambari-server/src/main/resources/stacks/HDP/2.2/services/TEZ/configuration/tez-site.xml
9bac52a
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration-mapred/mapred-site.xml
10b621f
ambari-server/src/main/resources/stacks/HDP/2.2/services/YARN/configuration/yarn-site.xml
6ba2c95
ambari-server/src/test/python/stacks/2.2/configs/default.json 888b0ca
ambari-server/src/test/python/stacks/2.2/configs/secured.json 7607a5d
ambari-web/app/data/HDP2/site_properties.js a13e94a
Diff: https://reviews.apache.org/r/27311/diff/
Testing (updated)
-------
mvn clean test
INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24:34.850s
[INFO] Finished at: Tue Oct 28 15:52:11 PDT 2014
[INFO] Final Memory: 64M/618M
[INFO] ------------------------------------------------------------------------
Ran a system test, and service checks for MR, Yarn, Pig, Hive, and Oozie passed.
1. Set properties
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env tez_tar_source
"/usr/hdp/current/tez-client/lib/tez.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env tez_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/tez/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env hive_tar_source
"/usr/hdp/current/hive-client/hive.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env hive_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/hive/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env pig_tar_source
"/usr/hdp/current/pig-client/pig.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env pig_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/pig/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env hadoop-streaming_tar_source
"/usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env hadoop-streaming_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/mapreduce/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env sqoop_tar_source
"/usr/hdp/current/sqoop-client/sqoop.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env sqoop_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/sqoop/"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env mapreduce_tar_source
"/usr/hdp/current/hadoop-client/mapreduce.tar.gz"
/var/lib/ambari-server/resources/scripts/configs.sh set localhost dev
cluster-env mapreduce_tar_destination_folder
"hdfs:///hdp/apps/{{ hdp_stack_version }}/mapreduce/"
Verify properties were saved
http://c6408.ambari.apache.org:8080/api/v1/clusters/dev/configurations?type=cluster-env&tag=version1414527575195546269
2. Save changed properties in tez-site, mapred-site, yarn-site and webhcat-site
because they need to be set through UI.
3. Set web server symlink to pick up changes to site_properties.js
4. Copy changed files
yes | cp
/vagrant/ambari/ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
/usr/lib/ambari-server/lib/resource_management/libraries/functions/dynamic_variable_interpretation.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/hive_server.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HIVE/package/scripts/webhcat.py
yes | cp
/vagrant/ambari/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/package/scripts/historyserver.py
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/YARN/package/scripts/historyserver.py
5. Delete existing tarballs in HDFS, and prepare new tarballs without the
version number
[hdfs@c6408 ~]$ hdfs --config /etc/hadoop/conf dfs -rm -r
/hdp/apps/2.2.0.0-1114/
[root@c6408 ~]# cd /usr/hdp/current
[root@c6408 current]# cp tez-client/lib/tez-*.tar.gz tez-client/lib/tez.tar.gz
[root@c6408 current]# cp hive-client/hive-*.tar.gz hive-client/hive.tar.gz
[root@c6408 current]# cp pig-client/pig-*.tar.gz pig-client/pig.tar.gz
[root@c6408 current]# cp sqoop-client/sqoop-*.tar.gz sqoop-client/sqoop.tar.gz
[root@c6408 current]# cp hadoop-client/mr-*.tar.gz
hadoop-client/mapreduce.tar.gz
6. Restart services, MR, YARN, HIVE (including WebHcat), OOZIE
Re-run tez-client (in order to copy tarball),
python
/var/lib/ambari-agent/cache/stacks/HDP/2.1/services/TEZ/package/scripts/tez_client.py
CONFIGURE /var/lib/ambari-agent/data/command-169.json
/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/HDFS
/var/lib/ambari-agent/data/output-169.txt DEBUG /var/lib/ambari-agent/data/tmp
Check if files were copied,
[hdfs@c6408 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-1114/
mapreduce/mapreduce.tar.gz and hadoop-streaming.jar
hive
tez
pig
sqoop
7. Run Service Checks, MR, YARN, TEZ, Hive, Oozie
8. Check file and folder permissions, "444" = "-r--r--r--", "555" = "dr-xr-xr-x"
[hdfs@c6408 ~]$ hdfs --config /etc/hadoop/conf dfs -ls /hdp/apps/2.2.0.0-1114/
Found 5 items
dr-xr-xr-x - hdfs hdfs 0 2014-10-28 21:08 /hdp/apps/2.2.0.0-1114/hive
dr-xr-xr-x - hdfs hdfs 0 2014-10-28 21:02
/hdp/apps/2.2.0.0-1114/mapreduce
dr-xr-xr-x - hdfs hdfs 0 2014-10-28 21:09 /hdp/apps/2.2.0.0-1114/pig
dr-xr-xr-x - hdfs hdfs 0 2014-10-28 21:02
/hdp/apps/2.2.0.0-1114/sqoop
dr-xr-xr-x - hdfs hdfs 0 2014-10-28 21:00 /hdp/apps/2.2.0.0-1114/tez
[hdfs@c6408 ~]$ hdfs --config /etc/hadoop/conf dfs -ls
/hdp/apps/2.2.0.0-1114/mapreduce/
Found 2 items
-r--r--r-- 3 hdfs hadoop 104936 2014-10-28 21:02
/hdp/apps/2.2.0.0-1114/mapreduce/hadoop-streaming.jar
-r--r--r-- 3 hdfs hadoop 183280502 2014-10-28 20:51
/hdp/apps/2.2.0.0-1114/mapreduce/mapreduce.tar.gz
Thanks,
Alejandro Fernandez
