-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31815/
-----------------------------------------------------------
(Updated March 9, 2015, 6:23 p.m.)
Review request for Ambari, Gautam Borad, Jonathan Hurley, Nate Cole, Srimanth
Gunturi, and Sid Wagle.
Changes
-------
Removed import * and minimized spark-job-history-server.json
Bugs: AMBARI-9954
https://issues.apache.org/jira/browse/AMBARI-9954
Repository: ambari
Description
-------
The spark on tez apps fails because tez.tar.gz needs to be copied to HDFS.
Currently, only Pig Service Check and Hive START copy it to HDFS.
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/metainfo.xml
ce8ad7a
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/job_history_server.py
518352f
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py
PRE-CREATION
ambari-server/src/test/python/stacks/2.2/configs/spark-job-history-server.json
PRE-CREATION
Diff: https://reviews.apache.org/r/31815/diff/
Testing
-------
I added a unit test,
----------------------------------------------------------------------
Total run:610
Total errors:0
Total failures:0
OK
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [6.420s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.032s]
[INFO] Ambari Web ........................................ SUCCESS [41.221s]
[INFO] Ambari Views ...................................... SUCCESS [2.345s]
[INFO] Ambari Admin View ................................. SUCCESS [10.019s]
[INFO] Ambari Metrics Common ............................. SUCCESS [1.446s]
[INFO] Ambari Server ..................................... SUCCESS [33:24.375s]
[INFO] Ambari Agent ...................................... SUCCESS [16.789s]
[INFO] Ambari Client ..................................... SUCCESS [0.038s]
[INFO] Ambari Python Client .............................. SUCCESS [0.257s]
[INFO] Ambari Groovy Client .............................. SUCCESS [9.922s]
[INFO] Ambari Shell ...................................... SUCCESS [0.033s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.036s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [6.632s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35:01.956s
[INFO] Finished at: Fri Mar 06 16:16:31 PST 2015
[INFO] Final Memory: 69M/565M
[INFO] ------------------------------------------------------------------------
I installed Spark and Tez at the same time, and put the Spark History Server on
host 1, and Tez Client on host 2. The installation for Spark History Server has
a dependency on Tez Client, so it also installed it on host 1. The Spark
History Server START command then copied the tarball to
/hdp/apps/${version}/tez/tez.tar.gz , and it will also copy it during a Rolling
Upgrade restart.
```
[root@c6408 ~]# sudo su hdfs -c 'hadoop fs -ls /hdp/apps/2.2.2.0-2538/tez/'
Found 1 items
-r--r--r-- 3 hdfs hadoop 40656789 2015-03-06 02:58
/hdp/apps/2.2.2.0-2538/tez/tez.tar.gz
[root@c6408 ~]# su - spark
[spark@c6408 ~]$ /usr/hdp/current/spark-client/bin/spark-submit --class
org.apache.spark.examples.SparkPi --master
execution-context:org.apache.spark.tez.TezJobExecutionContext
/usr/hdp/current/spark-client/lib/spark-examples-1.2.1.2.2.2.0-2538-hadoop2.6.0.2.2.2.0-2538.jar
3
[spark@c6408 ~]$ echo $?
0
```
Thanks,
Alejandro Fernandez
