[
https://issues.apache.org/jira/browse/BIGTOP-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311897#comment-14311897
]
YoungWoo Kim commented on BIGTOP-1648:
--------------------------------------
Steps to run and test spark-history-server:
1) Create a log directory on HDFS:
{noformat}
su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir -p /var/log/spark/apps'
su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 1777 /var/log/spark/apps'
su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown spark:spark
/var/log/spark/apps'
{noformat}
2) Create '/etc/spark/spark-default.conf'
{noformat}
cd /etc/spark/conf
cp spark-default.conf.template spark-default.conf
{noformat}
Edit the spark-default.conf:
{noformat}
spark.master spark://HOSTNAME:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://HOSTNAME:8020/var/log/spark/apps/
{noformat}
3) Run spark examples with yarn:
{noformat}
spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode client
--master yarn /usr/lib/spark/lib/spark-examples_2.10-1.2.0.jar 2
{noformat}
4) Browse the spark history server:
http://HOSTNAME:18082/
> Update to Spark 1x to Spark 1.2
> -------------------------------
>
> Key: BIGTOP-1648
> URL: https://issues.apache.org/jira/browse/BIGTOP-1648
> Project: Bigtop
> Issue Type: Task
> Components: spark
> Affects Versions: 0.8.0
> Reporter: jay vyas
> Fix For: 1.0.0
>
> Attachments: BIGTOP-1648.1.patch
>
>
> Time to update spark ? Looks like 1.2 is out. I don't see much value in
> lagging behind . Tons of improvements in 1.2, for pyspark, machine
> learning, and performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
