Re: Integrate Spark Editor with Hue for source compiled installation of spark/spark-jobServer
That's good to know. I will try it out. Thanks Romain On Friday, June 27, 2014, Romain Rigaux romain.rig...@gmail.com wrote: So far Spark Job Server does not work with Spark 1.0: https://github.com/ooyala/spark-jobserver So this works only with Spark 0.9 currently: http://gethue.com/get-started-with-spark-deploy-spark-server-and-compute-pi-from-your-web-browser/ Romain Romain On Tue, Jun 24, 2014 at 9:04 AM, Sunita Arvind sunitarv...@gmail.com javascript:_e(%7B%7D,'cvml','sunitarv...@gmail.com'); wrote: Hello Experts, I am attempting to integrate Spark Editor with Hue on CDH5.0.1. I have the spark installation build manually from the sources for spark1.0.0. I am able to integrate this with cloudera manager. Background: --- We have a 3 node VM cluster with CDH5.0.1 We requried spark1.0.0 due to some features in it, so I did a yum remove spark-core spark-master spark-worker spark-python of the default spark0.9.0 and compiled spark1.0.0 from source: Downloaded the spark-trunk from git clone https://github.com/apache/spark.git cd spark SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true ./sbt/sbt assembly The spark-assembly-1.0.0-SNAPSHOT-hadoop2.2.0.jar was built and spark by itself seems to work well. I was even able to run a text file count. Current attempt: Referring to this article - http://gethue.com/a-new-spark-web-ui-spark-app/ Now I am trying to add the Spark editor to Hue. AFAIK, this requires git clone https://github.com/ooyala/spark-jobserver.git cd spark-jobserver sbt re-start This was successful after lot of struggle with the proxy settings. However, is this the job Server itself? Will that mean the job Server has to be manually started. I intend to have the spark editor show up in hue web UI and I am no way close. Can some one please help? Note, the 3 VMs are Linux CentOS. Not sure if setting something like can be expected to work.: [desktop] app_blacklist= Also, I have made the changes to vim . /job-server/src/main/resources/application.conf as recommended, however, I do not expect this to impact hue in any way. Also, I intend to let the editor stay available, not spawn it everytime it is required. Thanks in advance. regards
Re: Integrate Spark Editor with Hue for source compiled installation of spark/spark-jobServer
So far Spark Job Server does not work with Spark 1.0: https://github.com/ooyala/spark-jobserver So this works only with Spark 0.9 currently: http://gethue.com/get-started-with-spark-deploy-spark-server-and-compute-pi-from-your-web-browser/ Romain Romain On Tue, Jun 24, 2014 at 9:04 AM, Sunita Arvind sunitarv...@gmail.com wrote: Hello Experts, I am attempting to integrate Spark Editor with Hue on CDH5.0.1. I have the spark installation build manually from the sources for spark1.0.0. I am able to integrate this with cloudera manager. Background: --- We have a 3 node VM cluster with CDH5.0.1 We requried spark1.0.0 due to some features in it, so I did a yum remove spark-core spark-master spark-worker spark-python of the default spark0.9.0 and compiled spark1.0.0 from source: Downloaded the spark-trunk from git clone https://github.com/apache/spark.git cd spark SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true ./sbt/sbt assembly The spark-assembly-1.0.0-SNAPSHOT-hadoop2.2.0.jar was built and spark by itself seems to work well. I was even able to run a text file count. Current attempt: Referring to this article - http://gethue.com/a-new-spark-web-ui-spark-app/ Now I am trying to add the Spark editor to Hue. AFAIK, this requires git clone https://github.com/ooyala/spark-jobserver.git cd spark-jobserver sbt re-start This was successful after lot of struggle with the proxy settings. However, is this the job Server itself? Will that mean the job Server has to be manually started. I intend to have the spark editor show up in hue web UI and I am no way close. Can some one please help? Note, the 3 VMs are Linux CentOS. Not sure if setting something like can be expected to work.: [desktop] app_blacklist= Also, I have made the changes to vim . /job-server/src/main/resources/application.conf as recommended, however, I do not expect this to impact hue in any way. Also, I intend to let the editor stay available, not spawn it everytime it is required. Thanks in advance. regards
