Hi, Thank you Phil for updating my script to support the latest version of EMR. I have edited my gist so that it includes some of your updates plus added some other additional changes.
https://gist.github.com/andershammar/224e1077021d0ea376dd While on the subject, has anyone be able to get Zeppelin to work together with the Amazon's Spark installation on Amazon EMR 4.x (by exporting SPARK_HOME and HADOOP_HOME instead)? When I try this then I get the following exception: org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former. at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$8.apply(SparkConf.scala:444) at org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$8.apply(SparkConf.scala:442) at scala.collection.immutable.List.foreach(List.scala:318) at org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:442) at org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:430) at scala.Option.foreach(Option.scala:236) at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:430) ... >From a quick look at it, the problem seems to be that the Amazon installation of Spark use SPARK_CLASSPATH to add additional libraries (/etc/spark/conf/spark-env.sh) while the Zeppelin use "spark-submit --driver-class-path" (zeppelin/bin/interpreter.sh). Any ideas? Best regards, Anders On Wed, Sep 9, 2015 at 5:09 PM, Eugene <blackorange...@gmail.com> wrote: > Here's a bit shorter alternative, too > > https://gist.github.com/snowindy/008f3e8b878a23c00679 > > 2015-09-09 18:58 GMT+04:00 shahab <shahab.mok...@gmail.com>: > >> Thanks Phil, it works. Great job and well done! >> >> best, >> /Shahab >> >> On Mon, Sep 7, 2015 at 6:32 PM, Phil Wills <otherp...@gmail.com> wrote: >> >>> Anders script is a bit out of date if you're using the latest version of >>> EMR. Here's my fork: >>> >>> https://gist.github.com/philwills/71539f833f57338236b5 >>> >>> which worked OK for me fairly recently. >>> >>> Phil >>> >>> On Mon, 7 Sep 2015 at 10:01 shahab <shahab.mok...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am trying to use Zeppelin to work with Spark on Amazon EMR. I used >>>> the script provided by Anders ( >>>> https://gist.github.com/andershammar/224e1077021d0ea376dd) to setup >>>> Zeppelin. The Zeppelin can connect to Spark but when I got error when I run >>>> the tutorials. and I get the following error: >>>> >>>> ...FileNotFoundException: File >>>> file:/home/hadoop/zeppelin/interpreter/spark/dep/zeppelin-spark-dependencies-0.6.0-incubating-SNAPSHOT.jar >>>> does not exist >>>> >>>> However, the above file does exists in that path on the Master node.' >>>> >>>> I do appreciate if anyone has any experience to share how to setup >>>> Zeppelin with EMR . >>>> >>>> best, >>>> /Shahab >>>> >>>> >> > > > -- > > > Best regards, > Eugene. >
