Re: Cloudera Spark 2.2
Ruslan, Quick observation. I see that you put Scala 2.10. How come you didn't use Scala 2.11? Just curious. Cheers, Ben On Fri, Aug 4, 2017 at 8:36 AM Ruslan Dautkhanov wrote: > This should do: > > >> export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >> export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive >> export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop >> export HADOOP_CONF_DIR=/etc/hadoop/conf >> export HIVE_CONF_DIR=/etc/hive/conf > > > >> >> mvn clean package -DskipTests -Pspark-2.1 >> -Dhadoop.version=2.6.0-cdh5.10.1 -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 >> -Psparkr -pl '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e > > > You may needs additional steps depending which interpreters you use (like > R etc). > > > -- > Ruslan Dautkhanov > > On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim wrote: > >> Hi Ruslan, >> >> Can you send me the steps you used to build it, especially the Maven >> command with the arguments? I will try to build it also. >> >> I do believe that the binaries are for official releases. >> >> Cheers, >> Ben >> >> >> On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov >> wrote: >> >>> It was built. I think binaries are only available for official releases? >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim wrote: >>> Did you build Zeppelin or download the binary? On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov wrote: > We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. > > Yes, we have that spark home in zeppelin-env.sh > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim > wrote: > >> Does this work with Zeppelin 0.7.1? We an error when setting >> SPARK_HOME in zeppelin-env.sh to what you have below. >> >> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov < >> dautkha...@gmail.com> wrote: >> >>> You don't have to use spark2-shell and spark2-submit to use Spark 2. >>> That can be controled by setting SPARK_HOME using regular >>> spark-submit/spark-shell. >>> >>> $ which spark-submit >>> /usr/bin/spark-submit >>> $ which spark-shell >>> /usr/bin/spark-shell >>> >>> $ spark-shell >>> Welcome to >>> __ >>> / __/__ ___ _/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>>/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >>> /_/ >>> >>> >>> >>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >>> >>> $ spark-shell >>> Welcome to >>> __ >>> / __/__ ___ _/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>>/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >>> /_/ >>> >>> >>> spark-submit and spark-shell are just shell script wrappers. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim >>> wrote: >>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin. Cheers, Ben On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov < dautkha...@gmail.com> wrote: > Might need to recompile Zeppelin with Scala 2.11? > Also Spark 2.2 now requires JDK8 I believe. > > > > -- > Ruslan Dautkhanov > > On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim > wrote: > >> Here is more. >> >> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >> User-defined SPARK_HOME >> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >> WARNING: Running spark-class from user-defined location. >> Exception in thread "main" java.lang.NoSuchMethodError: >> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >> at >> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at scala.Option.getOrElse(Option.scala:12
Re: Cloudera Spark 2.2
This should do: > export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive > export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop > export HADOOP_CONF_DIR=/etc/hadoop/conf > export HIVE_CONF_DIR=/etc/hive/conf > > mvn clean package -DskipTests -Pspark-2.1 -Dhadoop.version=2.6.0-cdh5.10.1 > -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 -Psparkr -pl > '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e You may needs additional steps depending which interpreters you use (like R etc). -- Ruslan Dautkhanov On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim wrote: > Hi Ruslan, > > Can you send me the steps you used to build it, especially the Maven > command with the arguments? I will try to build it also. > > I do believe that the binaries are for official releases. > > Cheers, > Ben > > > On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov > wrote: > >> It was built. I think binaries are only available for official releases? >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim wrote: >> >>> Did you build Zeppelin or download the binary? >>> >>> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov >>> wrote: >>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. Yes, we have that spark home in zeppelin-env.sh -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: > Does this work with Zeppelin 0.7.1? We an error when setting > SPARK_HOME in zeppelin-env.sh to what you have below. > > On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov > wrote: > >> You don't have to use spark2-shell and spark2-submit to use Spark 2. >> That can be controled by setting SPARK_HOME using regular >> spark-submit/spark-shell. >> >> $ which spark-submit >> /usr/bin/spark-submit >> $ which spark-shell >> /usr/bin/spark-shell >> >> $ spark-shell >> Welcome to >> __ >> / __/__ ___ _/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >>/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >> /_/ >> >> >> >> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >> >> $ spark-shell >> Welcome to >> __ >> / __/__ ___ _/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >>/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >> /_/ >> >> >> spark-submit and spark-shell are just shell script wrappers. >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim >> wrote: >> >>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports >>> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from >>> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does >>> the >>> calls to spark-shell and spark-submit. To force the use of Spark 2.x, >>> the >>> calls need to be spark2-shell and spark2-submit. I wonder if this is >>> causing the problem. By the way, we are using Java8 corporate wide, and >>> there seems to be no problems using Zeppelin. >>> >>> Cheers, >>> Ben >>> >>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov < >>> dautkha...@gmail.com> wrote: >>> Might need to recompile Zeppelin with Scala 2.11? Also Spark 2.2 now requires JDK8 I believe. -- Ruslan Dautkhanov On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: > Here is more. > > org.apache.zeppelin.interpreter.InterpreterException: WARNING: > User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- > 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides > detected (/opt/cloudera/parcels/SPARK2/lib/spark2). > WARNING: Running spark-class from user-defined location. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; > at org.apache.spark.util.Utils$.getDefaultPropertiesFile( > Utils.scala:2103) > at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ > mergeDefaultSparkProperties$1.apply(SparkSubmitArguments. > scala:124) > at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ > mergeDefaultSparkProperties$1.apply(SparkSubmitArguments. > scala:124) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.deploy.SparkSubmitArguments. > mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) > at org.apache.spark.deploy.SparkSubmitArguments.( > SparkSubmitArguments.scala:110) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > scala:112) > at org.
Re: Cloudera Spark 2.2
Hi Ruslan, Can you send me the steps you used to build it, especially the Maven command with the arguments? I will try to build it also. I do believe that the binaries are for official releases. Cheers, Ben On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov wrote: > It was built. I think binaries are only available for official releases? > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim wrote: > >> Did you build Zeppelin or download the binary? >> >> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov >> wrote: >> >>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. >>> >>> Yes, we have that spark home in zeppelin-env.sh >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: >>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below. On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov wrote: > You don't have to use spark2-shell and spark2-submit to use Spark 2. > That can be controled by setting SPARK_HOME using regular > spark-submit/spark-shell. > > $ which spark-submit > /usr/bin/spark-submit > $ which spark-shell > /usr/bin/spark-shell > > $ spark-shell > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 > /_/ > > > > $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > > $ spark-shell > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 > /_/ > > > spark-submit and spark-shell are just shell script wrappers. > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim > wrote: > >> According to the Zeppelin documentation, Zeppelin 0.7.1 supports >> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from >> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the >> calls to spark-shell and spark-submit. To force the use of Spark 2.x, the >> calls need to be spark2-shell and spark2-submit. I wonder if this is >> causing the problem. By the way, we are using Java8 corporate wide, and >> there seems to be no problems using Zeppelin. >> >> Cheers, >> Ben >> >> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov < >> dautkha...@gmail.com> wrote: >> >>> Might need to recompile Zeppelin with Scala 2.11? >>> Also Spark 2.2 now requires JDK8 I believe. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim >>> wrote: >>> Here is more. org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). WARNING: Running spark-class from user-defined location. Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Cheers, Ben On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: > > Then it is due to some classpath issue. I am not sure familiar > with CDH, please check whether spark of CDH include hadoop jar with > it. > > > Benjamin Kim 于2017年8月2日周三 上午8:22写道: > >> Here is the error that was sent to me. >> >> org.apache.zeppelin.interpreter.InterpreterException: Exception >> in thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/fs/FSDataInputStream >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.fs.FSDataInputStream >> >> Cheers, >> Ben >>
Re: Cloudera Spark 2.2
It was built. I think binaries are only available for official releases? -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim wrote: > Did you build Zeppelin or download the binary? > > On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov > wrote: > >> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. >> >> Yes, we have that spark home in zeppelin-env.sh >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: >> >>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME >>> in zeppelin-env.sh to what you have below. >>> >>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov >>> wrote: >>> You don't have to use spark2-shell and spark2-submit to use Spark 2. That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell. $ which spark-submit /usr/bin/spark-submit $ which spark-shell /usr/bin/spark-shell $ spark-shell Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 $ spark-shell Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 /_/ spark-submit and spark-shell are just shell script wrappers. -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim wrote: > According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark > 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. > For some reason, Cloudera defaults to Spark 1.6 and so does the calls to > spark-shell and spark-submit. To force the use of Spark 2.x, the calls > need > to be spark2-shell and spark2-submit. I wonder if this is causing the > problem. By the way, we are using Java8 corporate wide, and there seems to > be no problems using Zeppelin. > > Cheers, > Ben > > On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov > wrote: > >> Might need to recompile Zeppelin with Scala 2.11? >> Also Spark 2.2 now requires JDK8 I believe. >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim >> wrote: >> >>> Here is more. >>> >>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- >>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides >>> detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >>> WARNING: Running spark-class from user-defined location. >>> Exception in thread "main" java.lang.NoSuchMethodError: >>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile( >>> Utils.scala:2103) >>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>> at scala.Option.getOrElse(Option.scala:120) >>> at org.apache.spark.deploy.SparkSubmitArguments. >>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>> at org.apache.spark.deploy.SparkSubmitArguments.( >>> SparkSubmitArguments.scala:110) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> Cheers, >>> Ben >>> >>> >>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >>> Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. Benjamin Kim 于2017年8月2日周三 上午8:22写道: > Here is the error that was sent to me. > > org.apache.zeppelin.interpreter.InterpreterException: Exception > in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/fs/FSDataInputStream > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs. > FSDataInputStream > > Cheers, > Ben > > > On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang > wrote: > >> >> By default, 0.7.1 doesn't support spark 2.2. But you can set >> zeppelin.spark.enableSupportedVersionCheck in interpreter >> setting to disable the supported version check. >> >> >> Jeff Zhang 于2017年8月2日周三 上午8:18写道: >> >>> >>> What's the error you see in
Re: Cloudera Spark 2.2
Did you build Zeppelin or download the binary? On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov wrote: > We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. > > Yes, we have that spark home in zeppelin-env.sh > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: > >> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME >> in zeppelin-env.sh to what you have below. >> >> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov >> wrote: >> >>> You don't have to use spark2-shell and spark2-submit to use Spark 2. >>> That can be controled by setting SPARK_HOME using regular >>> spark-submit/spark-shell. >>> >>> $ which spark-submit >>> /usr/bin/spark-submit >>> $ which spark-shell >>> /usr/bin/spark-shell >>> >>> $ spark-shell >>> Welcome to >>> __ >>> / __/__ ___ _/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>>/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >>> /_/ >>> >>> >>> >>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >>> >>> $ spark-shell >>> Welcome to >>> __ >>> / __/__ ___ _/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>>/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >>> /_/ >>> >>> >>> spark-submit and spark-shell are just shell script wrappers. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim >>> wrote: >>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin. Cheers, Ben On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov wrote: > Might need to recompile Zeppelin with Scala 2.11? > Also Spark 2.2 now requires JDK8 I believe. > > > > -- > Ruslan Dautkhanov > > On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim > wrote: > >> Here is more. >> >> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >> User-defined SPARK_HOME >> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >> WARNING: Running spark-class from user-defined location. >> Exception in thread "main" java.lang.NoSuchMethodError: >> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >> at >> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at scala.Option.getOrElse(Option.scala:120) >> at >> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >> >>> >>> Then it is due to some classpath issue. I am not sure familiar with >>> CDH, please check whether spark of CDH include hadoop jar with it. >>> >>> >>> Benjamin Kim 于2017年8月2日周三 上午8:22写道: >>> Here is the error that was sent to me. org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream Cheers, Ben On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > > By default, 0.7.1 doesn't support spark 2.2. But you can set > zeppelin.spark.enableSupportedVersionCheck > in interpreter setting to disable the supported version check. > > > Jeff Zhang 于2017年8月2日周三 上午8:18写道: > >> >> What's the error you see in log ? >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >> >>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of >>> Spark 2.2? I can't get it to work. I downloaded the binary and set >>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be
Re: Cloudera Spark 2.2
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. Yes, we have that spark home in zeppelin-env.sh -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: > Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in > zeppelin-env.sh to what you have below. > > On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov > wrote: > >> You don't have to use spark2-shell and spark2-submit to use Spark 2. >> That can be controled by setting SPARK_HOME using regular >> spark-submit/spark-shell. >> >> $ which spark-submit >> /usr/bin/spark-submit >> $ which spark-shell >> /usr/bin/spark-shell >> >> $ spark-shell >> Welcome to >> __ >> / __/__ ___ _/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >>/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >> /_/ >> >> >> >> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >> >> $ spark-shell >> Welcome to >> __ >> / __/__ ___ _/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >>/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >> /_/ >> >> >> spark-submit and spark-shell are just shell script wrappers. >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim wrote: >> >>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark >>> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. >>> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to >>> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need >>> to be spark2-shell and spark2-submit. I wonder if this is causing the >>> problem. By the way, we are using Java8 corporate wide, and there seems to >>> be no problems using Zeppelin. >>> >>> Cheers, >>> Ben >>> >>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov >>> wrote: >>> Might need to recompile Zeppelin with Scala 2.11? Also Spark 2.2 now requires JDK8 I believe. -- Ruslan Dautkhanov On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: > Here is more. > > org.apache.zeppelin.interpreter.InterpreterException: WARNING: > User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- > 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected > (/opt/cloudera/parcels/SPARK2/lib/spark2). > WARNING: Running spark-class from user-defined location. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; > at org.apache.spark.util.Utils$.getDefaultPropertiesFile( > Utils.scala:2103) > at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ > mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) > at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ > mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.deploy.SparkSubmitArguments. > mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) > at org.apache.spark.deploy.SparkSubmitArguments.( > SparkSubmitArguments.scala:110) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > Cheers, > Ben > > > On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: > >> >> Then it is due to some classpath issue. I am not sure familiar with >> CDH, please check whether spark of CDH include hadoop jar with it. >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:22写道: >> >>> Here is the error that was sent to me. >>> >>> org.apache.zeppelin.interpreter.InterpreterException: Exception in >>> thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/ >>> FSDataInputStream >>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs. >>> FSDataInputStream >>> >>> Cheers, >>> Ben >>> >>> >>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: >>> By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check. Jeff Zhang 于2017年8月2日周三 上午8:18写道: > > What's the error you see in log ? > > > Benjamin Kim 于2017年8月2日周三 上午8:18写道: > >> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of >> Spark 2.2? I can't get it to work. I downloaded the binary and set >> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be >> missing something. >> >> Cheers, >> Ben >> > >>
Re: Cloudera Spark 2.2
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below. On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov wrote: > You don't have to use spark2-shell and spark2-submit to use Spark 2. > That can be controled by setting SPARK_HOME using regular > spark-submit/spark-shell. > > $ which spark-submit > /usr/bin/spark-submit > $ which spark-shell > /usr/bin/spark-shell > > $ spark-shell > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 > /_/ > > > > $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > > $ spark-shell > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 > /_/ > > > spark-submit and spark-shell are just shell script wrappers. > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim wrote: > >> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark >> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. >> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to >> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need >> to be spark2-shell and spark2-submit. I wonder if this is causing the >> problem. By the way, we are using Java8 corporate wide, and there seems to >> be no problems using Zeppelin. >> >> Cheers, >> Ben >> >> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov >> wrote: >> >>> Might need to recompile Zeppelin with Scala 2.11? >>> Also Spark 2.2 now requires JDK8 I believe. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: >>> Here is more. org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). WARNING: Running spark-class from user-defined location. Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Cheers, Ben On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: > > Then it is due to some classpath issue. I am not sure familiar with > CDH, please check whether spark of CDH include hadoop jar with it. > > > Benjamin Kim 于2017年8月2日周三 上午8:22写道: > >> Here is the error that was sent to me. >> >> org.apache.zeppelin.interpreter.InterpreterException: Exception in >> thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/fs/FSDataInputStream >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.fs.FSDataInputStream >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: >> >>> >>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>> zeppelin.spark.enableSupportedVersionCheck >>> in interpreter setting to disable the supported version check. >>> >>> >>> Jeff Zhang 于2017年8月2日周三 上午8:18写道: >>> What's the error you see in log ? Benjamin Kim 于2017年8月2日周三 上午8:18写道: > Has anyone configured Zeppelin 0.7.1 for Cloudera's release of > Spark 2.2? I can't get it to work. I downloaded the binary and set > SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be > missing > something. > > Cheers, > Ben > >>> >
Re: Cloudera Spark 2.2
You don't have to use spark2-shell and spark2-submit to use Spark 2. That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell. $ which spark-submit /usr/bin/spark-submit $ which spark-shell /usr/bin/spark-shell $ spark-shell Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 $ spark-shell Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 /_/ spark-submit and spark-shell are just shell script wrappers. -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim wrote: > According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark > 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. > For some reason, Cloudera defaults to Spark 1.6 and so does the calls to > spark-shell and spark-submit. To force the use of Spark 2.x, the calls need > to be spark2-shell and spark2-submit. I wonder if this is causing the > problem. By the way, we are using Java8 corporate wide, and there seems to > be no problems using Zeppelin. > > Cheers, > Ben > > On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov > wrote: > >> Might need to recompile Zeppelin with Scala 2.11? >> Also Spark 2.2 now requires JDK8 I believe. >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: >> >>> Here is more. >>> >>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- >>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected >>> (/opt/cloudera/parcels/SPARK2/lib/spark2). >>> WARNING: Running spark-class from user-defined location. >>> Exception in thread "main" java.lang.NoSuchMethodError: >>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile( >>> Utils.scala:2103) >>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>> at scala.Option.getOrElse(Option.scala:120) >>> at org.apache.spark.deploy.SparkSubmitArguments. >>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>> at org.apache.spark.deploy.SparkSubmitArguments.( >>> SparkSubmitArguments.scala:110) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> Cheers, >>> Ben >>> >>> >>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >>> Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. Benjamin Kim 于2017年8月2日周三 上午8:22写道: > Here is the error that was sent to me. > > org.apache.zeppelin.interpreter.InterpreterException: Exception in > thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/ > FSDataInputStream > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs. > FSDataInputStream > > Cheers, > Ben > > > On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > >> >> By default, 0.7.1 doesn't support spark 2.2. But you can set >> zeppelin.spark.enableSupportedVersionCheck in interpreter setting to >> disable the supported version check. >> >> >> Jeff Zhang 于2017年8月2日周三 上午8:18写道: >> >>> >>> What's the error you see in log ? >>> >>> >>> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. Cheers, Ben >>> >>
Re: Cloudera Spark 2.2
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin. Cheers, Ben On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov wrote: > Might need to recompile Zeppelin with Scala 2.11? > Also Spark 2.2 now requires JDK8 I believe. > > > > -- > Ruslan Dautkhanov > > On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: > >> Here is more. >> >> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >> User-defined SPARK_HOME >> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >> WARNING: Running spark-class from user-defined location. >> Exception in thread "main" java.lang.NoSuchMethodError: >> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at scala.Option.getOrElse(Option.scala:120) >> at >> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >> >>> >>> Then it is due to some classpath issue. I am not sure familiar with CDH, >>> please check whether spark of CDH include hadoop jar with it. >>> >>> >>> Benjamin Kim 于2017年8月2日周三 上午8:22写道: >>> Here is the error that was sent to me. org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream Cheers, Ben On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > > By default, 0.7.1 doesn't support spark 2.2. But you can set > zeppelin.spark.enableSupportedVersionCheck > in interpreter setting to disable the supported version check. > > > Jeff Zhang 于2017年8月2日周三 上午8:18写道: > >> >> What's the error you see in log ? >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >> >>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark >>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME >>> to >>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. >>> >>> Cheers, >>> Ben >>> >> >
Re: Cloudera Spark 2.2
Might need to recompile Zeppelin with Scala 2.11? Also Spark 2.2 now requires JDK8 I believe. -- Ruslan Dautkhanov On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: > Here is more. > > org.apache.zeppelin.interpreter.InterpreterException: WARNING: > User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- > 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected > (/opt/cloudera/parcels/SPARK2/lib/spark2). > WARNING: Running spark-class from user-defined location. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; > at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) > at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ > mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) > at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ > mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.deploy.SparkSubmitArguments. > mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) > at org.apache.spark.deploy.SparkSubmitArguments.( > SparkSubmitArguments.scala:110) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > Cheers, > Ben > > > On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: > >> >> Then it is due to some classpath issue. I am not sure familiar with CDH, >> please check whether spark of CDH include hadoop jar with it. >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:22写道: >> >>> Here is the error that was sent to me. >>> >>> org.apache.zeppelin.interpreter.InterpreterException: Exception in >>> thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/ >>> FSDataInputStream >>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs. >>> FSDataInputStream >>> >>> Cheers, >>> Ben >>> >>> >>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: >>> By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check. Jeff Zhang 于2017年8月2日周三 上午8:18写道: > > What's the error you see in log ? > > > Benjamin Kim 于2017年8月2日周三 上午8:18写道: > >> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark >> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME >> to >> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. >> >> Cheers, >> Ben >> >
Re: Cloudera Spark 2.2
Here is more. org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). WARNING: Running spark-class from user-defined location. Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Cheers, Ben On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: > > Then it is due to some classpath issue. I am not sure familiar with CDH, > please check whether spark of CDH include hadoop jar with it. > > > Benjamin Kim 于2017年8月2日周三 上午8:22写道: > >> Here is the error that was sent to me. >> >> org.apache.zeppelin.interpreter.InterpreterException: Exception in thread >> "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/fs/FSDataInputStream >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.fs.FSDataInputStream >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: >> >>> >>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>> zeppelin.spark.enableSupportedVersionCheck >>> in interpreter setting to disable the supported version check. >>> >>> >>> Jeff Zhang 于2017年8月2日周三 上午8:18写道: >>> What's the error you see in log ? Benjamin Kim 于2017年8月2日周三 上午8:18写道: > Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark > 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to > /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. > > Cheers, > Ben >
Re: Cloudera Spark 2.2
Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. Benjamin Kim 于2017年8月2日周三 上午8:22写道: > Here is the error that was sent to me. > > org.apache.zeppelin.interpreter.InterpreterException: Exception in thread > "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/fs/FSDataInputStream > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.fs.FSDataInputStream > > Cheers, > Ben > > > On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > >> >> By default, 0.7.1 doesn't support spark 2.2. But you can set >> zeppelin.spark.enableSupportedVersionCheck >> in interpreter setting to disable the supported version check. >> >> >> Jeff Zhang 于2017年8月2日周三 上午8:18写道: >> >>> >>> What's the error you see in log ? >>> >>> >>> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. Cheers, Ben >>>
Re: Cloudera Spark 2.2
Here is the error that was sent to me. org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream Cheers, Ben On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > > By default, 0.7.1 doesn't support spark 2.2. But you can set > zeppelin.spark.enableSupportedVersionCheck > in interpreter setting to disable the supported version check. > > > Jeff Zhang 于2017年8月2日周三 上午8:18写道: > >> >> What's the error you see in log ? >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >> >>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark >>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to >>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. >>> >>> Cheers, >>> Ben >>> >>
Re: Cloudera Spark 2.2
By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check. Jeff Zhang 于2017年8月2日周三 上午8:18写道: > > What's the error you see in log ? > > > Benjamin Kim 于2017年8月2日周三 上午8:18写道: > >> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? >> I can't get it to work. I downloaded the binary and set SPARK_HOME to >> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. >> >> Cheers, >> Ben >> >
Re: Cloudera Spark 2.2
What's the error you see in log ? Benjamin Kim 于2017年8月2日周三 上午8:18写道: > Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? > I can't get it to work. I downloaded the binary and set SPARK_HOME to > /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. > > Cheers, > Ben >
Cloudera Spark 2.2
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. Cheers, Ben