Re: configure Zeppelin to use existing external hive metastore
I suppose the %sql means %spark.sql, in that case you need to modify the hive-site.xml under SPARK_CONF_DIR Richard Xin于2017年8月3日周四 上午9:21写道: > on AWS EMR > I am trying to bind zeppelin to an existing hive metastore, I modified > hive-site.xml under /etc/hive/conf to pointing to existing metastore, and I > tested with commandline hive, it works as expected. > but under zeppelin %sql is still connected to previous hive metastore, > I added line > export HIVE_CONF_DIR=/etc/hive/conf > inside zeppelin-env.sh > > > And /etc/spark/conf's spark-env.sh has following line by default: > export HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf} > > What did I miss? > > Thanks >
configure Zeppelin to use existing external hive metastore
on AWS EMRI am trying to bind zeppelin to an existing hive metastore, I modified hive-site.xml under /etc/hive/conf to pointing to existing metastore, and I tested with commandline hive, it works as expected.but under zeppelin %sql is still connected to previous hive metastore, I added line export HIVE_CONF_DIR=/etc/hive/conf inside zeppelin-env.sh And /etc/spark/conf's spark-env.sh has following line by default:export HIVE_CONF_DIR=${HIVE_CONF_DIR:-/etc/hive/conf} What did I miss? Thanks
Re: Geo Map Charting
Zeppelin support bokeh which support Geo charting. Here's some links which might be useful for you https://community.hortonworks.com/articles/109837/use-bokeh-in-apache-zeppelin.html http://bokeh.pydata.org/en/latest/docs/user_guide/geo.html Benjamin Kim于2017年8月3日周四 上午6:23写道: > Anyone every try to chart density clusters or heat maps onto a geo map of > the earth in Zeppelin? Can it be done? > > Cheers, > Ben >
Re: Cloudera Spark 2.2
It was built. I think binaries are only available for official releases? -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kimwrote: > Did you build Zeppelin or download the binary? > > On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov > wrote: > >> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. >> >> Yes, we have that spark home in zeppelin-env.sh >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: >> >>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME >>> in zeppelin-env.sh to what you have below. >>> >>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov >>> wrote: >>> You don't have to use spark2-shell and spark2-submit to use Spark 2. That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell. $ which spark-submit /usr/bin/spark-submit $ which spark-shell /usr/bin/spark-shell $ spark-shell Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 $ spark-shell Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 /_/ spark-submit and spark-shell are just shell script wrappers. -- Ruslan Dautkhanov On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim wrote: > According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark > 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. > For some reason, Cloudera defaults to Spark 1.6 and so does the calls to > spark-shell and spark-submit. To force the use of Spark 2.x, the calls > need > to be spark2-shell and spark2-submit. I wonder if this is causing the > problem. By the way, we are using Java8 corporate wide, and there seems to > be no problems using Zeppelin. > > Cheers, > Ben > > On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov > wrote: > >> Might need to recompile Zeppelin with Scala 2.11? >> Also Spark 2.2 now requires JDK8 I believe. >> >> >> >> -- >> Ruslan Dautkhanov >> >> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim >> wrote: >> >>> Here is more. >>> >>> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >>> User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2- >>> 2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides >>> detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >>> WARNING: Running spark-class from user-defined location. >>> Exception in thread "main" java.lang.NoSuchMethodError: >>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile( >>> Utils.scala:2103) >>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>> at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$ >>> mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >>> at scala.Option.getOrElse(Option.scala:120) >>> at org.apache.spark.deploy.SparkSubmitArguments. >>> mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >>> at org.apache.spark.deploy.SparkSubmitArguments.( >>> SparkSubmitArguments.scala:110) >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> Cheers, >>> Ben >>> >>> >>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >>> Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. Benjamin Kim 于2017年8月2日周三 上午8:22写道: > Here is the error that was sent to me. > > org.apache.zeppelin.interpreter.InterpreterException: Exception > in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/fs/FSDataInputStream > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs. > FSDataInputStream > > Cheers, > Ben > > > On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang > wrote: > >> >> By default, 0.7.1 doesn't support spark 2.2. But you can set >> zeppelin.spark.enableSupportedVersionCheck
Re: Cloudera Spark 2.2
Did you build Zeppelin or download the binary? On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanovwrote: > We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1. > > Yes, we have that spark home in zeppelin-env.sh > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim wrote: > >> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME >> in zeppelin-env.sh to what you have below. >> >> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov >> wrote: >> >>> You don't have to use spark2-shell and spark2-submit to use Spark 2. >>> That can be controled by setting SPARK_HOME using regular >>> spark-submit/spark-shell. >>> >>> $ which spark-submit >>> /usr/bin/spark-submit >>> $ which spark-shell >>> /usr/bin/spark-shell >>> >>> $ spark-shell >>> Welcome to >>> __ >>> / __/__ ___ _/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>>/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 >>> /_/ >>> >>> >>> >>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 >>> >>> $ spark-shell >>> Welcome to >>> __ >>> / __/__ ___ _/ /__ >>> _\ \/ _ \/ _ `/ __/ '_/ >>>/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 >>> /_/ >>> >>> >>> spark-submit and spark-shell are just shell script wrappers. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim >>> wrote: >>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin. Cheers, Ben On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov wrote: > Might need to recompile Zeppelin with Scala 2.11? > Also Spark 2.2 now requires JDK8 I believe. > > > > -- > Ruslan Dautkhanov > > On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim > wrote: > >> Here is more. >> >> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >> User-defined SPARK_HOME >> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >> WARNING: Running spark-class from user-defined location. >> Exception in thread "main" java.lang.NoSuchMethodError: >> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >> at >> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at scala.Option.getOrElse(Option.scala:120) >> at >> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >> >>> >>> Then it is due to some classpath issue. I am not sure familiar with >>> CDH, please check whether spark of CDH include hadoop jar with it. >>> >>> >>> Benjamin Kim 于2017年8月2日周三 上午8:22写道: >>> Here is the error that was sent to me. org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream Cheers, Ben On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > > By default, 0.7.1 doesn't support spark 2.2. But you can set > zeppelin.spark.enableSupportedVersionCheck > in interpreter setting to disable the supported version check. > > > Jeff Zhang 于2017年8月2日周三 上午8:18写道: > >> >> What's the error you see in log ? >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >> >>> Has
Re: Cloudera Spark 2.2
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below. On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanovwrote: > You don't have to use spark2-shell and spark2-submit to use Spark 2. > That can be controled by setting SPARK_HOME using regular > spark-submit/spark-shell. > > $ which spark-submit > /usr/bin/spark-submit > $ which spark-shell > /usr/bin/spark-shell > > $ spark-shell > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 1.6.0 > /_/ > > > > $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2 > > $ spark-shell > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 > /_/ > > > spark-submit and spark-shell are just shell script wrappers. > > > > -- > Ruslan Dautkhanov > > On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim wrote: > >> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark >> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. >> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to >> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need >> to be spark2-shell and spark2-submit. I wonder if this is causing the >> problem. By the way, we are using Java8 corporate wide, and there seems to >> be no problems using Zeppelin. >> >> Cheers, >> Ben >> >> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov >> wrote: >> >>> Might need to recompile Zeppelin with Scala 2.11? >>> Also Spark 2.2 now requires JDK8 I believe. >>> >>> >>> >>> -- >>> Ruslan Dautkhanov >>> >>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: >>> Here is more. org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). WARNING: Running spark-class from user-defined location. Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Cheers, Ben On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: > > Then it is due to some classpath issue. I am not sure familiar with > CDH, please check whether spark of CDH include hadoop jar with it. > > > Benjamin Kim 于2017年8月2日周三 上午8:22写道: > >> Here is the error that was sent to me. >> >> org.apache.zeppelin.interpreter.InterpreterException: Exception in >> thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/fs/FSDataInputStream >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.fs.FSDataInputStream >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: >> >>> >>> By default, 0.7.1 doesn't support spark 2.2. But you can set >>> zeppelin.spark.enableSupportedVersionCheck >>> in interpreter setting to disable the supported version check. >>> >>> >>> Jeff Zhang 于2017年8月2日周三 上午8:18写道: >>> What's the error you see in log ? Benjamin Kim 于2017年8月2日周三 上午8:18写道: > Has anyone configured Zeppelin 0.7.1 for Cloudera's release of > Spark 2.2? I can't get it to work. I downloaded the binary and set > SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be > missing > something. > > Cheers, > Ben > >>> >
Geo Map Charting
Anyone every try to chart density clusters or heat maps onto a geo map of the earth in Zeppelin? Can it be done? Cheers, Ben
unsubscribe
Re: Cloudera Spark 2.2
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin. Cheers, Ben On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanovwrote: > Might need to recompile Zeppelin with Scala 2.11? > Also Spark 2.2 now requires JDK8 I believe. > > > > -- > Ruslan Dautkhanov > > On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim wrote: > >> Here is more. >> >> org.apache.zeppelin.interpreter.InterpreterException: WARNING: >> User-defined SPARK_HOME >> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) >> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2). >> WARNING: Running spark-class from user-defined location. >> Exception in thread "main" java.lang.NoSuchMethodError: >> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less; >> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124) >> at scala.Option.getOrElse(Option.scala:120) >> at >> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124) >> at >> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> Cheers, >> Ben >> >> >> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang wrote: >> >>> >>> Then it is due to some classpath issue. I am not sure familiar with CDH, >>> please check whether spark of CDH include hadoop jar with it. >>> >>> >>> Benjamin Kim 于2017年8月2日周三 上午8:22写道: >>> Here is the error that was sent to me. org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream Cheers, Ben On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang wrote: > > By default, 0.7.1 doesn't support spark 2.2. But you can set > zeppelin.spark.enableSupportedVersionCheck > in interpreter setting to disable the supported version check. > > > Jeff Zhang 于2017年8月2日周三 上午8:18写道: > >> >> What's the error you see in log ? >> >> >> Benjamin Kim 于2017年8月2日周三 上午8:18写道: >> >>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark >>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME >>> to >>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something. >>> >>> Cheers, >>> Ben >>> >> >