Re: Zeppelin 0.8

[Benjamin Kim]

Can anyone tell me what the status is for 0.8 release?

> On May 2, 2018, at 4:43 PM, Jeff Zhang  wrote:
> 
> 
> Yes, 0.8 will support spark 2.3
> 
> Benjamin Kim mailto:bbuil...@gmail.com>>于2018年5月3日周四 
> 上午1:59写道：
> Will Zeppelin 0.8 have Spark 2.3 support?
> 
>> On Apr 30, 2018, at 1:27 AM, Rotem Herzberg > <mailto:rotem.herzb...@gigaspaces.com>> wrote:
>> 
>> Thanks
>> 
>> On Mon, Apr 30, 2018 at 11:16 AM, Jeff Zhang > <mailto:zjf...@gmail.com>> wrote:
>> 
>> I am preparing the RC for 0.8
>> 
>> 
>> Rotem Herzberg > <mailto:rotem.herzb...@gigaspaces.com>>于2018年4月30日周一 下午3:57写道：
>> Hi,
>> 
>> What is the release date for Zeppelin 0.8? (support for spark 2.3)
>> 
>> Thanks,
>> 
>> -- 
>>  <http://www.gigaspaces.com/?utm_source=Signature_medium=Email>  
>> Rotem Herzberg
>> SW Engineer | GigaSpaces Technologies
>>  
>> rotem.herzb...@gigaspaces.com <mailto:rotem.herzb...@gigaspaces.com>   | M 
>> +972547718880 <> 
>> 
>>   <https://twitter.com/gigaspaces>
>> <https://www.linkedin.com/company/gigaspaces>
>> <https://www.facebook.com/gigaspaces>
>> 
>> 
>> -- 
>>  <http://www.gigaspaces.com/?utm_source=Signature_medium=Email>  
>> Rotem Herzberg
>> SW Engineer | GigaSpaces Technologies
>>  
>> rotem.herzb...@gigaspaces.com <mailto:rotem.herzb...@gigaspaces.com>   | M 
>> +972547718880 <> 
>> 
>>   <https://twitter.com/gigaspaces>
>> <https://www.linkedin.com/company/gigaspaces>
>> <https://www.facebook.com/gigaspaces>

Re: Credentials for JDBC

[Benjamin Kim]

Hi 종열,

Can you show me how?

Thanks,
Ben


> On Jun 6, 2018, at 10:32 PM, Jongyoul Lee  wrote:
> 
> We have a trick to get credential information from a credential page. I'll 
> take into it.
> 
> On Thu, Jun 7, 2018 at 7:53 AM, Benjamin Kim  <mailto:bbuil...@gmail.com>> wrote:
> I created a JDBC interpreter for AWS Athena, and it passes the access key as 
> UID and secret key as PWD in the URL connection string. Does anyone know if I 
> can setup each user to pass their own credentials in a, sort of, credentials 
> file or config?
> 
> Thanks,
> Ben
> 
> 
> 
> -- 
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net <http://madeng.net/>

Credentials for JDBC

[Benjamin Kim]

I created a JDBC interpreter for AWS Athena, and it passes the access key as 
UID and secret key as PWD in the URL connection string. Does anyone know if I 
can setup each user to pass their own credentials in a, sort of, credentials 
file or config?

Thanks,
Ben

Re: Zeppelin 0.8

[Benjamin Kim]

Will Zeppelin 0.8 have Spark 2.3 support?

> On Apr 30, 2018, at 1:27 AM, Rotem Herzberg  
> wrote:
> 
> Thanks
> 
> On Mon, Apr 30, 2018 at 11:16 AM, Jeff Zhang  > wrote:
> 
> I am preparing the RC for 0.8
> 
> 
> Rotem Herzberg  >于2018年4月30日周一 下午3:57写道：
> Hi,
> 
> What is the release date for Zeppelin 0.8? (support for spark 2.3)
> 
> Thanks,
> 
> -- 
>     
> Rotem Herzberg
> SW Engineer | GigaSpaces Technologies
>  
> rotem.herzb...@gigaspaces.com    | M 
> +972547718880 <> 
> 
>   
> 
> 
> 
> 
> -- 
>     
> Rotem Herzberg
> SW Engineer | GigaSpaces Technologies
>  
> rotem.herzb...@gigaspaces.com    | M 
> +972547718880 <> 
> 
>   
> 
> 

Re: Cloudera Spark 2.2

[Benjamin Kim]

Hi Ruslan,

Can you send me the steps you used to build it, especially the Maven
command with the arguments? I will try to build it also.

I do believe that the binaries are for official releases.

Cheers,
Ben


On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov <dautkha...@gmail.com>
wrote:

> It was built. I think binaries are only available for official releases?
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>
>> Did you build Zeppelin or download the binary?
>>
>> On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <dautkha...@gmail.com>
>> wrote:
>>
>>> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>>>
>>> Yes, we have that spark home in zeppelin-env.sh
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>>>
>>>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
>>>> in zeppelin-env.sh to what you have below.
>>>>
>>>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <dautkha...@gmail.com>
>>>> wrote:
>>>>
>>>>> You don't have to use spark2-shell and spark2-submit to use Spark 2.
>>>>> That can be controled by setting SPARK_HOME using regular
>>>>> spark-submit/spark-shell.
>>>>>
>>>>> $ which spark-submit
>>>>> /usr/bin/spark-submit
>>>>> $ which spark-shell
>>>>> /usr/bin/spark-shell
>>>>>
>>>>> $ spark-shell
>>>>> Welcome to
>>>>>     __
>>>>>  / __/__  ___ _/ /__
>>>>> _\ \/ _ \/ _ `/ __/  '_/
>>>>>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>>>>>       /_/
>>>>>
>>>>>
>>>>>
>>>>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>>>>>
>>>>> $ spark-shell
>>>>> Welcome to
>>>>>     __
>>>>>  / __/__  ___ _/ /__
>>>>> _\ \/ _ \/ _ `/ __/  '_/
>>>>>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>>>>>   /_/
>>>>>
>>>>>
>>>>> spark-submit and spark-shell are just shell script wrappers.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ruslan Dautkhanov
>>>>>
>>>>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports
>>>>>> Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from
>>>>>> Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the
>>>>>> calls to spark-shell and spark-submit. To force the use of Spark 2.x, the
>>>>>> calls need to be spark2-shell and spark2-submit. I wonder if this is
>>>>>> causing the problem. By the way, we are using Java8 corporate wide, and
>>>>>> there seems to be no problems using Zeppelin.
>>>>>>
>>>>>> Cheers,
>>>>>> Ben
>>>>>>
>>>>>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <
>>>>>> dautkha...@gmail.com> wrote:
>>>>>>
>>>>>>> Might need to recompile Zeppelin with Scala 2.11?
>>>>>>> Also Spark 2.2 now requires JDK8 I believe.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ruslan Dautkhanov
>>>>>>>
>>>>>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here is more.
>>>>>>>>
>>>>>>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>>>>>>>> User-defined SPARK_HOME
>>>>>>>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>>>>>>>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>>>>>>>> WARNING: Running spark-class from user-defined location.
>>>>>>>> Exception

Re: Cloudera Spark 2.2

[Benjamin Kim]

Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <dautkha...@gmail.com>
wrote:

> We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.
>
> Yes, we have that spark home in zeppelin-env.sh
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>
>> Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME
>> in zeppelin-env.sh to what you have below.
>>
>> On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <dautkha...@gmail.com>
>> wrote:
>>
>>> You don't have to use spark2-shell and spark2-submit to use Spark 2.
>>> That can be controled by setting SPARK_HOME using regular
>>> spark-submit/spark-shell.
>>>
>>> $ which spark-submit
>>> /usr/bin/spark-submit
>>> $ which spark-shell
>>> /usr/bin/spark-shell
>>>
>>> $ spark-shell
>>> Welcome to
>>>     __
>>>  / __/__  ___ _/ /__
>>> _\ \/ _ \/ _ `/ __/  '_/
>>>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>>>   /_/
>>>
>>>
>>>
>>> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>>>
>>> $ spark-shell
>>> Welcome to
>>>         __
>>>  / __/__  ___ _/ /__
>>> _\ \/ _ \/ _ `/ __/  '_/
>>>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>>>   /_/
>>>
>>>
>>> spark-submit and spark-shell are just shell script wrappers.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com>
>>> wrote:
>>>
>>>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
>>>> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
>>>> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
>>>> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
>>>> to be spark2-shell and spark2-submit. I wonder if this is causing the
>>>> problem. By the way, we are using Java8 corporate wide, and there seems to
>>>> be no problems using Zeppelin.
>>>>
>>>> Cheers,
>>>> Ben
>>>>
>>>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <dautkha...@gmail.com>
>>>> wrote:
>>>>
>>>>> Might need to recompile Zeppelin with Scala 2.11?
>>>>> Also Spark 2.2 now requires JDK8 I believe.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ruslan Dautkhanov
>>>>>
>>>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Here is more.
>>>>>>
>>>>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>>>>>> User-defined SPARK_HOME
>>>>>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>>>>>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>>>>>> WARNING: Running spark-class from user-defined location.
>>>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>>>>>> at
>>>>>> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>>>>>> at
>>>>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>>>>> at
>>>>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>>>>> at scala.Option.getOrElse(Option.scala:120)
>>>>>> at
>>>>>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>>>>>> at
>>>>>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>>
>>>>>> Cheers,
>>>>>> Ben
>>>>>>
>>>

Re: Cloudera Spark 2.2

[Benjamin Kim]

Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in
zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <dautkha...@gmail.com>
wrote:

> You don't have to use spark2-shell and spark2-submit to use Spark 2.
> That can be controled by setting SPARK_HOME using regular
> spark-submit/spark-shell.
>
> $ which spark-submit
> /usr/bin/spark-submit
> $ which spark-shell
> /usr/bin/spark-shell
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 1.6.0
>   /_/
>
>
>
> $ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
>
> $ spark-shell
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
>   /_/
>
>
> spark-submit and spark-shell are just shell script wrappers.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <bbuil...@gmail.com> wrote:
>
>> According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark
>> 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera.
>> For some reason, Cloudera defaults to Spark 1.6 and so does the calls to
>> spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
>> to be spark2-shell and spark2-submit. I wonder if this is causing the
>> problem. By the way, we are using Java8 corporate wide, and there seems to
>> be no problems using Zeppelin.
>>
>> Cheers,
>> Ben
>>
>> On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <dautkha...@gmail.com>
>> wrote:
>>
>>> Might need to recompile Zeppelin with Scala 2.11?
>>> Also Spark 2.2 now requires JDK8 I believe.
>>>
>>>
>>>
>>> --
>>> Ruslan Dautkhanov
>>>
>>> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>>>
>>>> Here is more.
>>>>
>>>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>>>> User-defined SPARK_HOME
>>>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>>>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>>>> WARNING: Running spark-class from user-defined location.
>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>>>> at
>>>> org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>>>> at scala.Option.getOrElse(Option.scala:120)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>
>>>> Cheers,
>>>> Ben
>>>>
>>>>
>>>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>>>
>>>>>
>>>>> Then it is due to some classpath issue. I am not sure familiar with
>>>>> CDH, please check whether spark of CDH include hadoop jar with it.
>>>>>
>>>>>
>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道：
>>>>>
>>>>>> Here is the error that was sent to me.
>>>>>>
>>>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception in
>>>>>> thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/hadoop/fs/FSDataInputStream
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> org.apache.hadoop.fs.FSDataInputStream
>>>>>>
>>>>>> Cheers,
>>>>>> Ben
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set 
>>>>>>> zeppelin.spark.enableSupportedVersionCheck
>>>>>>> in interpreter setting to disable the supported version check.
>>>>>>>
>>>>>>>
>>>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>>>>
>>>>>>>>
>>>>>>>> What's the error you see in log ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>>>>>
>>>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of
>>>>>>>>> Spark 2.2? I can't get it to work. I downloaded the binary and set
>>>>>>>>> SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be 
>>>>>>>>> missing
>>>>>>>>> something.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Ben
>>>>>>>>>
>>>>>>>>
>>>
>

Geo Map Charting

[Benjamin Kim]

Anyone every try to chart density clusters or heat maps onto a geo map of
the earth in Zeppelin? Can it be done?

Cheers,
Ben

Re: Cloudera Spark 2.2

[Benjamin Kim]

According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1.
But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For
some reason, Cloudera defaults to Spark 1.6 and so does the calls to
spark-shell and spark-submit. To force the use of Spark 2.x, the calls need
to be spark2-shell and spark2-submit. I wonder if this is causing the
problem. By the way, we are using Java8 corporate wide, and there seems to
be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <dautkha...@gmail.com>
wrote:

> Might need to recompile Zeppelin with Scala 2.11?
> Also Spark 2.2 now requires JDK8 I believe.
>
>
>
> --
> Ruslan Dautkhanov
>
> On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>
>> Here is more.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: WARNING:
>> User-defined SPARK_HOME
>> (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
>> overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
>> WARNING: Running spark-class from user-defined location.
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
>> at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
>> at scala.Option.getOrElse(Option.scala:120)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
>> at
>> org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>
>>>
>>> Then it is due to some classpath issue. I am not sure familiar with CDH,
>>> please check whether spark of CDH include hadoop jar with it.
>>>
>>>
>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道：
>>>
>>>> Here is the error that was sent to me.
>>>>
>>>> org.apache.zeppelin.interpreter.InterpreterException: Exception in
>>>> thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/hadoop/fs/FSDataInputStream
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.hadoop.fs.FSDataInputStream
>>>>
>>>> Cheers,
>>>> Ben
>>>>
>>>>
>>>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>>>
>>>>>
>>>>> By default, 0.7.1 doesn't support spark 2.2. But you can set 
>>>>> zeppelin.spark.enableSupportedVersionCheck
>>>>> in interpreter setting to disable the supported version check.
>>>>>
>>>>>
>>>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>>
>>>>>>
>>>>>> What's the error you see in log ?
>>>>>>
>>>>>>
>>>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>>>
>>>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>>>>>>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME 
>>>>>>> to
>>>>>>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Ben
>>>>>>>
>>>>>>
>

Re: Cloudera Spark 2.2

[Benjamin Kim]

Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined
SPARK_HOME
(/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2)
overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError:
scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at
org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at
org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <zjf...@gmail.com> wrote:

>
> Then it is due to some classpath issue. I am not sure familiar with CDH,
> please check whether spark of CDH include hadoop jar with it.
>
>
> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:22写道：
>
>> Here is the error that was sent to me.
>>
>> org.apache.zeppelin.interpreter.InterpreterException: Exception in thread
>> "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/fs/FSDataInputStream
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.fs.FSDataInputStream
>>
>> Cheers,
>> Ben
>>
>>
>> On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote:
>>
>>>
>>> By default, 0.7.1 doesn't support spark 2.2. But you can set 
>>> zeppelin.spark.enableSupportedVersionCheck
>>> in interpreter setting to disable the supported version check.
>>>
>>>
>>> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>
>>>>
>>>> What's the error you see in log ?
>>>>
>>>>
>>>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>>>
>>>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>>>>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to
>>>>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>>>>
>>>>> Cheers,
>>>>> Ben
>>>>>
>>>>

Re: Cloudera Spark 2.2

[Benjamin Kim]

Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread
"main" java.lang.NoClassDefFoundError:
org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <zjf...@gmail.com> wrote:

>
> By default, 0.7.1 doesn't support spark 2.2. But you can set 
> zeppelin.spark.enableSupportedVersionCheck
> in interpreter setting to disable the supported version check.
>
>
> Jeff Zhang <zjf...@gmail.com>于2017年8月2日周三 上午8:18写道：
>
>>
>> What's the error you see in log ?
>>
>>
>> Benjamin Kim <bbuil...@gmail.com>于2017年8月2日周三 上午8:18写道：
>>
>>> Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark
>>> 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to
>>> /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.
>>>
>>> Cheers,
>>> Ben
>>>
>>

Cloudera Spark 2.2

[Benjamin Kim]

Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I
can't get it to work. I downloaded the binary and set SPARK_HOME to
/opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben

Centos 7 Compatibility

[Benjamin Kim]

All,

I’m curious to know if Zeppelin will work with CentOS 7. I don’t see it in the 
list of OS’s supported.

Thanks,
Ben

Login/Logout Problem

[Benjamin Kim]

We are running into problems where users login and staying logged in. When they 
try to run JDBC queries or even opening a notebook, they get flickering in the 
browser where the green color dot next to the username turns red, then back to 
green, then back to red, etc. When it stops doing that, then users are able to 
use the notebook finally, but when they do, they get an ERROR when clicking the 
run arrow often. Since these notebooks were migrated from Zeppelin 0.6 to 
Zeppelin 0.7, I suspect that there might be incompatibility issues.

Thanks,
Ben

Zeppelin Service Install

[Benjamin Kim]

Anyone have installed Zeppelin onto a CentOS/RedHat server and made it into a 
service? I can’t seem to find the instructions on how to do this.

Cheers,
Ben

Re: PostgreSQL JDBC Connections

[Benjamin Kim]

We are using the JDBC interpreter. The business analysts only know SQL and run 
ad-hoc queries for their report exports to CSV.

Cheers,
Ben


> On Jan 5, 2017, at 2:21 PM, t p <tauis2...@gmail.com> wrote:
> 
> Are you using JDBC or the PSQL interpreter? I had encountered something 
> similar while using the PSQL interpreter and I had to restart Zeppelin. 
> 
> My experience using PSQL (Postgresql, HAWK) was not as good as using 
> spark/scala wrappers (JDBC data source) to connect via JDBC and then register 
> temp tables. This approach allowed me to work with dynamic forms in a more 
> meaningful way e.g. use SQL results to create a new drop down to drive the 
> next page etc…
> 
> 
> 
>> On Jan 5, 2017, at 12:57 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>> 
>> We are getting “out of shared memory” errors when multiple users are running 
>> SQL queries against our PostgreSQL DB either simultaneously or throughout 
>> the day. When this happens, Zeppelin 0.6.0 becomes unresponsive for any more 
>> SQL queries. It looks like this is being caused by too many locks being 
>> taken and not released, transactions never closing, and/or connections never 
>> closing. Has anyone encountered Zeppelin 0.6.0 such an issue as this? If so, 
>> is there a solution for it?
>> 
>> Thanks,
>> Ben
> 

JDBC Connections

[Benjamin Kim]

We are using Zeppelin 0.6.0 as a self-service for our clients to query our 
PostgreSQL databases. We are noticing that the connections are not closing 
after each one of them are done. What is the normal operating procedure to have 
these connections close when idle? Our scope for the JDBC interpreter is 
“shared”, which I thought would make 1 connection for all notebooks. It would 
seem that I am wrong. Anyone have any ideas on what would help?

Thanks,
Ben

Re: Pass Credentials through JDBC

[Benjamin Kim]

Thank you. I’ll take a look.


> On Jul 28, 2016, at 8:16 AM, Jongyoul Lee <jongy...@gmail.com> wrote:
> 
> You can find more information on 
> https://issues.apache.org/jira/browse/ZEPPELIN-1146 
> <https://issues.apache.org/jira/browse/ZEPPELIN-1146>
> 
> Hope this help,
> Jongyoul
> 
> On Fri, Jul 29, 2016 at 12:08 AM, Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> Hi Jonyoul,
> 
> How would I enter credentials with the current version of Zeppelin? Do you 
> know of a way to make it work now?
> 
> Thanks,
> Ben
> 
>> On Jul 28, 2016, at 8:06 AM, Jongyoul Lee <jongy...@gmail.com 
>> <mailto:jongy...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> In my plan, this is a next step after 
>> https://issues.apache.org/jira/browse/ZEPPELIN-1210 
>> <https://issues.apache.org/jira/browse/ZEPPELIN-1210>. But for now, there's 
>> no way to pass your credentials with hiding them. I hope that would be 
>> included in 0.7.0.
>> 
>> Regards,
>> Jongyoul
>> 
>> On Thu, Jul 28, 2016 at 11:22 PM, Benjamin Kim <bbuil...@gmail.com 
>> <mailto:bbuil...@gmail.com>> wrote:
>> How do I pass username and password to JDBC connections such as Phoenix and 
>> Hive that are my own? Can my credentials be passed from Shiro after logging 
>> in? Or do I have to set them at the Interpreter level without sharing them? 
>> I wish there was more information on this.
>> 
>> Thanks,
>> Ben
>> 
>> 
>> 
>> -- 
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net <http://madeng.net/>
> 
> 
> 
> 
> -- 
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net <http://madeng.net/>

Re: [ANNOUNCE] Apache Zeppelin 0.6.0 released

[Benjamin Kim]

Hi JL,

Spark is version 1.6.0 and Akka is 2.2.3. But, Cloudera always back ports 
things from newer versions. They told me that they ported some bug fixes from 
Spark 2.0.

Please let me know if you need any more information.

Cheers,
Ben


> On Jul 9, 2016, at 10:12 PM, Jongyoul Lee <jongy...@gmail.com> wrote:
> 
> Hi all,
> 
> Could you guys check the CDH's version of Spark? As I've tested it for a long 
> time ago, it is a little bit different from vanila one, for example, the 
> CDH's one has a different version of some depedencies including Akka.
> 
> Regards,
> JL
> 
> On Sat, Jul 9, 2016 at 11:47 PM, Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> Feix,
> 
> I added hive-site.xml to the conf directory and restarted Zeppelin. Now, I 
> get another error:
> 
> java.lang.ClassNotFoundException: 
> line1631424043$24.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at java.lang.Class.forName0(Native Method)
>   at java.lang.Class.forName(Class.java:348)
>   at 
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>   at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
>   at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>   at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
>   at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>   at java.io.Object

Re: [ANNOUNCE] Apache Zeppelin 0.6.0 released

[Benjamin Kim]

(ObjectInputStream.java:1924)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at 
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at 
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:64)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks for helping.

Ben


> On Jul 8, 2016, at 10:47 PM, Felix Cheung <felixcheun...@hotmail.com> wrote:
> 
> For #1, do you know if Spark can find the Hive metastore config (typically in 
> hive-site.xml) - Spark's log should indicate that.
> 
> 
> _________
> From: Benjamin Kim <bbuil...@gmail.com <mailto:bbuil...@gmail.com>>
> Sent: Friday, July 8, 2016 6:53 AM
> Subject: Re: [ANNOUNCE] Apache Zeppelin 0.6.0 released
> To: <users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>>
> Cc: <d...@zeppelin.apache.org <mailto:d...@zeppelin.apache.org>>
> 
> 
> Felix,
> 
> I forgot to add that I built Zeppelin from source 
> http://mirrors.ibiblio.org/apache/zeppelin/zeppelin-0.6.0/zeppelin-0.6.0.tgz 
> <http://mirrors.ibiblio.org/apache/zeppelin/zeppelin-0.6.0/zeppelin-0.6.0.tgz>
>  using this command "mvn clean package -DskipTests -Pspark-1.6 -Phadoop-2.6 
> -Dspark.version=1.6.0-cdh5.7.1 -Dhadoop.version=2.6.0-cdh5.7.1 -Ppyspark 
> -Pvendor-repo -Pbuild-distr -Dhbase.hbase.version=1.2.0-cdh5.7.1 
> -Dhbase.hadoop.version=2.6.0-cdh5.7.1”.
> 
> I did this because we are using HBase 1.2 within CDH 5.7.1.
> 
> Hope this helps clarify.
> 
> Thanks,
> Ben
> 
> 
> 
> On Jul 8, 2016, at 2:01 AM, Felix Cheung <felixcheun...@hotmail.com 
> <mailto:felixcheun...@hotmail.com>> wrote:
> 
> Is this possibly caused by CDH requiring a build-from-source instead of the 
> official binary releases?
> 
> 
> 
> 
> 
> On Thu, Jul 7, 2016 at 8:22 PM -0700, "Benjamin Kim" <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> 
> Moon,
> 
> My environmental setup consists of an 18 node CentOS 6.7 cluster with 24 
> cores, 64GB, 12TB storage each:
> 3 of those nodes are used as Zookeeper servers, HDFS name nodes, and a YARN 
> resource manager
> 15 are for data nodes
> jdk1.8_60 and CDH 5.7.1 installed
> 
> Another node is an app server, 24 cores, 128GB memory, 1TB storage. It has 
> Zeppelin 0.6.0 and Livy 0.2.0 running on it. Plus, Hive Metastore and 
> HiveServer2, Hue, and Oozie are running on it from CDH 5.7.1.
> 
> This is our QA cluster where we are testing before deploying to production.
> 
> If you need more information, please let me know.
> 
> Thanks,
> Ben
> 
>  
> 
> On Jul 7, 2016, at 7:54 PM, moon soo Lee <m...@apache.org 
> <mailto:m...@apache.org>> 

Re: [ANNOUNCE] Apache Zeppelin 0.6.0 released

[Benjamin Kim]

Felix,

I forgot to add that I built Zeppelin from source 
http://mirrors.ibiblio.org/apache/zeppelin/zeppelin-0.6.0/zeppelin-0.6.0.tgz 
<http://mirrors.ibiblio.org/apache/zeppelin/zeppelin-0.6.0/zeppelin-0.6.0.tgz> 
using this command "mvn clean package -DskipTests -Pspark-1.6 -Phadoop-2.6 
-Dspark.version=1.6.0-cdh5.7.1 -Dhadoop.version=2.6.0-cdh5.7.1 -Ppyspark 
-Pvendor-repo -Pbuild-distr -Dhbase.hbase.version=1.2.0-cdh5.7.1 
-Dhbase.hadoop.version=2.6.0-cdh5.7.1”.

I did this because we are using HBase 1.2 within CDH 5.7.1.

Hope this helps clarify.

Thanks,
Ben



> On Jul 8, 2016, at 2:01 AM, Felix Cheung <felixcheun...@hotmail.com> wrote:
> 
> Is this possibly caused by CDH requiring a build-from-source instead of the 
> official binary releases?
> 
> 
> 
> 
> 
> On Thu, Jul 7, 2016 at 8:22 PM -0700, "Benjamin Kim" <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> 
> Moon,
> 
> My environmental setup consists of an 18 node CentOS 6.7 cluster with 24 
> cores, 64GB, 12TB storage each:
> 3 of those nodes are used as Zookeeper servers, HDFS name nodes, and a YARN 
> resource manager
> 15 are for data nodes
> jdk1.8_60 and CDH 5.7.1 installed
> 
> Another node is an app server, 24 cores, 128GB memory, 1TB storage. It has 
> Zeppelin 0.6.0 and Livy 0.2.0 running on it. Plus, Hive Metastore and 
> HiveServer2, Hue, and Oozie are running on it from CDH 5.7.1.
> 
> This is our QA cluster where we are testing before deploying to production.
> 
> If you need more information, please let me know.
> 
> Thanks,
> Ben
> 
>  
> 
>> On Jul 7, 2016, at 7:54 PM, moon soo Lee <m...@apache.org 
>> <mailto:m...@apache.org>> wrote:
>> 
>> Randy,
>> 
>> Helium is not included in 0.6.0 release. Could you check which version are 
>> you using?
>> I created a fix for 500 errors from Helium URL in master branch. 
>> https://github.com/apache/zeppelin/pull/1150 
>> <https://github.com/apache/zeppelin/pull/1150>
>> 
>> Ben,
>> I can not reproduce the error, could you share how to reproduce error, or 
>> share your environment?
>> 
>> Thanks,
>> moon
>> 
>> On Thu, Jul 7, 2016 at 4:02 PM Randy Gelhausen <rgel...@gmail.com 
>> <mailto:rgel...@gmail.com>> wrote:
>> I don't- I hoped providing that information may help finding & fixing the 
>> problem.
>> 
>> On Thu, Jul 7, 2016 at 5:53 PM, Benjamin Kim <bbuil...@gmail.com 
>> <mailto:bbuil...@gmail.com>> wrote:
>> Hi Randy,
>> 
>> Do you know of any way to fix it or know of a workaround?
>> 
>> Thanks,
>> Ben
>> 
>>> On Jul 7, 2016, at 2:08 PM, Randy Gelhausen <rgel...@gmail.com 
>>> <mailto:rgel...@gmail.com>> wrote:
>>> 
>>> HTTP 500 errors from a Helium URL
>> 
>> 
> 

Re: [ANNOUNCE] Apache Zeppelin 0.6.0 released

[Benjamin Kim]

Moon,

My environmental setup consists of an 18 node CentOS 6.7 cluster with 24 cores, 
64GB, 12TB storage each:
3 of those nodes are used as Zookeeper servers, HDFS name nodes, and a YARN 
resource manager
15 are for data nodes
jdk1.8_60 and CDH 5.7.1 installed

Another node is an app server, 24 cores, 128GB memory, 1TB storage. It has 
Zeppelin 0.6.0 and Livy 0.2.0 running on it. Plus, Hive Metastore and 
HiveServer2, Hue, and Oozie are running on it from CDH 5.7.1.

This is our QA cluster where we are testing before deploying to production.

If you need more information, please let me know.

Thanks,
Ben

 

> On Jul 7, 2016, at 7:54 PM, moon soo Lee <m...@apache.org> wrote:
> 
> Randy,
> 
> Helium is not included in 0.6.0 release. Could you check which version are 
> you using?
> I created a fix for 500 errors from Helium URL in master branch. 
> https://github.com/apache/zeppelin/pull/1150 
> <https://github.com/apache/zeppelin/pull/1150>
> 
> Ben,
> I can not reproduce the error, could you share how to reproduce error, or 
> share your environment?
> 
> Thanks,
> moon
> 
> On Thu, Jul 7, 2016 at 4:02 PM Randy Gelhausen <rgel...@gmail.com 
> <mailto:rgel...@gmail.com>> wrote:
> I don't- I hoped providing that information may help finding & fixing the 
> problem.
> 
> On Thu, Jul 7, 2016 at 5:53 PM, Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> Hi Randy,
> 
> Do you know of any way to fix it or know of a workaround?
> 
> Thanks,
> Ben
> 
>> On Jul 7, 2016, at 2:08 PM, Randy Gelhausen <rgel...@gmail.com 
>> <mailto:rgel...@gmail.com>> wrote:
>> 
>> HTTP 500 errors from a Helium URL
> 
> 

Re: [ANNOUNCE] Apache Zeppelin 0.6.0 released

[Benjamin Kim]

Hi Randy,

Do you know of any way to fix it or know of a workaround?

Thanks,
Ben

> On Jul 7, 2016, at 2:08 PM, Randy Gelhausen  wrote:
> 
> HTTP 500 errors from a Helium URL

Re: Shiro LDAP w/ Search Bind Authentication

[Benjamin Kim]

Rob,

I got it to work without having to use those settings. I guess Shiro gets 
around our LDAP authentication.

Thanks,
Ben


> On Jul 6, 2016, at 3:33 PM, Rob Anderson <rockclimbings...@gmail.com> wrote:
> 
> You can find some documentation on it here: 
> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/security/shiroauthentication.html
>  
> <https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/security/shiroauthentication.html>
> 
> I believe you'll need to be running the .6 release or .7 snapshot to use 
> shiro.
> 
> We're authing against AD via ldaps calls without issue.  We're then using 
> group memberships to define roles and control access to notebooks.
> 
> Hope that helps.
> 
> Rob
> 
> 
> On Wed, Jul 6, 2016 at 2:01 PM, Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> I have been trying to find documentation on how to enable LDAP 
> authentication, but I cannot find how to enter the values for these 
> configurations. This is necessary because our LDAP server is secured. Here 
> are the properties that I need to set:
> ldap_cert
> use_start_tls
> bind_dn
> bind_password
> 
> Can someone help?
> 
> Thanks,
> Ben
> 
> 

Shiro LDAP w/ Search Bind Authentication

[Benjamin Kim]

I have been trying to find documentation on how to enable LDAP authentication, 
but I cannot find how to enter the values for these configurations. This is 
necessary because our LDAP server is secured. Here are the properties that I 
need to set:
ldap_cert
use_start_tls
bind_dn
bind_password

Can someone help?

Thanks,
Ben

Re: spark interpreter

[Benjamin Kim]

Moon,

I have downloaded and tested the bin-all tarball, and it has some deficiencies 
compared to the build-from-source version.
CSV, TSV download is missing
Doesn’t work with HBase 1.2 in CDH 5.7.0
Spark still does not work with Spark 1.6.0 in CDH 5.7.0 (JDK8)
Using Livy is a good workaround
Doesn’t work with Phoenix 4.7 in CDH 5.7.0

Everything else looks good especially in the area of multi-tenancy and 
security. I would like to know how to use the Credentials feature on securing 
usernames and passwords. I couldn’t find documentation on how.

Thanks,
Ben

> On Jul 1, 2016, at 9:04 AM, moon soo Lee <m...@apache.org> wrote:
> 
> 0.6.0 is currently in vote in dev@ list.
> http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html
>  
> <http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html>
> 
> Thanks,
> moon
> 
> On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <l...@ca.ibm.com 
> <mailto:l...@ca.ibm.com>> wrote:
> What is the expected day for v0.6?
> 
> 
> 
> 
> From:moon soo Lee <leemoon...@gmail.com <mailto:leemoon...@gmail.com>>
> To:users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>
> Date:2016/06/30 11:36 AM
> Subject:Re: spark interpreter
> 
> 
> 
> Hi Ben,
> 
> Livy interpreter is included in 0.6.0. If it is not listed when you create 
> interpreter setting, could you check if your 'zeppelin.interpreters' property 
> list Livy interpreter classes? (conf/zeppelin-site.xml)
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> On a side note…
> 
> Has anyone got the Livy interpreter to be added as an interpreter in the 
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. 
> Could this interfere?
> 
> Thanks,
> Ben
> 
> 
> On Jun 29, 2016, at 11:18 AM, moon soo Lee <m...@apache.org 
> <mailto:m...@apache.org>> wrote:
> 
> Livy interpreter internally creates multiple sessions for each user, 
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each 
> user, 'scoped' or 'isolated' mode will result create sessions per notebook, 
> per user.
> 
> Notebook is shared among users, they always use the same interpreter 
> instance/process, for now. I think supporting per user interpreter 
> instance/process would be future work.
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <chen.song...@gmail.com 
> <mailto:chen.song...@gmail.com>> wrote:
> Thanks for your explanation, Moon.
> 
> Following up on this, I can see the difference in terms of single or multiple 
> interpreter processes. 
> 
> With respect to spark drivers, since each interpreter spawns a separate Spark 
> driver in regular Spark interpreter setting, it is clear to me the different 
> implications of the 3 binding modes.
> 
> However, when it comes to Livy server with impersonation turned on, I am a 
> bit confused. Will Livy interpreter always create a new Spark driver (along 
> with a Spark Context instance) for each user session, regardless of the 
> binding mode of Livy interpreter? I am not very familiar with Livy, but from 
> what I could tell, I see no difference between different binding modes for 
> Livy on as far as how Spark drivers are concerned.
> 
> Last question, when a notebook is shared among users, will they always use 
> the same interpreter instance/process already created?
> 
> Thanks
> Chen
> 
> 
> 
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <m...@apache.org 
> <mailto:m...@apache.org>> wrote:
> Hi,
> 
> Thanks for asking question. It's not dumb question at all, Zeppelin docs does 
> not explain very well.
> 
> Spark Interpreter, 
> 
> 'shared' mode, a spark interpreter setting spawn a interpreter process to 
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter 
> processes per notebook which binded to this interpreter setting.
> 
> Using Livy interpreter,
> 
> Zeppelin propagate current user information to Livy interpreter. And Livy 
> interpreter creates different session per user via Livy Server.
> 
> 
> Hope this helps.
> 
> Thanks,
> moon
> 
> 
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <chen.song...@gmail.com 
> <mailto:chen.song...@gmail.com>> wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
> `shared` Spark interpreter, every user share the same SparkContext. Sorry for 
> the dumb question, how does it differ from Spark via Ivy Server?
> 
> 
> -- 
> Chen Song
> 
> 
> 
> 

Re: spark interpreter

[Benjamin Kim]

Moon,

That worked! There were quite a few more configuration properties added, so I 
added those too in both zeppelin-site.xml and zeppelin-env.sh. But, now, I’m 
getting errors starting a spark context.

Thanks,
Ben

> On Jun 30, 2016, at 8:10 AM, moon soo Lee <leemoon...@gmail.com> wrote:
> 
> Hi Ben,
> 
> Livy interpreter is included in 0.6.0. If it is not listed when you create 
> interpreter setting, could you check if your 'zeppelin.interpreters' property 
> list Livy interpreter classes? (conf/zeppelin-site.xml)
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> On a side note…
> 
> Has anyone got the Livy interpreter to be added as an interpreter in the 
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. 
> Could this interfere?
> 
> Thanks,
> Ben
> 
> 
>> On Jun 29, 2016, at 11:18 AM, moon soo Lee <m...@apache.org 
>> <mailto:m...@apache.org>> wrote:
>> 
>> Livy interpreter internally creates multiple sessions for each user, 
>> independently from 3 binding modes supported in Zeppelin.
>> Therefore, 'shared' mode, Livy interpreter will create sessions per each 
>> user, 'scoped' or 'isolated' mode will result create sessions per notebook, 
>> per user.
>> 
>> Notebook is shared among users, they always use the same interpreter 
>> instance/process, for now. I think supporting per user interpreter 
>> instance/process would be future work.
>> 
>> Thanks,
>> moon
>> 
>> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <chen.song...@gmail.com 
>> <mailto:chen.song...@gmail.com>> wrote:
>> Thanks for your explanation, Moon.
>> 
>> Following up on this, I can see the difference in terms of single or 
>> multiple interpreter processes. 
>> 
>> With respect to spark drivers, since each interpreter spawns a separate 
>> Spark driver in regular Spark interpreter setting, it is clear to me the 
>> different implications of the 3 binding modes.
>> 
>> However, when it comes to Livy server with impersonation turned on, I am a 
>> bit confused. Will Livy interpreter always create a new Spark driver (along 
>> with a Spark Context instance) for each user session, regardless of the 
>> binding mode of Livy interpreter? I am not very familiar with Livy, but from 
>> what I could tell, I see no difference between different binding modes for 
>> Livy on as far as how Spark drivers are concerned.
>> 
>> Last question, when a notebook is shared among users, will they always use 
>> the same interpreter instance/process already created?
>> 
>> Thanks
>> Chen
>> 
>> 
>> 
>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <m...@apache.org 
>> <mailto:m...@apache.org>> wrote:
>> Hi,
>> 
>> Thanks for asking question. It's not dumb question at all, Zeppelin docs 
>> does not explain very well.
>> 
>> Spark Interpreter, 
>> 
>> 'shared' mode, a spark interpreter setting spawn a interpreter process to 
>> serve all notebooks which binded to this interpreter setting.
>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter 
>> processes per notebook which binded to this interpreter setting.
>> 
>> Using Livy interpreter,
>> 
>> Zeppelin propagate current user information to Livy interpreter. And Livy 
>> interpreter creates different session per user via Livy Server.
>> 
>> 
>> Hope this helps.
>> 
>> Thanks,
>> moon
>> 
>> 
>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <chen.song...@gmail.com 
>> <mailto:chen.song...@gmail.com>> wrote:
>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
>> `shared` Spark interpreter, every user share the same SparkContext. Sorry 
>> for the dumb question, how does it differ from Spark via Ivy Server?
>> 
>> 
>> -- 
>> Chen Song
>> 
> 

Re: spark interpreter

[Benjamin Kim]

On a side note…

Has anyone got the Livy interpreter to be added as an interpreter in the latest 
build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. Could this 
interfere?

Thanks,
Ben


> On Jun 29, 2016, at 11:18 AM, moon soo Lee  wrote:
> 
> Livy interpreter internally creates multiple sessions for each user, 
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each 
> user, 'scoped' or 'isolated' mode will result create sessions per notebook, 
> per user.
> 
> Notebook is shared among users, they always use the same interpreter 
> instance/process, for now. I think supporting per user interpreter 
> instance/process would be future work.
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song  > wrote:
> Thanks for your explanation, Moon.
> 
> Following up on this, I can see the difference in terms of single or multiple 
> interpreter processes. 
> 
> With respect to spark drivers, since each interpreter spawns a separate Spark 
> driver in regular Spark interpreter setting, it is clear to me the different 
> implications of the 3 binding modes.
> 
> However, when it comes to Livy server with impersonation turned on, I am a 
> bit confused. Will Livy interpreter always create a new Spark driver (along 
> with a Spark Context instance) for each user session, regardless of the 
> binding mode of Livy interpreter? I am not very familiar with Livy, but from 
> what I could tell, I see no difference between different binding modes for 
> Livy on as far as how Spark drivers are concerned.
> 
> Last question, when a notebook is shared among users, will they always use 
> the same interpreter instance/process already created?
> 
> Thanks
> Chen
> 
> 
> 
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee  > wrote:
> Hi,
> 
> Thanks for asking question. It's not dumb question at all, Zeppelin docs does 
> not explain very well.
> 
> Spark Interpreter, 
> 
> 'shared' mode, a spark interpreter setting spawn a interpreter process to 
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter 
> processes per notebook which binded to this interpreter setting.
> 
> Using Livy interpreter,
> 
> Zeppelin propagate current user information to Livy interpreter. And Livy 
> interpreter creates different session per user via Livy Server.
> 
> 
> Hope this helps.
> 
> Thanks,
> moon
> 
> 
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song  > wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
> `shared` Spark interpreter, every user share the same SparkContext. Sorry for 
> the dumb question, how does it differ from Spark via Ivy Server?
> 
> 
> -- 
> Chen Song
> 

Re: Ask opinion regarding 0.6.0 release package

[Benjamin Kim]

Hi,

Our company’s use is spark, phoenix, jdbc/psql. So, if you make different 
packages, I would need the full one. In addition, for the minimized one, would 
there be a way to pick and choose interpreters to add/plug in?

Thanks,
Ben

> On Jun 17, 2016, at 1:02 AM, mina lee  wrote:
> 
> Hi all!
> 
> Zeppelin just started release process. Prior to creating release candidate I 
> want to ask users' opinion about how you want it to be packaged.
> 
> For the last release(0.5.6), we have released one binary package which 
> includes all interpreters.
> The concern with providing one type of binary package is that package size 
> will be quite big(~600MB).
> So I am planning to provide two binary packages:
>   - zeppelin-0.6.0-bin-all.tgz (includes all interpreters)
>   - zeppelin-0.6.0-bin-min.tgz (includes only most used interpreters)
> 
> I am thinking about putting spark(pyspark, sparkr, sql), python, jdbc, shell, 
> markdown, angular in minimized package.
> Could you give your opinion on whether these sets are enough, or some of them 
> are ok to be excluded?
> 
> Community's opinion will be helpful to make decision not only for 0.6.0 but 
> also for 0.7.0 release since we are planning to provide only minimized 
> package from 0.7.0 release. From the 0.7.0 version, interpreters those are 
> not included in binary package will be able to use dynamic interpreter 
> feature [1] which is in progress under [2].
> 
> Thanks,
> Mina
> 
> [1] 
> http://zeppelin.apache.org/docs/0.6.0-SNAPSHOT/manual/dynamicinterpreterload.html
>  
> 
> [2] https://github.com/apache/zeppelin/pull/908 
> 

Github Integration

[Benjamin Kim]

I heard that Zeppelin 0.6.0 is able to use its local notebook directory as a 
Github repo. Does anyone know of a way to have it work (workaround) with our 
company’s Github (Stash) repo server?

Any advice would be welcome.

Thanks,
Ben