Re: Spark interpreter Repl injection

[Carlos Diogo]

Thanks
I created the issue
Regards
Carlos

On Tue 9. Mar 2021 at 19:02, moon soo Lee  wrote:

> Pyspark interpreter have 'intp' variable exposed in its repl environment
> (for internal use). And we can resolve reference to Spark interpreter from
> the 'intp' variable. However, scala repl environment in Spark Interpreter
> doesn't expose any variables that is useful for finding Spark Interpreter
> itself. So had to find a way from pyspark interpreter.
>
> z.interpret() doesn't look like it can bring some problem, in my opinion.
>
> Thanks,
> moon
>
>
>
>
> On Tue, Mar 9, 2021 at 8:54 AM Carlos Diogo  wrote:
>
>> Looks good Moon
>> Is there a specific reason why you needed the pyspark interpreter  to
>> access the spark interpreter? Could not the spark interpreter
>> programmatically access itself (and the same for the pyspark interpreter)
>>
>> Would the issue be to expose the z.interpret() method?
>>
>> Best regards
>> Carlos
>>
>> On Tue, Mar 9, 2021 at 5:10 PM moon soo Lee  wrote:
>>
>>> I see. If you want to specify a file, precode might not the best option.
>>> I found a hacky way to do it. Accessing SparkInterpreter instance object
>>> from PysparkInterpreter.
>>>
>>> %pyspark
>>> sparkIntpField = intp.getClass().getDeclaredField("sparkInterpreter")
>>> sparkIntpField.setAccessible(True)
>>> sparkIntp = sparkIntpField.get(intp)
>>> # run my scala code
>>> sparkIntp.interpret("val a=10", z.getInterpreterContext())
>>>
>>>
>>> See attached screenshot.
>>>
>>> [image: image.png]
>>>
>>> This is accessing internal variables outside the official API. So it may
>>> break at any time.
>>>
>>> I think it's better to expose interpret() method through
>>> 'ZeppelinContext'. So inside Note,
>>>
>>> z.interpret(any_string)
>>>
>>> can work without accessing this method in a hacky way.
>>> Please feel free to file an issue.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>>
>>>
>>> On Mon, Mar 8, 2021 at 10:23 PM Carlos Diogo  wrote:
>>>
 Are you able to specify a file on the precode?
 For now my work around is from within the note and with the rest api ,
 to add a paragraph with the code I want to inject ( which can come from a
 file )
 It works ok , but with run all or schedule the code gets updated in the
 note , but the old Code still executes . Only on the next run it will take
 effect

 On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:

> Hi,
>
> How about precode
> ?
>  "zeppelin.SparkInterpreter.precode"
> can run scala code.
>
> Thanks,
> moon
>
>
> On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:
>
>> That does not work if you want to have Scala code in a file ( common
>> functions) which you want to invoke in the note
>> The alternative is to compile the code and then add the jar which
>> would be normal for an application.
>> But zeppelin is about scripting so this is a request I get very often
>> from the users.
>> Specially because the z.run does not work properly most of the times
>> Carlos
>>
>> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>>
>>> Why not copying scala code in zeppelin and run the notebook directly
>>> ?
>>>
>>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>>
 Dear all
 I have been  trying  to find a was to inject scala Code ( from
 String) into the spark interpreter
 In pyspark is easy with the exec function
 It should not be very difficult  to access from the Note scala repl
 interpreter but i could not find a way . I was even able to create a 
 new
 repl session but then I could not bind the objects
 Any tips ?
 Thanks
 --
 Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
 Carlos Diogo

>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
> --
 Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
 Carlos Diogo

>>>
>>
>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
> --
Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
Carlos Diogo

Re: Spark interpreter Repl injection

[moon soo Lee]

Pyspark interpreter have 'intp' variable exposed in its repl environment
(for internal use). And we can resolve reference to Spark interpreter from
the 'intp' variable. However, scala repl environment in Spark Interpreter
doesn't expose any variables that is useful for finding Spark Interpreter
itself. So had to find a way from pyspark interpreter.

z.interpret() doesn't look like it can bring some problem, in my opinion.

Thanks,
moon




On Tue, Mar 9, 2021 at 8:54 AM Carlos Diogo  wrote:

> Looks good Moon
> Is there a specific reason why you needed the pyspark interpreter  to
> access the spark interpreter? Could not the spark interpreter
> programmatically access itself (and the same for the pyspark interpreter)
>
> Would the issue be to expose the z.interpret() method?
>
> Best regards
> Carlos
>
> On Tue, Mar 9, 2021 at 5:10 PM moon soo Lee  wrote:
>
>> I see. If you want to specify a file, precode might not the best option.
>> I found a hacky way to do it. Accessing SparkInterpreter instance object
>> from PysparkInterpreter.
>>
>> %pyspark
>> sparkIntpField = intp.getClass().getDeclaredField("sparkInterpreter")
>> sparkIntpField.setAccessible(True)
>> sparkIntp = sparkIntpField.get(intp)
>> # run my scala code
>> sparkIntp.interpret("val a=10", z.getInterpreterContext())
>>
>>
>> See attached screenshot.
>>
>> [image: image.png]
>>
>> This is accessing internal variables outside the official API. So it may
>> break at any time.
>>
>> I think it's better to expose interpret() method through
>> 'ZeppelinContext'. So inside Note,
>>
>> z.interpret(any_string)
>>
>> can work without accessing this method in a hacky way.
>> Please feel free to file an issue.
>>
>> Thanks,
>> moon
>>
>>
>>
>>
>> On Mon, Mar 8, 2021 at 10:23 PM Carlos Diogo  wrote:
>>
>>> Are you able to specify a file on the precode?
>>> For now my work around is from within the note and with the rest api ,
>>> to add a paragraph with the code I want to inject ( which can come from a
>>> file )
>>> It works ok , but with run all or schedule the code gets updated in the
>>> note , but the old Code still executes . Only on the next run it will take
>>> effect
>>>
>>> On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:
>>>
 Hi,

 How about precode
 ?
  "zeppelin.SparkInterpreter.precode"
 can run scala code.

 Thanks,
 moon


 On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:

> That does not work if you want to have Scala code in a file ( common
> functions) which you want to invoke in the note
> The alternative is to compile the code and then add the jar which
> would be normal for an application.
> But zeppelin is about scripting so this is a request I get very often
> from the users.
> Specially because the z.run does not work properly most of the times
> Carlos
>
> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>
>> Why not copying scala code in zeppelin and run the notebook directly ?
>>
>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>
>>> Dear all
>>> I have been  trying  to find a was to inject scala Code ( from
>>> String) into the spark interpreter
>>> In pyspark is easy with the exec function
>>> It should not be very difficult  to access from the Note scala repl
>>> interpreter but i could not find a way . I was even able to create a new
>>> repl session but then I could not bind the objects
>>> Any tips ?
>>> Thanks
>>> --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>
 --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>>
>
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Spark interpreter Repl injection

[Carlos Diogo]

Looks good Moon
Is there a specific reason why you needed the pyspark interpreter  to
access the spark interpreter? Could not the spark interpreter
programmatically access itself (and the same for the pyspark interpreter)

Would the issue be to expose the z.interpret() method?

Best regards
Carlos

On Tue, Mar 9, 2021 at 5:10 PM moon soo Lee  wrote:

> I see. If you want to specify a file, precode might not the best option.
> I found a hacky way to do it. Accessing SparkInterpreter instance object
> from PysparkInterpreter.
>
> %pyspark
> sparkIntpField = intp.getClass().getDeclaredField("sparkInterpreter")
> sparkIntpField.setAccessible(True)
> sparkIntp = sparkIntpField.get(intp)
> # run my scala code
> sparkIntp.interpret("val a=10", z.getInterpreterContext())
>
>
> See attached screenshot.
>
> [image: image.png]
>
> This is accessing internal variables outside the official API. So it may
> break at any time.
>
> I think it's better to expose interpret() method through
> 'ZeppelinContext'. So inside Note,
>
> z.interpret(any_string)
>
> can work without accessing this method in a hacky way.
> Please feel free to file an issue.
>
> Thanks,
> moon
>
>
>
>
> On Mon, Mar 8, 2021 at 10:23 PM Carlos Diogo  wrote:
>
>> Are you able to specify a file on the precode?
>> For now my work around is from within the note and with the rest api , to
>> add a paragraph with the code I want to inject ( which can come from a file
>> )
>> It works ok , but with run all or schedule the code gets updated in the
>> note , but the old Code still executes . Only on the next run it will take
>> effect
>>
>> On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:
>>
>>> Hi,
>>>
>>> How about precode
>>> ?
>>>  "zeppelin.SparkInterpreter.precode"
>>> can run scala code.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>> On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:
>>>
 That does not work if you want to have Scala code in a file ( common
 functions) which you want to invoke in the note
 The alternative is to compile the code and then add the jar which would
 be normal for an application.
 But zeppelin is about scripting so this is a request I get very often
 from the users.
 Specially because the z.run does not work properly most of the times
 Carlos

 On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:

> Why not copying scala code in zeppelin and run the notebook directly ?
>
> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>
>> Dear all
>> I have been  trying  to find a was to inject scala Code ( from
>> String) into the spark interpreter
>> In pyspark is easy with the exec function
>> It should not be very difficult  to access from the Note scala repl
>> interpreter but i could not find a way . I was even able to create a new
>> repl session but then I could not bind the objects
>> Any tips ?
>> Thanks
>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
 --
 Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
 Carlos Diogo

>>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
>

-- 
Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
Carlos Diogo

Re: Spark interpreter Repl injection

[moon soo Lee]

I see. If you want to specify a file, precode might not the best option.
I found a hacky way to do it. Accessing SparkInterpreter instance object
from PysparkInterpreter.

%pyspark
sparkIntpField = intp.getClass().getDeclaredField("sparkInterpreter")
sparkIntpField.setAccessible(True)
sparkIntp = sparkIntpField.get(intp)
# run my scala code
sparkIntp.interpret("val a=10", z.getInterpreterContext())


See attached screenshot.

[image: image.png]

This is accessing internal variables outside the official API. So it may
break at any time.

I think it's better to expose interpret() method through 'ZeppelinContext'.
So inside Note,

z.interpret(any_string)

can work without accessing this method in a hacky way.
Please feel free to file an issue.

Thanks,
moon




On Mon, Mar 8, 2021 at 10:23 PM Carlos Diogo  wrote:

> Are you able to specify a file on the precode?
> For now my work around is from within the note and with the rest api , to
> add a paragraph with the code I want to inject ( which can come from a file
> )
> It works ok , but with run all or schedule the code gets updated in the
> note , but the old Code still executes . Only on the next run it will take
> effect
>
> On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:
>
>> Hi,
>>
>> How about precode
>> ?
>>  "zeppelin.SparkInterpreter.precode"
>> can run scala code.
>>
>> Thanks,
>> moon
>>
>>
>> On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:
>>
>>> That does not work if you want to have Scala code in a file ( common
>>> functions) which you want to invoke in the note
>>> The alternative is to compile the code and then add the jar which would
>>> be normal for an application.
>>> But zeppelin is about scripting so this is a request I get very often
>>> from the users.
>>> Specially because the z.run does not work properly most of the times
>>> Carlos
>>>
>>> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>>>
 Why not copying scala code in zeppelin and run the notebook directly ?

 Carlos Diogo  于2021年3月6日周六 下午3:51写道：

> Dear all
> I have been  trying  to find a was to inject scala Code ( from String)
> into the spark interpreter
> In pyspark is easy with the exec function
> It should not be very difficult  to access from the Note scala repl
> interpreter but i could not find a way . I was even able to create a new
> repl session but then I could not bind the objects
> Any tips ?
> Thanks
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>


 --
 Best Regards

 Jeff Zhang

>>> --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Spark interpreter Repl injection

[Carlos Diogo]

Are you able to specify a file on the precode?
For now my work around is from within the note and with the rest api , to
add a paragraph with the code I want to inject ( which can come from a file
)
It works ok , but with run all or schedule the code gets updated in the
note , but the old Code still executes . Only on the next run it will take
effect

On Mon 8. Mar 2021 at 22:48, moon soo Lee  wrote:

> Hi,
>
> How about precode
> ?
>  "zeppelin.SparkInterpreter.precode"
> can run scala code.
>
> Thanks,
> moon
>
>
> On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:
>
>> That does not work if you want to have Scala code in a file ( common
>> functions) which you want to invoke in the note
>> The alternative is to compile the code and then add the jar which would
>> be normal for an application.
>> But zeppelin is about scripting so this is a request I get very often
>> from the users.
>> Specially because the z.run does not work properly most of the times
>> Carlos
>>
>> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>>
>>> Why not copying scala code in zeppelin and run the notebook directly ?
>>>
>>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>>
 Dear all
 I have been  trying  to find a was to inject scala Code ( from String)
 into the spark interpreter
 In pyspark is easy with the exec function
 It should not be very difficult  to access from the Note scala repl
 interpreter but i could not find a way . I was even able to create a new
 repl session but then I could not bind the objects
 Any tips ?
 Thanks
 --
 Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
 Carlos Diogo

>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
> --
Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
Carlos Diogo

Re: Spark interpreter Repl injection

[moon soo Lee]

Hi,

How about precode
?
"zeppelin.SparkInterpreter.precode"
can run scala code.

Thanks,
moon


On Sat, Mar 6, 2021 at 4:51 AM Carlos Diogo  wrote:

> That does not work if you want to have Scala code in a file ( common
> functions) which you want to invoke in the note
> The alternative is to compile the code and then add the jar which would be
> normal for an application.
> But zeppelin is about scripting so this is a request I get very often from
> the users.
> Specially because the z.run does not work properly most of the times
> Carlos
>
> On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:
>
>> Why not copying scala code in zeppelin and run the notebook directly ?
>>
>> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>>
>>> Dear all
>>> I have been  trying  to find a was to inject scala Code ( from String)
>>> into the spark interpreter
>>> In pyspark is easy with the exec function
>>> It should not be very difficult  to access from the Note scala repl
>>> interpreter but i could not find a way . I was even able to create a new
>>> repl session but then I could not bind the objects
>>> Any tips ?
>>> Thanks
>>> --
>>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>>> Carlos Diogo
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>

Re: Spark interpreter Repl injection

[Carlos Diogo]

That does not work if you want to have Scala code in a file ( common
functions) which you want to invoke in the note
The alternative is to compile the code and then add the jar which would be
normal for an application.
But zeppelin is about scripting so this is a request I get very often from
the users.
Specially because the z.run does not work properly most of the times
Carlos

On Sat 6. Mar 2021 at 11:36, Jeff Zhang  wrote:

> Why not copying scala code in zeppelin and run the notebook directly ?
>
> Carlos Diogo  于2021年3月6日周六 下午3:51写道：
>
>> Dear all
>> I have been  trying  to find a was to inject scala Code ( from String)
>> into the spark interpreter
>> In pyspark is easy with the exec function
>> It should not be very difficult  to access from the Note scala repl
>> interpreter but i could not find a way . I was even able to create a new
>> repl session but then I could not bind the objects
>> Any tips ?
>> Thanks
>> --
>> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
>> Carlos Diogo
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
-- 
Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
Carlos Diogo

Re: Spark interpreter Repl injection

[Jeff Zhang]

Why not copying scala code in zeppelin and run the notebook directly ?

Carlos Diogo  于2021年3月6日周六 下午3:51写道：

> Dear all
> I have been  trying  to find a was to inject scala Code ( from String)
> into the spark interpreter
> In pyspark is easy with the exec function
> It should not be very difficult  to access from the Note scala repl
> interpreter but i could not find a way . I was even able to create a new
> repl session but then I could not bind the objects
> Any tips ?
> Thanks
> --
> Os meus cumprimentos / Best regards /  Mit freundlichen Grüße
> Carlos Diogo
>


-- 
Best Regards

Jeff Zhang

Re: spark interpreter "master" parameter always resets to yarn-client after restart zeppelin

[Jeff Zhang]

Do you mean you will manually change master to yarn after zeppelin service
start, and want to reset it to yarn-client after restart zeppelin ?

Manuel Sopena Ballesteros  于2019年8月20日周二 上午8:01写道：

> Dear Zeppelin user community,
>
>
>
> I would like I a zeppelin installation with spark integration and the
> “master” parameter in the spark interpreter configuration always resets its
> value from “yarn” to “yarn-client” after zeppelin service reboot.
>
>
>
> How can I stop that?
>
>
>
> Thank you
>
>
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>


-- 
Best Regards

Jeff Zhang

Re: Spark Interpreter error: 'not found: type'

[Jeff Zhang]

I try it in master branch, it looks like it fails to download the
dependencies. and it also fails when I try use spark-submit directly.  It
should not be a zeppelin issue, please check these 2 dependencies.

Exception in thread "main" java.lang.RuntimeException: problem during
retrieve of org.apache.spark#spark-submit-parent:
java.lang.RuntimeException: Multiple artifacts of the module
org.bytedeco.javacpp-presets#openblas;0.2.19-1.3 are retrieved to the same
file! Update the retrieve pattern to fix this error. at
org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:249)
at
org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:83)
at org.apache.ivy.Ivy.retrieve(Ivy.java:551) at
org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1200)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:304)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153) at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by:
java.lang.RuntimeException: Multiple artifacts of the module
org.bytedeco.javacpp-presets#openblas;0.2.19-1.3 are retrieved to the same
file! Update the retrieve pattern to fix this error. at
org.apache.ivy.core.retrieve.RetrieveEngine.determineArtifactsToCopy(RetrieveEngine.java:417)
at
org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:118)
... 7 more at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)
at
org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:65)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:105)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:158)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)

Marcus 于2018年3月20日周二 上午8:01写道：

> Hi Karan,
>
> thanks for your hint, and sorry for the late response. I've tried the
> import using _root_ as suggested on stackoverflow, but it didn't change
> anything. Also, the import statement runs. The error occurs when using the
> classname.
>
> As for datavec-api, it is a transient dependency of deeplearning4j-core,
> which is loaded using %spark.dep. I also added it to the
> interpreter-settings as a dependency, with no different effect.
>
> Regards, Marcus
>
> On Wed, Mar 14, 2018 at 1:56 PM, Karan Sewani 
> wrote:
>
>> Hello Marcus
>>
>>
>> Maybe it has something to do with
>>
>>
>> https://stackoverflow.com/questions/13008792/how-to-import-class-using-fully-qualified-name
>> https://stackoverflow.com/questions/13008792/how-to-import-class-using-fully-qualified-name
>>
>>
>>
>> I
>> have implemented user defined functions in spark and used them in my code
>> with jar being loaded in classpath and i didn't have any issues with import.
>>
>>
>> Can you give me idea of how you are loading this jar datavec-api for
>> zeppelin or spark-submit to access?
>>
>>
>> Best
>>
>> Karan
>> --
>> *From:* Marcus 
>> *Sent:* Saturday, March 10, 2018 10:43:25 AM
>> *To:* users@zeppelin.apache.org
>> *Subject:* Spark Interpreter error: 'not found: type'
>>
>> Hi,
>>
>> I am new to Zeppelin and encountered a strange behavior. When copying my
>> running scala-code to a notebook, I've got errors from the spark
>> interpreter, saying it could not find some types. Strangely the code
>> worked, when I used the fqcn instead of the simple name.
>> But since I want the create a workflow for me, where I use my IDE to
>> write scala and transfer it to a notebook, I'd prefer to not be forced to
>> using fqcn.
>>
>> Here's an example:
>>
>>
>> | %spark.dep
>> | z.reset()
>> | z.load("org.deeplearning4j:deeplearning4j-core:0.9.1")
>> | z.load("org.nd4j:nd4j-native-platform:0.9.1")
>>
>> res0: org.apache.zeppelin.dep.Dependency =
>> org.apache.zeppelin.dep.Dependency@2e10d1e4
>>
>> | import org.datavec.api.records.reader.impl.FileRecordReader
>> |
>> | class Test extends FileRecordReader {
>> | }
>> |
>> | val t = new Test()
>>
>> import org.datavec.api.records.reader.impl.FileRecordReader
>> :12: error: not found: type FileRecordReader
>> class Test extends FileRecordReader {
>>
>> Thanks, Marcus
>>
>
>

Re: Spark Interpreter error: 'not found: type'

[Marcus]

Hi Karan,

thanks for your hint, and sorry for the late response. I've tried the
import using _root_ as suggested on stackoverflow, but it didn't change
anything. Also, the import statement runs. The error occurs when using the
classname.

As for datavec-api, it is a transient dependency of deeplearning4j-core,
which is loaded using %spark.dep. I also added it to the
interpreter-settings as a dependency, with no different effect.

Regards, Marcus

On Wed, Mar 14, 2018 at 1:56 PM, Karan Sewani 
wrote:

> Hello Marcus
>
>
> Maybe it has something to do with
>
> https://stackoverflow.com/questions/13008792/how-to-
> import-class-using-fully-qualified-namehttps://
> stackoverflow.com/questions/13008792/how-to-import-class-
> using-fully-qualified-name
>
>
>
> I
> have implemented user defined functions in spark and used them in my code
> with jar being loaded in classpath and i didn't have any issues with import.
>
>
> Can you give me idea of how you are loading this jar datavec-api for
> zeppelin or spark-submit to access?
>
>
> Best
>
> Karan
> --
> *From:* Marcus 
> *Sent:* Saturday, March 10, 2018 10:43:25 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Spark Interpreter error: 'not found: type'
>
> Hi,
>
> I am new to Zeppelin and encountered a strange behavior. When copying my
> running scala-code to a notebook, I've got errors from the spark
> interpreter, saying it could not find some types. Strangely the code
> worked, when I used the fqcn instead of the simple name.
> But since I want the create a workflow for me, where I use my IDE to write
> scala and transfer it to a notebook, I'd prefer to not be forced to using
> fqcn.
>
> Here's an example:
>
>
> | %spark.dep
> | z.reset()
> | z.load("org.deeplearning4j:deeplearning4j-core:0.9.1")
> | z.load("org.nd4j:nd4j-native-platform:0.9.1")
>
> res0: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.
> Dependency@2e10d1e4
>
> | import org.datavec.api.records.reader.impl.FileRecordReader
> |
> | class Test extends FileRecordReader {
> | }
> |
> | val t = new Test()
>
> import org.datavec.api.records.reader.impl.FileRecordReader
> :12: error: not found: type FileRecordReader
> class Test extends FileRecordReader {
>
> Thanks, Marcus
>

Re: Spark Interpreter error: 'not found: type'

[Karan Sewani]

Hello Marcus


Maybe it has something to do with

https://stackoverflow.com/questions/13008792/how-to-import-class-using-fully-qualified-namehttps://stackoverflow.com/questions/13008792/how-to-import-class-using-fully-qualified-name


I
 have implemented user defined functions in spark and used them in my code with 
jar being loaded in classpath and i didn't have any issues with import.


Can you give me idea of how you are loading this jar datavec-api for zeppelin 
or spark-submit to access?


Best

Karan


From: Marcus 
Sent: Saturday, March 10, 2018 10:43:25 AM
To: users@zeppelin.apache.org
Subject: Spark Interpreter error: 'not found: type'

Hi,

I am new to Zeppelin and encountered a strange behavior. When copying my 
running scala-code to a notebook, I've got errors from the spark interpreter, 
saying it could not find some types. Strangely the code worked, when I used the 
fqcn instead of the simple name.
But since I want the create a workflow for me, where I use my IDE to write 
scala and transfer it to a notebook, I'd prefer to not be forced to using fqcn.

Here's an example:


| %spark.dep
| z.reset()
| z.load("org.deeplearning4j:deeplearning4j-core:0.9.1")
| z.load("org.nd4j:nd4j-native-platform:0.9.1")

res0: org.apache.zeppelin.dep.Dependency = 
org.apache.zeppelin.dep.Dependency@2e10d1e4

| import org.datavec.api.records.reader.impl.FileRecordReader
|
| class Test extends FileRecordReader {
| }
|
| val t = new Test()

import org.datavec.api.records.reader.impl.FileRecordReader
:12: error: not found: type FileRecordReader
class Test extends FileRecordReader {

Thanks, Marcus

Re: Spark Interpreter: Change default scheduler pool

[Fabian Böhnlein]

Hi moon,

exactly, thanks for the pointer.

Added the issue: https://issues.apache.org/jira/browse/ZEPPELIN-2413

Best,
Fabian


On Tue, 28 Mar 2017 at 15:48 moon soo Lee  wrote:

> Hi Fabian,
>
> Thanks for sharing the issue.
> SparkSqlInterpreter set scheduler to "fair" depends on interpreter
> property [1]. I think we can do the similar for SparkInterpreter.
> Do you mind file a new JIRA issue for it?
>
> Regards,
> moon
>
> [1]
> https://github.com/apache/zeppelin/blob/0e1964877654c56c72473ad07dac1de6f9646816/spark/src/main/java/org/apache/zeppelin/spark/SparkSqlInterpreter.java#L98
>
>
> On Tue, Mar 28, 2017 at 5:24 AM Fabian Böhnlein <
> fabian.boehnl...@gmail.com> wrote:
>
>> Hi all,
>>
>> how can I change (globally, for Zeppelin) the default scheduler pool
>> which SparkInterpreter submits jobs to. Currently all jobs go into the pool
>> 'default' but I want them to go into the pool 'fair'.
>> We use "Per Note" and "scoped" processes for best resource sharing.
>>
>> "spark.scheduler.pool"="fair" in Interpreter Settings does not work,
>> should it?
>>
>> What works is
>> sc.setLocalProperty("spark.scheduler.pool","fair")
>> but it's required in every *note* (not just notebook) since it's on
>> thread level.
>>
>> Is there a possibility to globally/per notebook set the 'fair' pool as
>> the default pool?
>>
>> Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'.
>> Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.
>>
>> This is awesome and together with dynamicAllocation allows for super
>> flexible usage for multiple users but above behavior is a bit complicated.
>>
>> Thanks,
>> Fabian
>>
>>
>>
>>

Re: spark interpreter

[moon soo Lee]

Thanks for testing it.

When i run 0.6.0-rc1 bin-all binary, i can see CSV, TSV download buttons.
Could you try clear browser cache?

Regarding credential menu,
Ahyoung is working on improvement and documentation on
https://github.com/apache/zeppelin/pull/1100.

Thanks,
moon

On Fri, Jul 1, 2016 at 11:34 AM Benjamin Kim <bbuil...@gmail.com> wrote:

> Moon,
>
> I have downloaded and tested the bin-all tarball, and it has some
> deficiencies compared to the build-from-source version.
>
>- CSV, TSV download is missing
>- Doesn’t work with HBase 1.2 in CDH 5.7.0
>- Spark still does not work with Spark 1.6.0 in CDH 5.7.0 (JDK8)
>   - Using Livy is a good workaround
>- Doesn’t work with Phoenix 4.7 in CDH 5.7.0
>
>
> Everything else looks good especially in the area of multi-tenancy and
> security. I would like to know how to use the Credentials feature on
> securing usernames and passwords. I couldn’t find documentation on how.
>
> Thanks,
> Ben
>
> On Jul 1, 2016, at 9:04 AM, moon soo Lee <m...@apache.org> wrote:
>
> 0.6.0 is currently in vote in dev@ list.
>
> http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html
>
> Thanks,
> moon
>
> On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <l...@ca.ibm.com> wrote:
>
>> What is the expected day for v0.6?
>>
>>
>>
>>
>> From:    moon soo Lee <leemoon...@gmail.com>
>> To:users@zeppelin.apache.org
>> Date:2016/06/30 11:36 AM
>> Subject:Re: spark interpreter
>> --
>>
>>
>>
>> Hi Ben,
>>
>> Livy interpreter is included in 0.6.0. If it is not listed when you
>> create interpreter setting, could you check if your 'zeppelin.interpreters'
>> property list Livy interpreter classes? (conf/zeppelin-site.xml)
>>
>> Thanks,
>> moon
>>
>> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <*bbuil...@gmail.com*
>> <bbuil...@gmail.com>> wrote:
>> On a side note…
>>
>> Has anyone got the Livy interpreter to be added as an interpreter in the
>> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
>> Could this interfere?
>>
>> Thanks,
>> Ben
>>
>>
>> On Jun 29, 2016, at 11:18 AM, moon soo Lee <*m...@apache.org*
>> <m...@apache.org>> wrote:
>>
>> Livy interpreter internally creates multiple sessions for each user,
>> independently from 3 binding modes supported in Zeppelin.
>> Therefore, 'shared' mode, Livy interpreter will create sessions per each
>> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
>> per user.
>>
>> Notebook is shared among users, they always use the same interpreter
>> instance/process, for now. I think supporting per user interpreter
>> instance/process would be future work.
>>
>> Thanks,
>> moon
>>
>> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <*chen.song...@gmail.com*
>> <chen.song...@gmail.com>> wrote:
>> Thanks for your explanation, Moon.
>>
>> Following up on this, I can see the difference in terms of single or
>> multiple interpreter processes.
>>
>> With respect to spark drivers, since each interpreter spawns a separate
>> Spark driver in regular Spark interpreter setting, it is clear to me the
>> different implications of the 3 binding modes.
>>
>> However, when it comes to Livy server with impersonation turned on, I am
>> a bit confused. Will Livy interpreter always create a new Spark driver
>> (along with a Spark Context instance) for each user session, regardless of
>> the binding mode of Livy interpreter? I am not very familiar with Livy, but
>> from what I could tell, I see no difference between different binding modes
>> for Livy on as far as how Spark drivers are concerned.
>>
>> Last question, when a notebook is shared among users, will they always
>> use the same interpreter instance/process already created?
>>
>> Thanks
>> Chen
>>
>>
>>
>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <*m...@apache.org*
>> <m...@apache.org>> wrote:
>> Hi,
>>
>> Thanks for asking question. It's not dumb question at all, Zeppelin docs
>> does not explain very well.
>>
>> Spark Interpreter,
>>
>> 'shared' mode, a spark interpreter setting spawn a interpreter process to
>> serve all notebooks which binded to this interpreter setting.
>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
>> processes per notebook which binded to this interpreter setting.
>>
>> Using Livy interpreter,
>>
>> Zeppelin propagate current user information to Livy interpreter. And Livy
>> interpreter creates different session per user via Livy Server.
>>
>>
>> Hope this helps.
>>
>> Thanks,
>> moon
>>
>>
>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <*chen.song...@gmail.com*
>> <chen.song...@gmail.com>> wrote:
>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
>> `shared` Spark interpreter, every user share the same SparkContext. Sorry
>> for the dumb question, how does it differ from Spark via Ivy Server?
>>
>>
>> --
>> Chen Song
>>
>>
>>
>>
>>
>

Re: spark interpreter

[Benjamin Kim]

Moon,

I have downloaded and tested the bin-all tarball, and it has some deficiencies 
compared to the build-from-source version.
CSV, TSV download is missing
Doesn’t work with HBase 1.2 in CDH 5.7.0
Spark still does not work with Spark 1.6.0 in CDH 5.7.0 (JDK8)
Using Livy is a good workaround
Doesn’t work with Phoenix 4.7 in CDH 5.7.0

Everything else looks good especially in the area of multi-tenancy and 
security. I would like to know how to use the Credentials feature on securing 
usernames and passwords. I couldn’t find documentation on how.

Thanks,
Ben

> On Jul 1, 2016, at 9:04 AM, moon soo Lee <m...@apache.org> wrote:
> 
> 0.6.0 is currently in vote in dev@ list.
> http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html
>  
> <http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html>
> 
> Thanks,
> moon
> 
> On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <l...@ca.ibm.com 
> <mailto:l...@ca.ibm.com>> wrote:
> What is the expected day for v0.6?
> 
> 
> 
> 
> From:moon soo Lee <leemoon...@gmail.com <mailto:leemoon...@gmail.com>>
> To:users@zeppelin.apache.org <mailto:users@zeppelin.apache.org>
> Date:2016/06/30 11:36 AM
> Subject:Re: spark interpreter
> 
> 
> 
> Hi Ben,
> 
> Livy interpreter is included in 0.6.0. If it is not listed when you create 
> interpreter setting, could you check if your 'zeppelin.interpreters' property 
> list Livy interpreter classes? (conf/zeppelin-site.xml)
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <bbuil...@gmail.com 
> <mailto:bbuil...@gmail.com>> wrote:
> On a side note…
> 
> Has anyone got the Livy interpreter to be added as an interpreter in the 
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. 
> Could this interfere?
> 
> Thanks,
> Ben
> 
> 
> On Jun 29, 2016, at 11:18 AM, moon soo Lee <m...@apache.org 
> <mailto:m...@apache.org>> wrote:
> 
> Livy interpreter internally creates multiple sessions for each user, 
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each 
> user, 'scoped' or 'isolated' mode will result create sessions per notebook, 
> per user.
> 
> Notebook is shared among users, they always use the same interpreter 
> instance/process, for now. I think supporting per user interpreter 
> instance/process would be future work.
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <chen.song...@gmail.com 
> <mailto:chen.song...@gmail.com>> wrote:
> Thanks for your explanation, Moon.
> 
> Following up on this, I can see the difference in terms of single or multiple 
> interpreter processes. 
> 
> With respect to spark drivers, since each interpreter spawns a separate Spark 
> driver in regular Spark interpreter setting, it is clear to me the different 
> implications of the 3 binding modes.
> 
> However, when it comes to Livy server with impersonation turned on, I am a 
> bit confused. Will Livy interpreter always create a new Spark driver (along 
> with a Spark Context instance) for each user session, regardless of the 
> binding mode of Livy interpreter? I am not very familiar with Livy, but from 
> what I could tell, I see no difference between different binding modes for 
> Livy on as far as how Spark drivers are concerned.
> 
> Last question, when a notebook is shared among users, will they always use 
> the same interpreter instance/process already created?
> 
> Thanks
> Chen
> 
> 
> 
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <m...@apache.org 
> <mailto:m...@apache.org>> wrote:
> Hi,
> 
> Thanks for asking question. It's not dumb question at all, Zeppelin docs does 
> not explain very well.
> 
> Spark Interpreter, 
> 
> 'shared' mode, a spark interpreter setting spawn a interpreter process to 
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter 
> processes per notebook which binded to this interpreter setting.
> 
> Using Livy interpreter,
> 
> Zeppelin propagate current user information to Livy interpreter. And Livy 
> interpreter creates different session per user via Livy Server.
> 
> 
> Hope this helps.
> 
> Thanks,
> moon
> 
> 
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <chen.song...@gmail.com 
> <mailto:chen.song...@gmail.com>> wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
> `shared` Spark interpreter, every user share the same SparkContext. Sorry for 
> the dumb question, how does it differ from Spark via Ivy Server?
> 
> 
> -- 
> Chen Song
> 
> 
> 
> 

Re: spark interpreter

[moon soo Lee]

0.6.0 is currently in vote in dev@ list.
http://apache-zeppelin-dev-mailing-list.75694.x6.nabble.com/VOTE-Apache-Zeppelin-release-0-6-0-rc1-tp11505.html

Thanks,
moon

On Thu, Jun 30, 2016 at 1:54 PM Leon Katsnelson <l...@ca.ibm.com> wrote:

> What is the expected day for v0.6?
>
>
>
>
> From:moon soo Lee <leemoon...@gmail.com>
> To:users@zeppelin.apache.org
> Date:2016/06/30 11:36 AM
> Subject:Re: spark interpreter
> --
>
>
>
> Hi Ben,
>
> Livy interpreter is included in 0.6.0. If it is not listed when you create
> interpreter setting, could you check if your 'zeppelin.interpreters'
> property list Livy interpreter classes? (conf/zeppelin-site.xml)
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <*bbuil...@gmail.com*
> <bbuil...@gmail.com>> wrote:
> On a side note…
>
> Has anyone got the Livy interpreter to be added as an interpreter in the
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
> Could this interfere?
>
> Thanks,
> Ben
>
>
> On Jun 29, 2016, at 11:18 AM, moon soo Lee <*m...@apache.org*
> <m...@apache.org>> wrote:
>
> Livy interpreter internally creates multiple sessions for each user,
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each
> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
> per user.
>
> Notebook is shared among users, they always use the same interpreter
> instance/process, for now. I think supporting per user interpreter
> instance/process would be future work.
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song <*chen.song...@gmail.com*
> <chen.song...@gmail.com>> wrote:
> Thanks for your explanation, Moon.
>
> Following up on this, I can see the difference in terms of single or
> multiple interpreter processes.
>
> With respect to spark drivers, since each interpreter spawns a separate
> Spark driver in regular Spark interpreter setting, it is clear to me the
> different implications of the 3 binding modes.
>
> However, when it comes to Livy server with impersonation turned on, I am a
> bit confused. Will Livy interpreter always create a new Spark driver (along
> with a Spark Context instance) for each user session, regardless of the
> binding mode of Livy interpreter? I am not very familiar with Livy, but
> from what I could tell, I see no difference between different binding modes
> for Livy on as far as how Spark drivers are concerned.
>
> Last question, when a notebook is shared among users, will they always use
> the same interpreter instance/process already created?
>
> Thanks
> Chen
>
>
>
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <*m...@apache.org*
> <m...@apache.org>> wrote:
> Hi,
>
> Thanks for asking question. It's not dumb question at all, Zeppelin docs
> does not explain very well.
>
> Spark Interpreter,
>
> 'shared' mode, a spark interpreter setting spawn a interpreter process to
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
> processes per notebook which binded to this interpreter setting.
>
> Using Livy interpreter,
>
> Zeppelin propagate current user information to Livy interpreter. And Livy
> interpreter creates different session per user via Livy Server.
>
>
> Hope this helps.
>
> Thanks,
> moon
>
>
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song <*chen.song...@gmail.com*
> <chen.song...@gmail.com>> wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
> `shared` Spark interpreter, every user share the same SparkContext. Sorry
> for the dumb question, how does it differ from Spark via Ivy Server?
>
>
> --
> Chen Song
>
>
>
>
>

Re: spark interpreter

[Leon Katsnelson]

What is the expected day for v0.6?




From:   moon soo Lee <leemoon...@gmail.com>
To: users@zeppelin.apache.org
Date:   2016/06/30 11:36 AM
Subject:        Re: spark interpreter



Hi Ben,

Livy interpreter is included in 0.6.0. If it is not listed when you create 
interpreter setting, could you check if your 'zeppelin.interpreters' 
property list Livy interpreter classes? (conf/zeppelin-site.xml)

Thanks,
moon

On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim <bbuil...@gmail.com> wrote:
On a side note…

Has anyone got the Livy interpreter to be added as an interpreter in the 
latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication 
on. Could this interfere?

Thanks,
Ben


On Jun 29, 2016, at 11:18 AM, moon soo Lee <m...@apache.org> wrote:

Livy interpreter internally creates multiple sessions for each user, 
independently from 3 binding modes supported in Zeppelin.
Therefore, 'shared' mode, Livy interpreter will create sessions per each 
user, 'scoped' or 'isolated' mode will result create sessions per 
notebook, per user.

Notebook is shared among users, they always use the same interpreter 
instance/process, for now. I think supporting per user interpreter 
instance/process would be future work.

Thanks,
moon

On Wed, Jun 29, 2016 at 7:57 AM Chen Song <chen.song...@gmail.com> wrote:
Thanks for your explanation, Moon.

Following up on this, I can see the difference in terms of single or 
multiple interpreter processes. 

With respect to spark drivers, since each interpreter spawns a separate 
Spark driver in regular Spark interpreter setting, it is clear to me the 
different implications of the 3 binding modes.

However, when it comes to Livy server with impersonation turned on, I am a 
bit confused. Will Livy interpreter always create a new Spark driver 
(along with a Spark Context instance) for each user session, regardless of 
the binding mode of Livy interpreter? I am not very familiar with Livy, 
but from what I could tell, I see no difference between different binding 
modes for Livy on as far as how Spark drivers are concerned.

Last question, when a notebook is shared among users, will they always use 
the same interpreter instance/process already created?

Thanks
Chen



On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee <m...@apache.org> wrote:
Hi,

Thanks for asking question. It's not dumb question at all, Zeppelin docs 
does not explain very well.

Spark Interpreter, 

'shared' mode, a spark interpreter setting spawn a interpreter process to 
serve all notebooks which binded to this interpreter setting.
'scoped' mode, a spark interpreter setting spawn multiple interpreter 
processes per notebook which binded to this interpreter setting.

Using Livy interpreter,

Zeppelin propagate current user information to Livy interpreter. And Livy 
interpreter creates different session per user via Livy Server.


Hope this helps.

Thanks,
moon


On Tue, Jun 21, 2016 at 6:41 PM Chen Song <chen.song...@gmail.com> wrote:
Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
`shared` Spark interpreter, every user share the same SparkContext. Sorry 
for the dumb question, how does it differ from Spark via Ivy Server?


-- 
Chen Song

Re: spark interpreter

[Jongyoul Lee]

Hi Ben,

I suggest you stop Z, remove conf/interpreter.json, and start Z again.

Regards,
JL

On Friday, 1 July 2016, moon soo Lee  wrote:

> Hi Ben,
>
> Livy interpreter is included in 0.6.0. If it is not listed when you create
> interpreter setting, could you check if your 'zeppelin.interpreters'
> property list Livy interpreter classes? (conf/zeppelin-site.xml)
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim  > wrote:
>
>> On a side note…
>>
>> Has anyone got the Livy interpreter to be added as an interpreter in the
>> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
>> Could this interfere?
>>
>> Thanks,
>> Ben
>>
>>
>> On Jun 29, 2016, at 11:18 AM, moon soo Lee > > wrote:
>>
>> Livy interpreter internally creates multiple sessions for each user,
>> independently from 3 binding modes supported in Zeppelin.
>> Therefore, 'shared' mode, Livy interpreter will create sessions per each
>> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
>> per user.
>>
>> Notebook is shared among users, they always use the same interpreter
>> instance/process, for now. I think supporting per user interpreter
>> instance/process would be future work.
>>
>> Thanks,
>> moon
>>
>> On Wed, Jun 29, 2016 at 7:57 AM Chen Song > > wrote:
>>
>>> Thanks for your explanation, Moon.
>>>
>>> Following up on this, I can see the difference in terms of single or
>>> multiple interpreter processes.
>>>
>>> With respect to spark drivers, since each interpreter spawns a separate
>>> Spark driver in regular Spark interpreter setting, it is clear to me the
>>> different implications of the 3 binding modes.
>>>
>>> However, when it comes to Livy server with impersonation turned on, I am
>>> a bit confused. Will Livy interpreter always create a new Spark driver
>>> (along with a Spark Context instance) for each user session, regardless of
>>> the binding mode of Livy interpreter? I am not very familiar with Livy, but
>>> from what I could tell, I see no difference between different binding modes
>>> for Livy on as far as how Spark drivers are concerned.
>>>
>>> Last question, when a notebook is shared among users, will they always
>>> use the same interpreter instance/process already created?
>>>
>>> Thanks
>>> Chen
>>>
>>>
>>>
>>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee >> > wrote:
>>>
 Hi,

 Thanks for asking question. It's not dumb question at all, Zeppelin
 docs does not explain very well.

 Spark Interpreter,

 'shared' mode, a spark interpreter setting spawn a interpreter process
 to serve all notebooks which binded to this interpreter setting.
 'scoped' mode, a spark interpreter setting spawn multiple interpreter
 processes per notebook which binded to this interpreter setting.

 Using Livy interpreter,

 Zeppelin propagate current user information to Livy interpreter. And
 Livy interpreter creates different session per user via Livy Server.


 Hope this helps.

 Thanks,
 moon


 On Tue, Jun 21, 2016 at 6:41 PM Chen Song > wrote:

> Zeppelin provides 3 binding modes for each interpreter. With `scoped`
> or `shared` Spark interpreter, every user share the same SparkContext.
> Sorry for the dumb question, how does it differ from Spark via Ivy Server?
>
>
> --
> Chen Song
>
>
>>

-- 
이종열, Jongyoul Lee, 李宗烈
http://madeng.net

Re: spark interpreter

[Benjamin Kim]

Moon,

That worked! There were quite a few more configuration properties added, so I 
added those too in both zeppelin-site.xml and zeppelin-env.sh. But, now, I’m 
getting errors starting a spark context.

Thanks,
Ben

> On Jun 30, 2016, at 8:10 AM, moon soo Lee  wrote:
> 
> Hi Ben,
> 
> Livy interpreter is included in 0.6.0. If it is not listed when you create 
> interpreter setting, could you check if your 'zeppelin.interpreters' property 
> list Livy interpreter classes? (conf/zeppelin-site.xml)
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim  > wrote:
> On a side note…
> 
> Has anyone got the Livy interpreter to be added as an interpreter in the 
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. 
> Could this interfere?
> 
> Thanks,
> Ben
> 
> 
>> On Jun 29, 2016, at 11:18 AM, moon soo Lee > > wrote:
>> 
>> Livy interpreter internally creates multiple sessions for each user, 
>> independently from 3 binding modes supported in Zeppelin.
>> Therefore, 'shared' mode, Livy interpreter will create sessions per each 
>> user, 'scoped' or 'isolated' mode will result create sessions per notebook, 
>> per user.
>> 
>> Notebook is shared among users, they always use the same interpreter 
>> instance/process, for now. I think supporting per user interpreter 
>> instance/process would be future work.
>> 
>> Thanks,
>> moon
>> 
>> On Wed, Jun 29, 2016 at 7:57 AM Chen Song > > wrote:
>> Thanks for your explanation, Moon.
>> 
>> Following up on this, I can see the difference in terms of single or 
>> multiple interpreter processes. 
>> 
>> With respect to spark drivers, since each interpreter spawns a separate 
>> Spark driver in regular Spark interpreter setting, it is clear to me the 
>> different implications of the 3 binding modes.
>> 
>> However, when it comes to Livy server with impersonation turned on, I am a 
>> bit confused. Will Livy interpreter always create a new Spark driver (along 
>> with a Spark Context instance) for each user session, regardless of the 
>> binding mode of Livy interpreter? I am not very familiar with Livy, but from 
>> what I could tell, I see no difference between different binding modes for 
>> Livy on as far as how Spark drivers are concerned.
>> 
>> Last question, when a notebook is shared among users, will they always use 
>> the same interpreter instance/process already created?
>> 
>> Thanks
>> Chen
>> 
>> 
>> 
>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee > > wrote:
>> Hi,
>> 
>> Thanks for asking question. It's not dumb question at all, Zeppelin docs 
>> does not explain very well.
>> 
>> Spark Interpreter, 
>> 
>> 'shared' mode, a spark interpreter setting spawn a interpreter process to 
>> serve all notebooks which binded to this interpreter setting.
>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter 
>> processes per notebook which binded to this interpreter setting.
>> 
>> Using Livy interpreter,
>> 
>> Zeppelin propagate current user information to Livy interpreter. And Livy 
>> interpreter creates different session per user via Livy Server.
>> 
>> 
>> Hope this helps.
>> 
>> Thanks,
>> moon
>> 
>> 
>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song > > wrote:
>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
>> `shared` Spark interpreter, every user share the same SparkContext. Sorry 
>> for the dumb question, how does it differ from Spark via Ivy Server?
>> 
>> 
>> -- 
>> Chen Song
>> 
> 

Re: spark interpreter

[moon soo Lee]

Hi Ben,

Livy interpreter is included in 0.6.0. If it is not listed when you create
interpreter setting, could you check if your 'zeppelin.interpreters'
property list Livy interpreter classes? (conf/zeppelin-site.xml)

Thanks,
moon

On Wed, Jun 29, 2016 at 11:52 AM Benjamin Kim  wrote:

> On a side note…
>
> Has anyone got the Livy interpreter to be added as an interpreter in the
> latest build of Zeppelin 0.6.0? By the way, I have Shiro authentication on.
> Could this interfere?
>
> Thanks,
> Ben
>
>
> On Jun 29, 2016, at 11:18 AM, moon soo Lee  wrote:
>
> Livy interpreter internally creates multiple sessions for each user,
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each
> user, 'scoped' or 'isolated' mode will result create sessions per notebook,
> per user.
>
> Notebook is shared among users, they always use the same interpreter
> instance/process, for now. I think supporting per user interpreter
> instance/process would be future work.
>
> Thanks,
> moon
>
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song  wrote:
>
>> Thanks for your explanation, Moon.
>>
>> Following up on this, I can see the difference in terms of single or
>> multiple interpreter processes.
>>
>> With respect to spark drivers, since each interpreter spawns a separate
>> Spark driver in regular Spark interpreter setting, it is clear to me the
>> different implications of the 3 binding modes.
>>
>> However, when it comes to Livy server with impersonation turned on, I am
>> a bit confused. Will Livy interpreter always create a new Spark driver
>> (along with a Spark Context instance) for each user session, regardless of
>> the binding mode of Livy interpreter? I am not very familiar with Livy, but
>> from what I could tell, I see no difference between different binding modes
>> for Livy on as far as how Spark drivers are concerned.
>>
>> Last question, when a notebook is shared among users, will they always
>> use the same interpreter instance/process already created?
>>
>> Thanks
>> Chen
>>
>>
>>
>> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee  wrote:
>>
>>> Hi,
>>>
>>> Thanks for asking question. It's not dumb question at all, Zeppelin docs
>>> does not explain very well.
>>>
>>> Spark Interpreter,
>>>
>>> 'shared' mode, a spark interpreter setting spawn a interpreter process
>>> to serve all notebooks which binded to this interpreter setting.
>>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
>>> processes per notebook which binded to this interpreter setting.
>>>
>>> Using Livy interpreter,
>>>
>>> Zeppelin propagate current user information to Livy interpreter. And
>>> Livy interpreter creates different session per user via Livy Server.
>>>
>>>
>>> Hope this helps.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song 
>>> wrote:
>>>
 Zeppelin provides 3 binding modes for each interpreter. With `scoped`
 or `shared` Spark interpreter, every user share the same SparkContext.
 Sorry for the dumb question, how does it differ from Spark via Ivy Server?


 --
 Chen Song


>

Re: spark interpreter

[Benjamin Kim]

On a side note…

Has anyone got the Livy interpreter to be added as an interpreter in the latest 
build of Zeppelin 0.6.0? By the way, I have Shiro authentication on. Could this 
interfere?

Thanks,
Ben


> On Jun 29, 2016, at 11:18 AM, moon soo Lee  wrote:
> 
> Livy interpreter internally creates multiple sessions for each user, 
> independently from 3 binding modes supported in Zeppelin.
> Therefore, 'shared' mode, Livy interpreter will create sessions per each 
> user, 'scoped' or 'isolated' mode will result create sessions per notebook, 
> per user.
> 
> Notebook is shared among users, they always use the same interpreter 
> instance/process, for now. I think supporting per user interpreter 
> instance/process would be future work.
> 
> Thanks,
> moon
> 
> On Wed, Jun 29, 2016 at 7:57 AM Chen Song  > wrote:
> Thanks for your explanation, Moon.
> 
> Following up on this, I can see the difference in terms of single or multiple 
> interpreter processes. 
> 
> With respect to spark drivers, since each interpreter spawns a separate Spark 
> driver in regular Spark interpreter setting, it is clear to me the different 
> implications of the 3 binding modes.
> 
> However, when it comes to Livy server with impersonation turned on, I am a 
> bit confused. Will Livy interpreter always create a new Spark driver (along 
> with a Spark Context instance) for each user session, regardless of the 
> binding mode of Livy interpreter? I am not very familiar with Livy, but from 
> what I could tell, I see no difference between different binding modes for 
> Livy on as far as how Spark drivers are concerned.
> 
> Last question, when a notebook is shared among users, will they always use 
> the same interpreter instance/process already created?
> 
> Thanks
> Chen
> 
> 
> 
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee  > wrote:
> Hi,
> 
> Thanks for asking question. It's not dumb question at all, Zeppelin docs does 
> not explain very well.
> 
> Spark Interpreter, 
> 
> 'shared' mode, a spark interpreter setting spawn a interpreter process to 
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter 
> processes per notebook which binded to this interpreter setting.
> 
> Using Livy interpreter,
> 
> Zeppelin propagate current user information to Livy interpreter. And Livy 
> interpreter creates different session per user via Livy Server.
> 
> 
> Hope this helps.
> 
> Thanks,
> moon
> 
> 
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song  > wrote:
> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or 
> `shared` Spark interpreter, every user share the same SparkContext. Sorry for 
> the dumb question, how does it differ from Spark via Ivy Server?
> 
> 
> -- 
> Chen Song
> 

Re: spark interpreter

[moon soo Lee]

Livy interpreter internally creates multiple sessions for each user,
independently from 3 binding modes supported in Zeppelin.
Therefore, 'shared' mode, Livy interpreter will create sessions per each
user, 'scoped' or 'isolated' mode will result create sessions per notebook,
per user.

Notebook is shared among users, they always use the same interpreter
instance/process, for now. I think supporting per user interpreter
instance/process would be future work.

Thanks,
moon

On Wed, Jun 29, 2016 at 7:57 AM Chen Song  wrote:

> Thanks for your explanation, Moon.
>
> Following up on this, I can see the difference in terms of single or
> multiple interpreter processes.
>
> With respect to spark drivers, since each interpreter spawns a separate
> Spark driver in regular Spark interpreter setting, it is clear to me the
> different implications of the 3 binding modes.
>
> However, when it comes to Livy server with impersonation turned on, I am a
> bit confused. Will Livy interpreter always create a new Spark driver (along
> with a Spark Context instance) for each user session, regardless of the
> binding mode of Livy interpreter? I am not very familiar with Livy, but
> from what I could tell, I see no difference between different binding modes
> for Livy on as far as how Spark drivers are concerned.
>
> Last question, when a notebook is shared among users, will they always use
> the same interpreter instance/process already created?
>
> Thanks
> Chen
>
>
>
> On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee  wrote:
>
>> Hi,
>>
>> Thanks for asking question. It's not dumb question at all, Zeppelin docs
>> does not explain very well.
>>
>> Spark Interpreter,
>>
>> 'shared' mode, a spark interpreter setting spawn a interpreter process to
>> serve all notebooks which binded to this interpreter setting.
>> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
>> processes per notebook which binded to this interpreter setting.
>>
>> Using Livy interpreter,
>>
>> Zeppelin propagate current user information to Livy interpreter. And Livy
>> interpreter creates different session per user via Livy Server.
>>
>>
>> Hope this helps.
>>
>> Thanks,
>> moon
>>
>>
>> On Tue, Jun 21, 2016 at 6:41 PM Chen Song  wrote:
>>
>>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
>>> `shared` Spark interpreter, every user share the same SparkContext. Sorry
>>> for the dumb question, how does it differ from Spark via Ivy Server?
>>>
>>>
>>> --
>>> Chen Song
>>>
>>>

Re: spark interpreter

[Chen Song]

Thanks for your explanation, Moon.

Following up on this, I can see the difference in terms of single or
multiple interpreter processes.

With respect to spark drivers, since each interpreter spawns a separate
Spark driver in regular Spark interpreter setting, it is clear to me the
different implications of the 3 binding modes.

However, when it comes to Livy server with impersonation turned on, I am a
bit confused. Will Livy interpreter always create a new Spark driver (along
with a Spark Context instance) for each user session, regardless of the
binding mode of Livy interpreter? I am not very familiar with Livy, but
from what I could tell, I see no difference between different binding modes
for Livy on as far as how Spark drivers are concerned.

Last question, when a notebook is shared among users, will they always use
the same interpreter instance/process already created?

Thanks
Chen



On Fri, Jun 24, 2016 at 11:51 AM moon soo Lee  wrote:

> Hi,
>
> Thanks for asking question. It's not dumb question at all, Zeppelin docs
> does not explain very well.
>
> Spark Interpreter,
>
> 'shared' mode, a spark interpreter setting spawn a interpreter process to
> serve all notebooks which binded to this interpreter setting.
> 'scoped' mode, a spark interpreter setting spawn multiple interpreter
> processes per notebook which binded to this interpreter setting.
>
> Using Livy interpreter,
>
> Zeppelin propagate current user information to Livy interpreter. And Livy
> interpreter creates different session per user via Livy Server.
>
>
> Hope this helps.
>
> Thanks,
> moon
>
>
> On Tue, Jun 21, 2016 at 6:41 PM Chen Song  wrote:
>
>> Zeppelin provides 3 binding modes for each interpreter. With `scoped` or
>> `shared` Spark interpreter, every user share the same SparkContext. Sorry
>> for the dumb question, how does it differ from Spark via Ivy Server?
>>
>>
>> --
>> Chen Song
>>
>>