[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

2017-11-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249087#comment-16249087
 ] 

ASF GitHub Bot commented on TOREE-430:
--

Github user Myllyenko closed the pull request at:

https://github.com/apache/incubator-toree/pull/129


> ClassNotFoundException with lambdas on CDH 5.10.0
> -
>
> Key: TOREE-430
> URL: https://issues.apache.org/jira/browse/TOREE-430
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0
> Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined 
> and compiled in REPL. For instance, the following Spark example fails with 
> ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store 
> runtime-compiled classes is different from a location where classloader tries 
> to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

2017-08-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123256#comment-16123256
 ] 

ASF GitHub Bot commented on TOREE-430:
--

Github user Myllyenko commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/129#discussion_r132670358
  
--- Diff: 
scala-interpreter/src/main/scala-2.10/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala
 ---
@@ -61,6 +61,7 @@ trait ScalaInterpreterSpecific { this: ScalaInterpreter =>
   ): SparkIMain = {
 val s = new SparkIMain(settings, out)
 s.initializeSynchronous()
+System.setProperty("spark.repl.class.outputDir", 
s.getClassOutputDirectory.getAbsolutePath)
--- End diff --

I've tried to figure out how to do that without heavy code modifications 
but haven't succeeded.

It worth noting that currently in a Scala 2.11 environment Toree [sets this 
property in a similar 
manner](https://github.com/apache/incubator-toree/blob/master/kernel/src/main/scala/org/apache/toree/boot/KernelBootstrap.scala#L70).
 And in Scala 2.11 we can pass it directly to `SparkConf` with ease because 
`spark.repl.class.outputDir`'s required value becomes known very early ([when 
`org.apache.spark.repl.Main` is 
initialised](https://github.com/apache/spark/blob/branch-1.6/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala#L33)).

But in a Scala 2.10 environment, things are different. 
`spark.repl.class.outputDir`' [is initialised only with 
`org.apache.spark.repl.SparkIMain`](https://github.com/apache/spark/blob/branch-2.2/repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkIMain.scala#L100),
 that is after creation of `ScalaInterpreter` in Toree. But in order to pass 
`spark.repl.class.outputDir` directly to `SparkConf` we have to 
initialise`SparkIMain` earlier than Toree's `Kernel`.

This is applicable for both Spark 2 and Spark 1.6.


> ClassNotFoundException with lambdas on CDH 5.10.0
> -
>
> Key: TOREE-430
> URL: https://issues.apache.org/jira/browse/TOREE-430
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0
> Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined 
> and compiled in REPL. For instance, the following Spark example fails with 
> ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store 
> runtime-compiled classes is different from a location where classloader tries 
> to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

2017-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120375#comment-16120375
 ] 

ASF GitHub Bot commented on TOREE-430:
--

Github user lresende commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/129#discussion_r132261271
  
--- Diff: 
scala-interpreter/src/main/scala-2.10/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala
 ---
@@ -61,6 +61,7 @@ trait ScalaInterpreterSpecific { this: ScalaInterpreter =>
   ): SparkIMain = {
 val s = new SparkIMain(settings, out)
 s.initializeSynchronous()
+System.setProperty("spark.repl.class.outputDir", 
s.getClassOutputDirectory.getAbsolutePath)
--- End diff --

Could we pass this as conf while creating the Spark Session? 


> ClassNotFoundException with lambdas on CDH 5.10.0
> -
>
> Key: TOREE-430
> URL: https://issues.apache.org/jira/browse/TOREE-430
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0
> Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined 
> and compiled in REPL. For instance, the following Spark example fails with 
> ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store 
> runtime-compiled classes is different from a location where classloader tries 
> to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-430) ClassNotFoundException with lambdas on CDH 5.10.0

2017-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/TOREE-430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119632#comment-16119632
 ] 

ASF GitHub Bot commented on TOREE-430:
--

GitHub user Myllyenko opened a pull request:

https://github.com/apache/incubator-toree/pull/129

Fix for TOREE-430

To fix TOREE-430, REPL-generated classes must be made visible to a 
classloader.

The only working way I found is to set the `spark.repl.class.outputDir` 
system property (and subsequently the corresponding Spark configuration 
property) to a path to an output directory of a REPL compiler. To achieve that, 
I had to move `SparkIMain` initialisation before creation of `SparkContext`.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Myllyenko/incubator-toree TOREE-430

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #129


commit 2de78a0c50237bd2166843fd89982022d45a41b0
Author: Igor Melnichenko 
Date:   2017-08-04T11:11:13Z

Fixed TOREE-430 by setting the spark.repl.class.outputDir system property 
to a path to an output directory of a REPL compiler in the Scala 2.10 
environment.

(cherry picked from commit 72261a6eb9759b0f0cae89793fdac25976d10c48)

# Conflicts:
#   
scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala




> ClassNotFoundException with lambdas on CDH 5.10.0
> -
>
> Key: TOREE-430
> URL: https://issues.apache.org/jira/browse/TOREE-430
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.1.0
> Environment: CDH 5.10.0 (Spark 1.6.0, Scala 2.10.5)
>Reporter: Igor Melnichenko
>
> In the specified environment, Toree classloader fails to load classes defined 
> and compiled in REPL. For instance, the following Spark example fails with 
> ClassNotFoundException and works fine if the lambda is removed:
> {code:title=Example.scala|borderStyle=solid}
> val NUM_SAMPLES = 100
> val count = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
>   val x = math.random
>   val y = math.random
>   x*x + y*y < 1
> }.count()
> println(s"Pi is roughly ${4.0 * count / NUM_SAMPLES}")
> {code}
> Cause: a location of a directory used by a REPL compiler to store 
> runtime-compiled classes is different from a location where classloader tries 
> to find these classes.
> PR with fix will be opened soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)