[jira] [Created] (TOREE-461) Compile Time error for scala.Tuple3

[Randy Gelhausen (JIRA)]

Randy Gelhausen created TOREE-461:
-

 Summary: Compile Time error for scala.Tuple3
 Key: TOREE-461
 URL: https://issues.apache.org/jira/browse/TOREE-461
 Project: TOREE
  Issue Type: Bug
  Components: Kernel
Affects Versions: 0.2.0
Reporter: Randy Gelhausen


Using Toree 0.2 RC2, I am unable to execute the following Jupyter cell:

```
def time[A](f: => A) = {
  val s = System.nanoTime
  val ret = f
  println("time: "+(System.nanoTime-s)/1e6+"ms")
  ret
}
```

[I 21:22:28.027 NotebookApp] Saving file at /Scala.ipynb
17/12/14 21:22:59 ERROR OneForOneStrategy: compile-time error
javax.script.ScriptException: compile-time error
at scala.tools.nsc.interpreter.IMain.compiled(IMain.scala:602)
at scala.tools.nsc.interpreter.IMain.eval(IMain.scala:1074)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreterSpecific$class.read(ScalaInterpreterSpecific.scala:250)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.read(ScalaInterpreter.scala:43)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$prepareResult$1.apply(ScalaInterpreter.scala:206)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$prepareResult$1.apply(ScalaInterpreter.scala:205)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter.prepareResult(ScalaInterpreter.scala:205)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretMapToResultAndOutput$1.apply(ScalaInterpreter.scala:285)
at 
org.apache.toree.kernel.interpreter.scala.ScalaInterpreter$$anonfun$interpretMapToResultAndOutput$1.apply(ScalaInterpreter.scala:280)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at 
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[I 21:23:34.640 NotebookApp] Starting buffering for 
31b4d0ec-640b-4200-a75e-cc446536f939:CD995E798E5043EE89DE08C79EF9D270
17/12/14 21:23:35 WARN KernelOutputStream: Suppressing empty output: ''
17/12/14 21:23:35 ERROR InterpreterActor: Could not execute code def time[A](f: 
=> A) = {
  val s = System.nanoTime
  val ret = f
  println("time: "+(System.nanoTime-s)/1e6+"ms")
  ret
} because of exception: Ask timed out on 
[Actor[akka://spark-kernel-actor-system/user/interpreter/execute_request_task#530568081]]
 after [21474835000 ms]. 
Sender[Actor[akka://spark-kernel-actor-system/user/interpreter#-1495230403]] 
sent message of type "scala.Tuple3".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (TOREE-439) Brunel Visualization no longer working with Apache Toree Kernel

[Luciano Resende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luciano Resende resolved TOREE-439.
---
   Resolution: Resolved
 Assignee: Luciano Resende
Fix Version/s: 0.2.0

I can reproduce this up to Brunel 2.4, but the Brunel 2.5 seems to have 
resolved the issue (Brunel 2.5 and Spark 2.2.0 with Toree Master)

> Brunel Visualization no longer working with Apache Toree Kernel 
> 
>
> Key: TOREE-439
> URL: https://issues.apache.org/jira/browse/TOREE-439
> Project: TOREE
>  Issue Type: Bug
> Environment: Brunel 2.3 or 2.4
> Apache Toree (most recent in master branch) 
> Spark 2.1.1
>Reporter: Yunli Tang
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> When running a jupyter notebook with Apache Toree - Scala kernel and Brunel 
> 2.3 , the brunel commands worked perfectly. We recently updated our toree to 
> the most recent source code (what is currently in master) And I seem to run 
> into the following problem when running any brunel command. For instance if I 
> run,
> {code}
> %%brunel data('df') x(_1) y(_2) bar style("fill:red") filter(_1:2) :: 
> width=300, height=300
> {code}
> I get the following error:
> {code}
> 17/09/18 18:04:14 [WARN] o.a.t.m.MagicManager - Magic brunel failed to 
> execute with error: 
> org.apache.toree.magic.package$.CellMagicOutput()Lscala/collection/immutable/Map$;
> java.lang.NoSuchMethodError: 
> org.apache.toree.magic.package$.CellMagicOutput()Lscala/collection/immutable/Map$;
>   at org.brunel.toree.Brunel.execute(Brunel.scala:134)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.toree.plugins.PluginMethod$$anonfun$invoke$2.apply(PluginMethod.scala:116)
>   at scala.util.Try$.apply(Try.scala:192)
>   at org.apache.toree.plugins.PluginMethod.invoke(PluginMethod.scala:84)
>   at 
> org.apache.toree.plugins.PluginManager$$anonfun$8.apply(PluginManager.scala:334)
>   at 
> org.apache.toree.plugins.PluginManager$$anonfun$8.apply(PluginManager.scala:333)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.List.foreach(List.scala:381)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.immutable.List.map(List.scala:285)
>   at 
> org.apache.toree.plugins.PluginManager.invokePluginMethods(PluginManager.scala:333)
>   at 
> org.apache.toree.plugins.PluginManager.fireEvent(PluginManager.scala:300)
>   at 
> org.apache.toree.plugins.PluginManager.fireEvent(PluginManager.scala:283)
>   at 
> org.apache.toree.plugins.PluginManager.fireEventFirstResult(PluginManager.scala:250)
>   at 
> org.apache.toree.magic.MagicManager.applyDynamic(MagicManager.scala:82)
>   at 
> $line35.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:40)
>   at 
> $line35.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:45)
>   at $line35.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:47)
>   at $line35.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:49)
>   at $line35.$read$$iw$$iw$$iw$$iw$$iw$$iw.(:51)
>   at $line35.$read$$iw$$iw$$iw$$iw$$iw.(:53)
>   at $line35.$read$$iw$$iw$$iw$$iw.(:55)
>   at $line35.$read$$iw$$iw$$iw.(:57)
>   at $line35.$read$$iw$$iw.(:59)
>   at $line35.$read$$iw.(:61)
>   at $line35.$read.(:63)
>   at $line35.$read$.(:67)
>   at $line35.$read$.()
>   at $line35.$eval$.$print$lzycompute(:7)
>   at $line35.$eval$.$print(:6)
>   at $line35.$eval.$print()
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
>   at 
> scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
>   at 
> scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
>   at 
> scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
>   at 
> scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
>   at 
> 

[jira] [Reopened] (TOREE-459) Failure when adding a Jar.

[Naveen Swamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Swamy reopened TOREE-459:


Please see the error I pasted in the comment.

> Failure when adding a Jar.
> --
>
> Key: TOREE-459
> URL: https://issues.apache.org/jira/browse/TOREE-459
> Project: TOREE
>  Issue Type: Bug
>Reporter: Naveen Swamy
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> I am trying to use toree with Spark 2.2.7(latest) and Scala 2.11 however when 
> I add a jar it fails with the error below
> {{
> spark-2.2.7/jars/scala-library-2.11.8.jar(scala/reflect/package.class)' is 
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> error: error while loading package, class file 
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/package.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> error: error while loading Serializable, class file 
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Serializable.class)' is 
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> res2: 
> Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput]
>  = Right(())
> error: error while loading Predef, class file 
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Predef.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> }}
> The jar I am trying to add is:
> %AddJar 
> https://search.maven.org/remotecontent?filepath=ml/dmlc/mxnet/mxnet-full_2.11-osx-x86_64-cpu/0.9.3a/mxnet-full_2.11-osx-x86_64-cpu-0.9.3a.jar
> Spark version 2.2.1
> Scala version 2.11.8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TOREE-457) spark context seen corrupted after load KAfka libraries

[Luciano Resende (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291730#comment-16291730
 ] 

Luciano Resende commented on TOREE-457:
---

This is a very old version of Toree and most likely was not built with Spark 
2.2.0, I would recommend you building your own toree distribution like :

APACHE_SPARK_VERSION=2.2.0 BASE_VERSION=0.2.0  IS_SNAPSHOT=False make clean 
release

And then installing from pip package created in the dist folder.

Or try installing from the Toree 0.2 RC

https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0-incubating-rc2/toree-pip/toree-0.2.0.tar.gz

> spark context seen corrupted after load KAfka libraries
> ---
>
> Key: TOREE-457
> URL: https://issues.apache.org/jira/browse/TOREE-457
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Reporter: fxoSa
>Priority: Minor
>
> I am trying to set up a jupyter notebook (apache-toree Scala) to access kafka 
> logs from spark a streaming.
> First I add dependencies using AddDeps:
>  
> {code:java}
> %AddDeps org.apache.spark spark-streaming-kafka-0-10_2.11 2.2.0. 
> Marking org.apache.spark:spark-streaming-kafka-0-10_2.11:2.2.0 for download 
> Preparing to fetch from:
> -> file:/tmp/toree_add_deps8235567186565695423/
> -> https://repo1.maven.org/maven2
> -> New file at 
> /tmp/toree_add_deps8235567186565695423/https/repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka-0-10_2.11/2.2.0/spark-streaming-kafka-0-10_2.11-2.2.0.jar
> {code}
> After that I am able to import successfully part of necesary libraries:
> {code:java}
> import org.apache.spark.SparkConf
> import org.apache.spark.streaming._
> import org.apache.spark.streaming.kafka010._
> {code}
> However code fails when I try to create streaming context:
> {code:java}
> val ssc = new StreamingContext(sc, Seconds(2))
> Name: Compile Error
> Message: :38: error: overloaded method constructor StreamingContext 
> with alternatives:
>   (path: String,sparkContext: 
> org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)org.apache.spark.streaming.StreamingContext
>  
>   (path: String,hadoopConf: 
> org.apache.hadoop.conf.Configuration)org.apache.spark.streaming.StreamingContext
>  
>   (conf: org.apache.spark.SparkConf,batchDuration: 
> org.apache.spark.streaming.Duration)org.apache.spark.streaming.StreamingContext
>  
>   (sparkContext: 
> org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext,batchDuration:
>  
> org.apache.spark.streaming.Duration)org.apache.spark.streaming.StreamingContext
>  cannot be applied to 
> (org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext,
>  org.apache.spark.streaming.Duration)
>val ssc = new StreamingContext(sc, Seconds(2))
>  ^
> StackTrace: 
> {code}
> I have try it, in a jupyter docker 
> https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook
> and in spark cluster set up in Google cloud platform with the same results
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Installing Python Dependencies on Spark Cluster Hosts with Toree

[Luciano Resende]

On Mon, Dec 11, 2017 at 11:07 AM, Mana M  wrote:

> Hello,
>
> I am new to Spark + Jupyter and setting them up for our data analysis team.
> I had one question for which I cannot really find answer anywhere - hope
> someone can help here.
>
> I have setup multi-host Spark cluster and also have successfully installed
> Jupyter with Jupyter Hub. This setup will be shared among several data
> analysis team.
>
> The Spark cluster is setup with some common Python libraries. But each user
> may require additional libraries for their experimentation time to time. Is
> it possible for Jupyter user to install Python dependencies for her/his
> notebook, so dependencies are available on all Spark cluster nodes before
> user runs the notebook through Jupyter?
>
> I read about line magics (addDeps) in Apache toree, but I did not find any
> information on adding Python dependencies.
>
> Thanks,
> Mana
>

Toree does not provide any capabilities to manage the required dependencies
on remote execution nodes. Some approaches used in the industry community
are:

Anaconda or Anaconda Enterprise that enables you to build env that
are available/replicated in all nodes
Mapped user folders that can be used to host the necessary packages

I have also seen some discussions on Spark to better handle that, but I
don't believe this has been completely solved.


-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

[jira] [Resolved] (TOREE-459) Failure when adding a Jar.

[Luciano Resende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luciano Resende resolved TOREE-459.
---
   Resolution: Cannot Reproduce
 Assignee: Luciano Resende
Fix Version/s: 0.2.0

It seems that the result from search.maven.org is being redirected and does not 
work properly. I have tried using the jar url from the repository and it seems 
to work (at least no errors on adding the jar)

%AddJar 
http://repo1.maven.org/maven2/ml/dmlc/mxnet/mxnet-full_2.11-osx-x86_64-cpu/0.9.3a/mxnet-full_2.11-osx-x86_64-cpu-0.9.3a.jar

If you still see the issue with the jar url above, please reopen the jira.

Btw, my environment was
Spark version 2.2.0
Scala version 2.11.8

> Failure when adding a Jar.
> --
>
> Key: TOREE-459
> URL: https://issues.apache.org/jira/browse/TOREE-459
> Project: TOREE
>  Issue Type: Bug
>Reporter: Naveen Swamy
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> I am trying to use toree with Spark 2.2.7(latest) and Scala 2.11 however when 
> I add a jar it fails with the error below
> {{
> spark-2.2.7/jars/scala-library-2.11.8.jar(scala/reflect/package.class)' is 
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> error: error while loading package, class file 
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/package.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> error: error while loading Serializable, class file 
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Serializable.class)' is 
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> res2: 
> Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput]
>  = Right(())
> error: error while loading Predef, class file 
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Predef.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala 
> annotation)
> }}
> The jar I am trying to add is:
> %AddJar 
> https://search.maven.org/remotecontent?filepath=ml/dmlc/mxnet/mxnet-full_2.11-osx-x86_64-cpu/0.9.3a/mxnet-full_2.11-osx-x86_64-cpu-0.9.3a.jar
> Spark version 2.2.1
> Scala version 2.11.8



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (TOREE-460) Difference in output between Spark 2.1.0 and Spark 2.1.2+

[Luciano Resende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luciano Resende resolved TOREE-460.
---
   Resolution: Cannot Reproduce
 Assignee: Luciano Resende
Fix Version/s: 0.2.0

I have tried Toree master with both Spark 2.1.0 and 2.2.0 and both have the 
same behavior where:
```
val a="abc"
```
will produce
```
a = abc
Out[5]: abc
```

> Difference in output between Spark 2.1.0 and Spark 2.1.2+
> -
>
> Key: TOREE-460
> URL: https://issues.apache.org/jira/browse/TOREE-460
> Project: TOREE
>  Issue Type: Bug
>  Components: Kernel
>Affects Versions: 0.2.0
>Reporter: Vinayak Joshi
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> Using the current version of Toree (0.2.0) & Scala 2.11 when Scala code such 
> as this is executed:
> ```
> var v = "abc"
> ```
> against Spark 2.1.0, there is no output generated on the notebook. However, 
> against Spark 2.1.2 and above, the notebook displays the following lines as 
> output for the same code:
> ```
> v = abc
> Out[2]: abc
> ```
> When used via a kernel-gateway, I could trace the websocket messages and 
> observed that a couple of extra messages are received by the notebook when 
> the code is executed against Spark 2.1.2+ :
> ```
>   {
>   "parent_header": {
>...
>   },
>   "msg_type": "display_data",
>   "msg_id": "7ee0fe40-3283-45dc-8632-a9f013ae7052",
>   "content": {
>   "source": "user",
>   "data": {
>   "text/plain": "v = abc"
>   },
>   "metadata": {}
>   },
>   "header": {
>...
>   },
>   "channel": "iopub",
>   "buffers": [],
>   "metadata": {
>   "timestamp": "1513154854626"
>   }
>   }
> ```
> and
> ```
>   {
>   "parent_header": {
>...
>   },
>   "msg_type": "execute_result",
>   "msg_id": "98dae777-c5cf-4ae7-9903-306e1fdf7bc1",
>   "content": {
>   "execution_count": 2,
>   "data": {
>   "text/plain": "abc"
>   },
>   "metadata": {}
>   },
>   "header": {
>...
>   },
>   "channel": "iopub",
>   "buffers": [],
>   "metadata": {
>   "timestamp": "1513154854754"
>   }
>   }
> ```
> These messages appear to be getting rendered on the notebook ui. 
> This is breaking consistency/compatibility for expected outputs when Spark 
> version changes for notebooks. Also there are users that use some kind of 
> code cell hiding to prevent credentials etc being displayed in a notebook - 
> this use case now breaks since the cell output now displays any credential 
> values that are assigned to variables. 
> So looks like something's changed between Spark 2.1.0 and later versions that 
> causes Toree to produce different replies to the same code being executed? Is 
> it something Toree needs to adapt to?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)