[jira] [Commented] (TOREE-355) java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
[
https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857272#comment-15857272
]
Mingzhou Zhuang commented on TOREE-355:
---
I met this issue too.
Toree 0.2.0.dev1 CDH 5.10.0 with spark-2.0.0-beta2.
> java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> ---
>
> Key: TOREE-355
> URL: https://issues.apache.org/jira/browse/TOREE-355
> Project: TOREE
> Issue Type: Bug
> Environment: Toree 0.2.0.dev1
> CDH 5.9's embedded spark-2.0.0-beta2
>Reporter: Adrien Lavoillotte
> Labels: easyfix
>
> Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327
> for me. Instead, I got this error:
> {code}
> Exception in thread "main" java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
> at
> org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at scala.collection.immutable.Stream.map(Stream.scala:418)
> at
> org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
> at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
> at
> org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
> at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
> at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at scala.util.Try$.apply(Try.scala:192)
> at
> org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
> at
> org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
> at
> org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
> at
> org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
> at
> org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
> at
[jira] [Commented] (TOREE-363) Syntax Highlighting Breaks
[ https://issues.apache.org/jira/browse/TOREE-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857194#comment-15857194 ] Marius Van Niekerk commented on TOREE-363: -- So i believe we are passing the wrong mimetype to the frontend. In spylon-kernel (another alternative kernel for scala spark) I pass the following and it seems to work https://github.com/maxpoint/spylon-kernel/blob/master/spylon_kernel/scala_kernel.py#L21-L29 > Syntax Highlighting Breaks > -- > > Key: TOREE-363 > URL: https://issues.apache.org/jira/browse/TOREE-363 > Project: TOREE > Issue Type: Bug >Reporter: Aleksei Aleksinov >Priority: Minor > Attachments: Screen Shot 2017-01-26 at 18.40.54.png > > > Syntax highlighting breaks after a while of usage. But Scala code execution > and Spark jobs work well. > Affects Toree 0.2.0-dev1 > Jupyter 4.2.1 > Scala 2.12.1 or 2.11.8 > Spark 2.1.0 > Python 3.6 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-363) Syntax Highlighting Breaks
[ https://issues.apache.org/jira/browse/TOREE-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857170#comment-15857170 ] Felix Schüler commented on TOREE-363: - I can confirm this but I'm not sure if it's toree related. It usually happens for new cells (e.g. when evaluating the last cell with Shift + Enter). When refreshing the notebook web-page the rendering is correct again. > Syntax Highlighting Breaks > -- > > Key: TOREE-363 > URL: https://issues.apache.org/jira/browse/TOREE-363 > Project: TOREE > Issue Type: Bug >Reporter: Aleksei Aleksinov >Priority: Minor > Attachments: Screen Shot 2017-01-26 at 18.40.54.png > > > Syntax highlighting breaks after a while of usage. But Scala code execution > and Spark jobs work well. > Affects Toree 0.2.0-dev1 > Jupyter 4.2.1 > Scala 2.12.1 or 2.11.8 > Spark 2.1.0 > Python 3.6 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TOREE-354) Scala Error with Apache Spark when run in Jupyter
[
https://issues.apache.org/jira/browse/TOREE-354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Odersky closed TOREE-354.
---
Resolution: Not A Problem
There is a mismatch between the scala versions used by Toree and Spark.
The output of Toree shows it is using Scala 2.10, yet the spark Shell shows
that Spark was compiled with 2.11.
The easiest way to fix this is to upgrade toree to latest master. Alternatively
you can also [build spark with Scala
2.10|http://spark.apache.org/docs/latest/building-spark.html#building-for-scala-210]
but I would recommend against that since that version of scala has reached its
end of life.
> Scala Error with Apache Spark when run in Jupyter
> -
>
> Key: TOREE-354
> URL: https://issues.apache.org/jira/browse/TOREE-354
> Project: TOREE
> Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: Apache Spark 2.0.2
> Scala 2.11(Built with Apache Spark by default)
>Reporter: Ming Yu
> Labels: jupyter, scala, spark-shell
> Fix For: 0.1.0
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I'm having problems running Scala Spark on Jupyter. Below is my error message
> when I load Apache Toree - Scala notebook in jupyter.
> {noformat}
> root@ubuntu-2gb-sgp1-01:~# jupyter notebook --ip 0.0.0.0 --port
> [I 03:14:54.281 NotebookApp] Serving notebooks from local directory: /root
> [I 03:14:54.281 NotebookApp] 0 active kernels
> [I 03:14:54.281 NotebookApp] The Jupyter Notebook is running at:
> http://0.0.0.0:/
> [I 03:14:54.281 NotebookApp] Use Control-C to stop this server and shut down
> all kernels (twice to skip confirmation).
> [W 03:14:54.282 NotebookApp] No web browser found: could not locate runnable
> browser.
> [I 03:15:09.976 NotebookApp] 302 GET / (61.6.68.44) 1.21ms
> [I 03:15:15.924 NotebookApp] Creating new notebook in
> [W 03:15:16.592 NotebookApp] 404 GET
> /nbextensions/widgets/notebook/js/extension.js?v=20161120031454 (61.6.68.44)
> 15.49ms
> referer=http://188.166.235.21:/notebooks/Untitled2.ipynb?kernel_name=apache_toree_scala
> [I 03:15:16.677 NotebookApp] Kernel started:
> 94a63354-d294-4de7-a12c-2e05905e0c45
> Starting Spark Kernel with SPARK_HOME=/usr/local/spark
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Kernel version:
> 0.1.0.dev8-incubating-SNAPSHOT
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Scala version: Some(2.10.4)
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - ZeroMQ (JeroMQ) version: 3.2.2
> 16/11/20 03:15:18 [INFO] o.a.t.Main$$anon$1 - Initializing internal actor
> system
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;
> at akka.actor.ActorCell$.(ActorCell.scala:336)
> at akka.actor.ActorCell$.(ActorCell.scala)
> at akka.actor.RootActorPath.$div(ActorPath.scala:185)
> at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:465)
> at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:453)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
> at scala.util.Try$.apply(Try.scala:192)
> at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
> at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
> at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
> at scala.util.Success.flatMap(Try.scala:231)
> at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
> at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585)
> at akka.actor.ActorSystemImpl.(ActorSystem.scala:578)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)
> at
> org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71)
> at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35)
> at
> org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60)
> at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35)
> at
> org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:72)
> at
[jira] [Commented] (TOREE-363) Syntax Highlighting Breaks
[ https://issues.apache.org/jira/browse/TOREE-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857140#comment-15857140 ] Jakob Odersky commented on TOREE-363: - AFAIK syntax highlighting is not handled by the kernels themselves, they only pass a hint to the notebooks, telling them what language it uses. How often does this happen? I am unable to reproduce it. > Syntax Highlighting Breaks > -- > > Key: TOREE-363 > URL: https://issues.apache.org/jira/browse/TOREE-363 > Project: TOREE > Issue Type: Bug >Reporter: Aleksei Aleksinov >Priority: Minor > Attachments: Screen Shot 2017-01-26 at 18.40.54.png > > > Syntax highlighting breaks after a while of usage. But Scala code execution > and Spark jobs work well. > Affects Toree 0.2.0-dev1 > Jupyter 4.2.1 > Scala 2.12.1 or 2.11.8 > Spark 2.1.0 > Python 3.6 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (TOREE-281) Fix some typos in comments/docs/testnames.
[
https://issues.apache.org/jira/browse/TOREE-281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Odersky closed TOREE-281.
---
Resolution: Fixed
> Fix some typos in comments/docs/testnames.
> --
>
> Key: TOREE-281
> URL: https://issues.apache.org/jira/browse/TOREE-281
> Project: TOREE
> Issue Type: Bug
>Reporter: Dongjoon Hyun
>Priority: Trivial
>
> There exists some minor typos like the followings in iPython Notebooks,
> comments, and testcase names.
> {code}
> // TODO Handle the case where there is no [-delimeter-]{+delimiter+}
> "This example is a Scala [-adapatation-]{+adaptation+} of
> [this](https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/stream_demo/meetup-streaming.ipynb)
> notebook from
> [jupyter_dashboards](https://github.com/jupyter-incubator/dashboards).\n",
> "In Toree, declarativewidgets need to be initialized by adding the JAR
> with the scala [-implamentation-]{+implementation+} and calling
> `initWidgets`. This is must take place very close to the top of the notebook."
> "The rest of the call are [-methos-]{+methods+} for starting and stopping
> the streaming application as well as functions that define the streaming
> flow."
> # This file is populated when doing a make release. It should be empty by
> [-defaut.-]{+default.+}
> it("should truncate or not [-turncate-]{+truncate+} based on %truncate") {
> * Contains helpers and [-contants-]{+constants+} associated with the
> dependency manager.
> // [-Overriden-]{+Overridden+} to link before sending open message
> // Single property fields are not well supported by play, this is a little
> funky workaround [-founde-]{+found+} here:
> #' Returns the [-dimentions-]{+dimensions+} (number of rows and columns) of a
> DataFrame
> # @return a new RDD created by performing the simple union
> [-(witout-]{+(without+} removing
> # Allow the user to have a more flexible [-definiton-]{+definition+} of the
> text file path
> # Allow the user to have a more flexible [-definiton-]{+definition+} of the
> text file path
> # Allow the user to have a more flexible [-definiton-]{+definition+} of the
> text file path
> # Allow the user to have a more flexible [-definiton-]{+definition+} of the
> text file path
> # that [-termintates-]{+terminates+} when the next row is empty.
> # \item mergeCombiners, to combine two C's into a single one (e.g.,
> [-concatentates-]{+concatenates+}
> # "y" should be in the [-environemnt-]{+environment+} of g.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (TOREE-373) scala 2.11 library incompatible with pyspark 2.1.0
[ https://issues.apache.org/jira/browse/TOREE-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Odersky closed TOREE-373. --- Resolution: Not A Bug +1 to Marius' answer. I would be surprised it worked with 2.12 though, considering it is binary incompatible > scala 2.11 library incompatible with pyspark 2.1.0 > -- > > Key: TOREE-373 > URL: https://issues.apache.org/jira/browse/TOREE-373 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 > Environment: 64 bit Ubuntu 16.04 LTS >Reporter: Peter > Labels: pyspark, scala > Fix For: 0.1.0 > > > When doengraded to 2.10 or upgraded to 2.12, then I think it will work fine, > but some testing is required. > Refer to the following stackexchange note: > http://stackoverflow.com/questions/29339005/run-main-0-java-lang-nosuchmethoderror -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected
[ https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856905#comment-15856905 ] Marius Van Niekerk commented on TOREE-374: -- As for actual fixes for the cell repl classes, I dont really think that is possible. Its certainly out of scope for Apache Toree. Basically i tend to run Toree on some beefier edge nodes, so consequently i allocate at least 8g of memory per driver process. > Variables declared on the Notebook are not garbage collected > > > Key: TOREE-374 > URL: https://issues.apache.org/jira/browse/TOREE-374 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: David Taieb > > I'm not sure if it's a bug or a limitation of the underlying scala REPL. > As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) > auto-visualization feature within Scala gateway, I have implemented a weak > hashmap that tracks objects declared on the Scala REPL. However, I have found > that objects are not correctly gc'ed when the object is declared in a cell > with a val or var keyword and then the cell is ran again. One would expect > that the original object has no more references and should be gc'ed but it's > not. > However, when the object is declare with var keyword and then set to null in > another cell, then it is correctly gc'ed. > I'm concerned that users who run the same cell multiple times would > unwittingly have memory leaks which can eventually lead to OOM errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected
[ https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856898#comment-15856898 ] Marius Van Niekerk commented on TOREE-374: -- So zeppelin has a partial workaround for the valueOfTerm style thing in this https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1134 > Variables declared on the Notebook are not garbage collected > > > Key: TOREE-374 > URL: https://issues.apache.org/jira/browse/TOREE-374 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: David Taieb > > I'm not sure if it's a bug or a limitation of the underlying scala REPL. > As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) > auto-visualization feature within Scala gateway, I have implemented a weak > hashmap that tracks objects declared on the Scala REPL. However, I have found > that objects are not correctly gc'ed when the object is declared in a cell > with a val or var keyword and then the cell is ran again. One would expect > that the original object has no more references and should be gc'ed but it's > not. > However, when the object is declare with var keyword and then set to null in > another cell, then it is correctly gc'ed. > I'm concerned that users who run the same cell multiple times would > unwittingly have memory leaks which can eventually lead to OOM errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TOREE-374) Variables declared on the Notebook are not garbage collected
[ https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856880#comment-15856880 ] Jakob Odersky edited comment on TOREE-374 at 2/7/17 9:57 PM: - Thanks for the reproduction steps, [~dtaieb]. The reason why results aren't garbage collected is due to the way the Scala REPL represents evaluated expressions. This is true for both in the standard object-wrapped repl as well as the class based one used by Toree. Unfortunately I can't think of an easy fix that we could quickly get into upstream. How severe is this bug in a typical use-case? Are cells re-evaluated enough without a kernel restart, for it to become a problem? [~mariusvniekerk] do you know if/how zeppelin has worked around this? was (Author: jodersky): Thanks for the reproduction steps, [~dtaieb]. The reason why results aren't garbage collected is due to the way the Scala REPL represents evaluated expressions. This is true for both in the standard object-wrapped repl as well as the class based one used by Toree. Unfortunately I can't think of an easy fix that we could quickly get into upstream. How severe is this bug in a typical use-case? Are cells reevaluated enough without a kernel restart for it to become a problem? [~mariusvniekerk] do you know if/how zeppelin has worked around this? > Variables declared on the Notebook are not garbage collected > > > Key: TOREE-374 > URL: https://issues.apache.org/jira/browse/TOREE-374 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: David Taieb > > I'm not sure if it's a bug or a limitation of the underlying scala REPL. > As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) > auto-visualization feature within Scala gateway, I have implemented a weak > hashmap that tracks objects declared on the Scala REPL. However, I have found > that objects are not correctly gc'ed when the object is declared in a cell > with a val or var keyword and then the cell is ran again. One would expect > that the original object has no more references and should be gc'ed but it's > not. > However, when the object is declare with var keyword and then set to null in > another cell, then it is correctly gc'ed. > I'm concerned that users who run the same cell multiple times would > unwittingly have memory leaks which can eventually lead to OOM errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-375) Incorrect fully qualified name for spark context
[
https://issues.apache.org/jira/browse/TOREE-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856688#comment-15856688
]
Marius Van Niekerk commented on TOREE-375:
--
Yeah, zeppelin works around this with a pretty crazy hack, but having
valueOfTerm work in class repl would be great.
> Incorrect fully qualified name for spark context
>
>
> Key: TOREE-375
> URL: https://issues.apache.org/jira/browse/TOREE-375
> Project: TOREE
> Issue Type: Bug
> Environment: Jupyter Notebook with Toree latest master
> (1a9c11f5f1381c15b691a716acd0e1f0432a9a35) and Spark 2.0.2, Scala 2.11
>Reporter: Felix Schüler
>
> When running below snippet in a cell I get a compile error for the MLContext
> Constructor. Somehow the fully qualified name of the SparkContext gets messed
> up.
> The same does not happen when I start a Spark shell with the --jars command
> and create the MLContext there.
> Snippet (the systemml jar is build with the latest master of SystemML):
> {code}
> %addjar
> file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar
> -f
> import org.apache.sysml.api.mlcontext._
> import org.apache.sysml.api.mlcontext.ScriptFactory._
> val ml = new MLContext(sc)
> Starting download from
> file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar
> Finished download of systemml-0.13.0-incubating-SNAPSHOT.jar
> Name: Compile Error
> Message: :25: error: overloaded method constructor MLContext with
> alternatives:
> (x$1:
> org.apache.spark.api.java.JavaSparkContext)org.apache.sysml.api.mlcontext.MLContext
>
> (x$1:
> org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)org.apache.sysml.api.mlcontext.MLContext
> cannot be applied to
> (org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)
>val ml = new MLContext(sc)
> ^
> StackTrace:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-375) Incorrect fully qualified name for spark context
[
https://issues.apache.org/jira/browse/TOREE-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856668#comment-15856668
]
Jakob Odersky commented on TOREE-375:
-
I checked the implementation of `valueOfTerm` in the Scala source and it does
in fact seem that they assume a non-class-based representation of results when
looking up symbols. I'll escalate this issue.
> Incorrect fully qualified name for spark context
>
>
> Key: TOREE-375
> URL: https://issues.apache.org/jira/browse/TOREE-375
> Project: TOREE
> Issue Type: Bug
> Environment: Jupyter Notebook with Toree latest master
> (1a9c11f5f1381c15b691a716acd0e1f0432a9a35) and Spark 2.0.2, Scala 2.11
>Reporter: Felix Schüler
>
> When running below snippet in a cell I get a compile error for the MLContext
> Constructor. Somehow the fully qualified name of the SparkContext gets messed
> up.
> The same does not happen when I start a Spark shell with the --jars command
> and create the MLContext there.
> Snippet (the systemml jar is build with the latest master of SystemML):
> {code}
> %addjar
> file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar
> -f
> import org.apache.sysml.api.mlcontext._
> import org.apache.sysml.api.mlcontext.ScriptFactory._
> val ml = new MLContext(sc)
> Starting download from
> file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar
> Finished download of systemml-0.13.0-incubating-SNAPSHOT.jar
> Name: Compile Error
> Message: :25: error: overloaded method constructor MLContext with
> alternatives:
> (x$1:
> org.apache.spark.api.java.JavaSparkContext)org.apache.sysml.api.mlcontext.MLContext
>
> (x$1:
> org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)org.apache.sysml.api.mlcontext.MLContext
> cannot be applied to
> (org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)
>val ml = new MLContext(sc)
> ^
> StackTrace:
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected
[
https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856266#comment-15856266
]
David Taieb commented on TOREE-374:
---
[~lbustelo] Totally understand the technical limitation, but looking from the
perspective of the user, it looks like a bug. At the very least we should
document best practices workaround
Also, looking at the results from // show, I see this
import $line22$read.$iw.$iw.$iw.$iw.$iw.$iw.x;
class $iw extends Serializable
def () = {
super.;
()
};
val res4 = println(x)
};
Wonder how $line22$read.$iw.$iw.$iw.$iw.$iw.$iw.x; is created and whether we
have an opportunity to clean it up within a pre_run_cell event?
> Variables declared on the Notebook are not garbage collected
>
>
> Key: TOREE-374
> URL: https://issues.apache.org/jira/browse/TOREE-374
> Project: TOREE
> Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: David Taieb
>
> I'm not sure if it's a bug or a limitation of the underlying scala REPL.
> As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust)
> auto-visualization feature within Scala gateway, I have implemented a weak
> hashmap that tracks objects declared on the Scala REPL. However, I have found
> that objects are not correctly gc'ed when the object is declared in a cell
> with a val or var keyword and then the cell is ran again. One would expect
> that the original object has no more references and should be gc'ed but it's
> not.
> However, when the object is declare with var keyword and then set to null in
> another cell, then it is correctly gc'ed.
> I'm concerned that users who run the same cell multiple times would
> unwittingly have memory leaks which can eventually lead to OOM errors.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected
[
https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856197#comment-15856197
]
Chip Senkbeil commented on TOREE-374:
-
Here's what I used to do. It still works with our 0.1.x branch. Assuming it'll
work on master using Scala 2.11's REPL implementation.
{code}
val x = 3
println(x) // show
{code}
Just tack on a {code}// show{code} at the end of your code, the space between
the forward slash and show being required.
> Variables declared on the Notebook are not garbage collected
>
>
> Key: TOREE-374
> URL: https://issues.apache.org/jira/browse/TOREE-374
> Project: TOREE
> Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: David Taieb
>
> I'm not sure if it's a bug or a limitation of the underlying scala REPL.
> As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust)
> auto-visualization feature within Scala gateway, I have implemented a weak
> hashmap that tracks objects declared on the Scala REPL. However, I have found
> that objects are not correctly gc'ed when the object is declared in a cell
> with a val or var keyword and then the cell is ran again. One would expect
> that the original object has no more references and should be gc'ed but it's
> not.
> However, when the object is declare with var keyword and then set to null in
> another cell, then it is correctly gc'ed.
> I'm concerned that users who run the same cell multiple times would
> unwittingly have memory leaks which can eventually lead to OOM errors.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected
[ https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856179#comment-15856179 ] Gino Bustelo commented on TOREE-374: [~chipsenkbeil] can you provide here the trick to visualize the repl's representation of the code? > Variables declared on the Notebook are not garbage collected > > > Key: TOREE-374 > URL: https://issues.apache.org/jira/browse/TOREE-374 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: David Taieb > > I'm not sure if it's a bug or a limitation of the underlying scala REPL. > As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) > auto-visualization feature within Scala gateway, I have implemented a weak > hashmap that tracks objects declared on the Scala REPL. However, I have found > that objects are not correctly gc'ed when the object is declared in a cell > with a val or var keyword and then the cell is ran again. One would expect > that the original object has no more references and should be gc'ed but it's > not. > However, when the object is declare with var keyword and then set to null in > another cell, then it is correctly gc'ed. > I'm concerned that users who run the same cell multiple times would > unwittingly have memory leaks which can eventually lead to OOM errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected
[ https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856177#comment-15856177 ] Gino Bustelo commented on TOREE-374: I'm not a JVM expert, but [~dtaieb], you need to consider that you are not in a normal execution environment. Those steps are not taking place in a single class... each time you execute a cell, there is a class that is generated that imports the previous classes from a previous cell. You basically get a hierarchy of nested classes. There might be side effects to that. [~jodersky] brings a good point. Try that set of calls in the scala repl and lets compare. > Variables declared on the Notebook are not garbage collected > > > Key: TOREE-374 > URL: https://issues.apache.org/jira/browse/TOREE-374 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: David Taieb > > I'm not sure if it's a bug or a limitation of the underlying scala REPL. > As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) > auto-visualization feature within Scala gateway, I have implemented a weak > hashmap that tracks objects declared on the Scala REPL. However, I have found > that objects are not correctly gc'ed when the object is declared in a cell > with a val or var keyword and then the cell is ran again. One would expect > that the original object has no more references and should be gc'ed but it's > not. > However, when the object is declare with var keyword and then set to null in > another cell, then it is correctly gc'ed. > I'm concerned that users who run the same cell multiple times would > unwittingly have memory leaks which can eventually lead to OOM errors. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
