[RESULT][VOTE] Apache Toree 0.4.0 RC1
On Thu, Jul 16, 2020 at 12:25 PM Luciano Resende wrote: > > Please vote to approve the release of Apache Toree 0.4.0-incubating (RC1). > > Tag: v0.4.0-incubating-rc1 (b2374e28ce66ae28079703d725b7c0db4d4a) > > https://github.com/apache/incubator-toree/tree/v0.4.0-incubating-rc1 > > All distribution packages, including signatures, digests, etc. can be found > at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1016 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) > > 2. Install Apache Toree via `pip install > https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.4.0-incubating > [ ] -1 Do not release this package because ... > > Vote passed with 4 +1 votes from: Chip Senkbeil Luciano Resende Kevin Bates Marius van Niekerk -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: [VOTE] Apache Toree 0.4.0 RC1
On Fri, Jul 17, 2020 at 8:32 AM Chip Senkbeil wrote: > > +1, LGTM. Did we want to update the release notes with the changes? I > was using > https://github.com/apache/incubator-toree/compare/v0.3.0-incubating...v0.4.0-incubating-rc1 > to get a summary of what has changed since 0.3.0's release. > Good catch on the release notes https://github.com/apache/incubator-toree/pull/188 > On Thu, Jul 16, 2020 at 2:26 PM Luciano Resende wrote: > > > > Please vote to approve the release of Apache Toree 0.4.0-incubating (RC1). > > > > Tag: v0.4.0-incubating-rc1 (b2374e28ce66ae28079703d725b7c0db4d4a) > > > > https://github.com/apache/incubator-toree/tree/v0.4.0-incubating-rc1 > > > > All distribution packages, including signatures, digests, etc. can be found > > at: > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/ > > > > Staging artifacts can be found at: > > > > https://repository.apache.org/content/repositories/orgapachetoree-1016 > > > > ## Testing Instructions > > > > The fastest way to get up and running is to using Jupyter. > > > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) > > > > 2. Install Apache Toree via `pip install > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz` > > followed by `jupyter toree install` > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > `jupyter toree install --spark_home=/usr/local/spark` > > > > - You may need to run with `sudo` for installation permission > > > > - For all installation options, run `jupyter toree install --help-all` > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > you can install via `pip install notebook` > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > > new > > dev > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > > notebook > > > > ## Voting Instructions > > > > The vote is open for at least 72 hours and passes if a majority of at least > > 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Toree 0.4.0-incubating > > [ ] -1 Do not release this package because ... > > > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: [VOTE] Apache Toree 0.4.0 RC1
Off course, my +1 On Thu, Jul 16, 2020 at 12:25 PM Luciano Resende wrote: > > Please vote to approve the release of Apache Toree 0.4.0-incubating (RC1). > > Tag: v0.4.0-incubating-rc1 (b2374e28ce66ae28079703d725b7c0db4d4a) > > https://github.com/apache/incubator-toree/tree/v0.4.0-incubating-rc1 > > All distribution packages, including signatures, digests, etc. can be found > at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1016 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) > > 2. Install Apache Toree via `pip install > https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.4.0-incubating > [ ] -1 Do not release this package because ... > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-508) Reply errors to iopub.error instead of iopub.exec_result
[ https://issues.apache.org/jira/browse/TOREE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-508. --- Fix Version/s: 0.4.0 Assignee: Fabiano V. Santos Resolution: Fixed > Reply errors to iopub.error instead of iopub.exec_result > > > Key: TOREE-508 > URL: https://issues.apache.org/jira/browse/TOREE-508 > Project: TOREE > Issue Type: Bug >Reporter: Fabiano V. Santos >Assignee: Fabiano V. Santos >Priority: Major > Fix For: 0.4.0 > > > Errors of any kind (compilation, exceptions, etc) returns as normal results > instead of using iopub.error, so jupyter interpret the cell execution as > success which lead to problems if you use tools like papermill to schedule > notebooks, as you never know if the notebook execution was successfully or > not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TOREE-518) File Path for toree-0.2.0 devl.tar.gz is broken
[
https://issues.apache.org/jira/browse/TOREE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende closed TOREE-518.
-
Fix Version/s: Not Applicable
Assignee: Luciano Resende
Resolution: Invalid
Releases are now available directly from pypi.
> File Path for toree-0.2.0 devl.tar.gz is broken
> ---
>
> Key: TOREE-518
> URL: https://issues.apache.org/jira/browse/TOREE-518
> Project: TOREE
> Issue Type: Bug
>Reporter: Ajana Chandiruthil Sathian
> Assignee: Luciano Resende
>Priority: Major
> Fix For: Not Applicable
>
> Attachments: apapche_toree_exception.png
>
>
> The file path in the README.md documentation is broken.
> Executing this command as per the documentation is giving an 404 excepiton.
>
> {{pip install
> [https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz]}}
> {{}}
> {{So please update the document or put the gz file on that location.}}
> {{}}
> {{Any kind of help is greatly appreciated!}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (TOREE-516) Kerberos error while working with Toree Kernel
[
https://issues.apache.org/jira/browse/TOREE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende closed TOREE-516.
-
Fix Version/s: Not Applicable
Assignee: Luciano Resende
Resolution: Workaround
Suggested using Jupyter Enterprise Gateway to provide the end-to-end requested
flow.
> Kerberos error while working with Toree Kernel
> --
>
> Key: TOREE-516
> URL: https://issues.apache.org/jira/browse/TOREE-516
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: RHEL 7.7, HDP 3 , Spark 2.3.2
>Reporter: Bharath
> Assignee: Luciano Resende
>Priority: Critical
> Labels: security
> Fix For: Not Applicable
>
>
> We have a Kerberied HDP 3.1.0 cluster, configured Jupyter/Toree - It throws
> Keberos error, we have a valid kerberos ticket in place. Here is the
> kernel.json
> {
> "argv": [
> "/opt/jupyterhub/share/jupyter/kernels/apache_toree_scala/bin/run.sh",
> "--profile",
> "\{connection_file}"
> ],
> "env": {
> "DEFAULT_INTERPRETER": "Scala",
> "TOREE_SPARK_OPTS": "--master yarn --deploy-mode cluster",
> "TOREE_OPTS": "",
> "HADOOP_CONF_DIR": "/etc/hadoop/conf",
> "SPARK_HOME": "/usr/hdp/3.1.0.0-78/spark2",
> "PYTHONPATH":
> "/opt/jupyterhub/bin/python:/opt/anaconda3/bin/python:/usr/hdp/3.1.0.0-78/spark2/python/lib/py4j-0.10.7-src.zip:/usr/hdp/3.1.0.0-78/spark2/python:/usr/hdp/3.1.0.0-78/spark2/python/lib",
> "PYTHON_EXEC": "python",
> "HBASE_CONF_DIR": "/etc/hbase/conf",
> "HBASE_HOME": "/usr/hdp/current/hbase-client",
> "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64"
> },
> "display_name": "Toree - Scala-kernel",
> "language": "scala",
> "interrupt_mode": "signal",
> "metadata": {}
> }
> Error Message:
> 20/03/26 18:32:28 INFO RpcRetryingCallerImpl: Call exception, tries=8,
> retries=36, started=18874 ms ago, cancelled=false, msg=Call to
> compute-4.datalake.ntt/xxx.xxx.xx.xxx:16020 failed on local exception:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)], details=row 'name_data:exchg_yearly,,99' on
> table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=compute-4.datalake.ntt
> With above configuration, kernel dies sometimes.
> Please suggest if the way we are passing configs in Kernel.Json is the right
> way ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (TOREE-499) K8: Not be able to submit Job in k8
[ https://issues.apache.org/jira/browse/TOREE-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende closed TOREE-499. - Fix Version/s: Not Applicable Assignee: Kevin Bates Resolution: Workaround Workaround provided. Please reopen if this is still an issue. > K8: Not be able to submit Job in k8 > --- > > Key: TOREE-499 > URL: https://issues.apache.org/jira/browse/TOREE-499 > Project: TOREE > Issue Type: Bug >Reporter: Leonardo Barrientos C. >Assignee: Kevin Bates >Priority: Blocker > Fix For: Not Applicable > > > Hello, i can't submit jobs to k8, the assembly jar resides in the local > folder directory, so the Driver and executors in K8 cannot get the .jar; it > is possible? Please help. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TOREE-426) Spark 2.2.0 and scala 2.12.1
[ https://issues.apache.org/jira/browse/TOREE-426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende closed TOREE-426. - Fix Version/s: 0.4.0 Resolution: Not A Problem Works after building with the proper scala version > Spark 2.2.0 and scala 2.12.1 > > > Key: TOREE-426 > URL: https://issues.apache.org/jira/browse/TOREE-426 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: MAC OS 10.12.5 >Reporter: Manish Ranjan >Priority: Major > Fix For: 0.4.0 > > > on running command `jupyter notebook` > greeted with > Starting Spark Kernel with > SPARK_HOME=/Users/manish.ranjan/Spark/Installer/spark-2.2/ > log4j:WARN No appenders could be found for logger > (org.apache.toree.Main$$anon$1). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet; > at akka.actor.ActorCell$.(ActorCell.scala:336) > at akka.actor.ActorCell$.(ActorCell.scala) > at akka.actor.RootActorPath.$div(ActorPath.scala:185) > at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:465) > at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:453) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78) > at scala.util.Try$.apply(Try.scala:192) > at > akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73) > at > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) > at > akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84) > at scala.util.Success.flatMap(Try.scala:231) > at > akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84) > at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585) > at akka.actor.ActorSystemImpl.(ActorSystem.scala:578) > at akka.actor.ActorSystem$.apply(ActorSystem.scala:142) > at akka.actor.ActorSystem$.apply(ActorSystem.scala:109) > at > org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71) > at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:34) > at > org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60) > at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:34) > at > org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:70) > at org.apache.toree.Main$delayedInit$body.apply(Main.scala:39) > at scala.Function0$class.apply$mcV$sp(Function0.scala:34) > at > scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) > at scala.App$$anonfun$main$1.apply(App.scala:76) > at scala.App$$anonfun$main$1.apply(App.scala:76) > at scala.collection.immutable.List.foreach(List.scala:381) > at > scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) > at scala.App$class.main(App.scala:76) > at org.apache.toree.Main$.main(Main.scala:23) > at org.apache.toree.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > [W 23:39:20.049 NotebookApp] Timeout waiting for kernel_info reply from > 282ebec0-b01a-405e-b002-f4569179a43d -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TOREE-519) Toree 0.5.0 Release (Spark 3.0 support)
[ https://issues.apache.org/jira/browse/TOREE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-519: -- Component/s: Build Fix Version/s: 0.5.0 > Toree 0.5.0 Release (Spark 3.0 support) > --- > > Key: TOREE-519 > URL: https://issues.apache.org/jira/browse/TOREE-519 > Project: TOREE > Issue Type: Bug > Components: Build > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.5.0 > > > We need to merge the Scala 2.12 change and publish a Toree release that > supports Spark 3.0 right after toree release 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TOREE-519) Toree 0.5.0 Release (Spark 3.0 support)
[ https://issues.apache.org/jira/browse/TOREE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-519: - Assignee: Luciano Resende > Toree 0.5.0 Release (Spark 3.0 support) > --- > > Key: TOREE-519 > URL: https://issues.apache.org/jira/browse/TOREE-519 > Project: TOREE > Issue Type: Bug > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > > We need to merge the Scala 2.12 change and publish a Toree release that > supports Spark 3.0 right after toree release 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-519) Toree 0.5.0 Release (Spark 3.0 support)
Luciano Resende created TOREE-519: - Summary: Toree 0.5.0 Release (Spark 3.0 support) Key: TOREE-519 URL: https://issues.apache.org/jira/browse/TOREE-519 Project: TOREE Issue Type: Bug Reporter: Luciano Resende We need to merge the Scala 2.12 change and publish a Toree release that supports Spark 3.0 right after toree release 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159418#comment-17159418 ] Luciano Resende commented on TOREE-510: --- Sorry it took a long time, the release vote is up on the mailing list https://lists.apache.org/thread.html/re7030486dd143fbbcdbb53166b222fb04f0ae536be64b2a751722697%40%3Cdev.toree.apache.org%3E > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés > Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[VOTE] Apache Toree 0.4.0 RC1
Please vote to approve the release of Apache Toree 0.4.0-incubating (RC1). Tag: v0.4.0-incubating-rc1 (b2374e28ce66ae28079703d725b7c0db4d4a) https://github.com/apache/incubator-toree/tree/v0.4.0-incubating-rc1 All distribution packages, including signatures, digests, etc. can be found at: https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/ Staging artifacts can be found at: https://repository.apache.org/content/repositories/orgapachetoree-1016 ## Testing Instructions The fastest way to get up and running is to using Jupyter. 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) 2. Install Apache Toree via `pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz` followed by `jupyter toree install` - You need to set a valid Apache Spark 2.x home, which can be done via `jupyter toree install --spark_home=/usr/local/spark` - You may need to run with `sudo` for installation permission - For all installation options, run `jupyter toree install --help-all` 4. Run a Jupyter notebook server via `jupyter notebook` - If the notebook portion of Jupyter is not installed but Jupyter is, you can install via `pip install notebook` 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new dev 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the notebook ## Voting Instructions The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Toree 0.4.0-incubating [ ] -1 Do not release this package because ... -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Re: Podling Toree Report Reminder - May 2020
Thanks Justin, I have updated the report and explained why only 1 thing is needed for graduation. On Mon, May 11, 2020 at 3:08 AM Justin Mclean wrote: > > Hi, > > Thanks for submitting our report but it could do with a little more detail. > What for instance is "guitter"? We'd also like to see 3 things that are > needed before you graduate. What steps is teh project taking to grow the > committer base? The report in this state is likely to be rejected and the > project asked to report next again next month. > > Thanks, > Justin > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Commented] (TOREE-516) Kerberos error while working with Toree Kernel
[
https://issues.apache.org/jira/browse/TOREE-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071306#comment-17071306
]
Luciano Resende commented on TOREE-516:
---
While Toree kernel can be submitted in yarn cluster mode, vanilla Jupyter
Notebooks expect kernels as local processes and cannot discover where in the
YARN cluster the actual kernel launched. The proper way to handle this is to
have Jupyter Notebook -> Jupyter Enterprise Gateway -> Toree Kernel, and this
is described in
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/kernel-yarn-cluster-mode.html
> Kerberos error while working with Toree Kernel
> --
>
> Key: TOREE-516
> URL: https://issues.apache.org/jira/browse/TOREE-516
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: RHEL 7.7, HDP 3 , Spark 2.3.2
>Reporter: Bharath
>Priority: Critical
> Labels: security
>
> We have a Kerberied HDP 3.1.0 cluster, configured Jupyter/Toree - It throws
> Keberos error, we have a valid kerberos ticket in place. Here is the
> kernel.json
> {
> "argv": [
> "/opt/jupyterhub/share/jupyter/kernels/apache_toree_scala/bin/run.sh",
> "--profile",
> "\{connection_file}"
> ],
> "env": {
> "DEFAULT_INTERPRETER": "Scala",
> "TOREE_SPARK_OPTS": "--master yarn --deploy-mode cluster",
> "TOREE_OPTS": "",
> "HADOOP_CONF_DIR": "/etc/hadoop/conf",
> "SPARK_HOME": "/usr/hdp/3.1.0.0-78/spark2",
> "PYTHONPATH":
> "/opt/jupyterhub/bin/python:/opt/anaconda3/bin/python:/usr/hdp/3.1.0.0-78/spark2/python/lib/py4j-0.10.7-src.zip:/usr/hdp/3.1.0.0-78/spark2/python:/usr/hdp/3.1.0.0-78/spark2/python/lib",
> "PYTHON_EXEC": "python",
> "HBASE_CONF_DIR": "/etc/hbase/conf",
> "HBASE_HOME": "/usr/hdp/current/hbase-client",
> "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64"
> },
> "display_name": "Toree - Scala-kernel",
> "language": "scala",
> "interrupt_mode": "signal",
> "metadata": {}
> }
> Error Message:
> 20/03/26 18:32:28 INFO RpcRetryingCallerImpl: Call exception, tries=8,
> retries=36, started=18874 ms ago, cancelled=false, msg=Call to
> compute-4.datalake.ntt/xxx.xxx.xx.xxx:16020 failed on local exception:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)], details=row 'name_data:exchg_yearly,,99' on
> table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=compute-4.datalake.ntt
> With above configuration, kernel dies sometimes.
> Please suggest if the way we are passing configs in Kernel.Json is the right
> way ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010134#comment-17010134 ] Luciano Resende commented on TOREE-510: --- [~andresdoncel-stratio] done a few dependency updates as part of https://github.com/apache/incubator-toree/pull/183 > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés > Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-514) Update dependencies in preparation for 0.4.0 release
[ https://issues.apache.org/jira/browse/TOREE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-514. --- Fix Version/s: 0.4.0 Assignee: Luciano Resende Resolution: Fixed > Update dependencies in preparation for 0.4.0 release > > > Key: TOREE-514 > URL: https://issues.apache.org/jira/browse/TOREE-514 > Project: TOREE > Issue Type: Task > Components: Build >Affects Versions: 0.4.0 > Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-513) Add logo resource for Toree Kernel
[ https://issues.apache.org/jira/browse/TOREE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-513. --- Fix Version/s: 0.4.0 Assignee: Luciano Resende Resolution: Fixed > Add logo resource for Toree Kernel > -- > > Key: TOREE-513 > URL: https://issues.apache.org/jira/browse/TOREE-513 > Project: TOREE > Issue Type: Task >Affects Versions: 0.4.0 > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > Attachments: jupyterlab-launcher-with-kernel-logo.png > > > JupyterLab exposes logos for kernels in the new launcher widget, this adds a > logo for Toree installed kernels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-514) Update dependencies in preparation for 0.4.0 release
Luciano Resende created TOREE-514: - Summary: Update dependencies in preparation for 0.4.0 release Key: TOREE-514 URL: https://issues.apache.org/jira/browse/TOREE-514 Project: TOREE Issue Type: Task Components: Build Affects Versions: 0.4.0 Reporter: Luciano Resende -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TOREE-513) Add logo resource for Toree Kernel
[ https://issues.apache.org/jira/browse/TOREE-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-513: -- Attachment: jupyterlab-launcher-with-kernel-logo.png > Add logo resource for Toree Kernel > -- > > Key: TOREE-513 > URL: https://issues.apache.org/jira/browse/TOREE-513 > Project: TOREE > Issue Type: Task >Affects Versions: 0.4.0 > Reporter: Luciano Resende >Priority: Major > Attachments: jupyterlab-launcher-with-kernel-logo.png > > > JupyterLab exposes logos for kernels in the new launcher widget, this adds a > logo for Toree installed kernels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-513) Add logo resource for Toree Kernel
Luciano Resende created TOREE-513: - Summary: Add logo resource for Toree Kernel Key: TOREE-513 URL: https://issues.apache.org/jira/browse/TOREE-513 Project: TOREE Issue Type: Task Affects Versions: 0.4.0 Reporter: Luciano Resende JupyterLab exposes logos for kernels in the new launcher widget, this adds a logo for Toree installed kernels. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TOREE-511) Rework "make dev/make jupyter" build tasks
[ https://issues.apache.org/jira/browse/TOREE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-511. --- Resolution: Fixed > Rework "make dev/make jupyter" build tasks > -- > > Key: TOREE-511 > URL: https://issues.apache.org/jira/browse/TOREE-511 > Project: TOREE > Issue Type: Task > Components: Build >Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > > Toree has `make dev` and `make jupyter` commands that are using some old > dependencies and some weird way to build them. We should create a docker > file that extends from `jupyter/all-spark-notebook` to make maintenance > easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-512) Refresh support for running Toree in Jupyter Binder
Luciano Resende created TOREE-512: - Summary: Refresh support for running Toree in Jupyter Binder Key: TOREE-512 URL: https://issues.apache.org/jira/browse/TOREE-512 Project: TOREE Issue Type: Task Components: Build Reporter: Luciano Resende Assignee: Luciano Resende Fix For: 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-511) Rework "make dev/make jupyter" build tasks
Luciano Resende created TOREE-511: - Summary: Rework "make dev/make jupyter" build tasks Key: TOREE-511 URL: https://issues.apache.org/jira/browse/TOREE-511 Project: TOREE Issue Type: Task Components: Build Reporter: Luciano Resende Assignee: Luciano Resende Fix For: 0.4.0 Toree has `make dev` and `make jupyter` commands that are using some old dependencies and some weird way to build them. We should create a docker file that extends from `jupyter/all-spark-notebook` to make maintenance easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-510: - Assignee: Luciano Resende > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés > Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17001806#comment-17001806 ] Luciano Resende commented on TOREE-510: --- Let me take a look if there are any cleanups needed, and update Apache Spark default builds and start working on a release. > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés > Assignee: Luciano Resende >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TOREE-510) Toree 0.4.0 Release
[ https://issues.apache.org/jira/browse/TOREE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-510: -- Summary: Toree 0.4.0 Release (was: New Toree Release) > Toree 0.4.0 Release > --- > > Key: TOREE-510 > URL: https://issues.apache.org/jira/browse/TOREE-510 > Project: TOREE > Issue Type: Wish >Affects Versions: 0.4.0 >Reporter: Andrés >Priority: Minor > > Hello, > I'd like to know if there is an estimated release date for Toree 0.4.0. > > Greetings, > Andrés Doncel -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Podling Toree Report Reminder - November 2019
Report available at https://cwiki.apache.org/confluence/display/INCUBATOR/November2019 Mentors, please sign-off the report. On Fri, Nov 1, 2019 at 1:24 AM wrote: > > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare your quarterly board report. > > The board meeting is scheduled for Wed, 20 November 2019, 10:30 am PDT. > The report for your podling will form a part of the Incubator PMC > report. The Incubator PMC requires your report to be submitted 2 weeks > before the board meeting, to allow sufficient time for review and > submission (Wed, November 06). > > Please submit your report with sufficient time to allow the Incubator > PMC, and subsequently board members to review and digest. Again, the > very latest you should submit your report is 2 weeks prior to the board > meeting. > > Candidate names should not be made public before people are actually > elected, so please do not include the names of potential committers or > PPMC members in your report. > > Thanks, > > The Apache Incubator PMC > > Submitting your Report > > -- > > Your report should contain the following: > > * Your project name > * A brief description of your project, which assumes no knowledge of > the project or necessarily of its field > * A list of the three most important issues to address in the move > towards graduation. > * Any issues that the Incubator PMC or ASF Board might wish/need to be > aware of > * How has the community developed since the last report > * How has the project developed since the last report. > * How does the podling rate their own maturity. > > This should be appended to the Incubator Wiki page at: > > https://cwiki.apache.org/confluence/display/INCUBATOR/November2019 > > Note: This is manually populated. You may need to wait a little before > this page is created from a template. > > Note: The format of the report has changed to use markdown. > > Mentors > --- > > Mentors should review reports for their project(s) and sign them off on > the Incubator wiki page. Signing off reports shows that you are > following the project - projects that are not signed may raise alarms > for the Incubator PMC. > > Incubator PMC -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-421.
---
Fix Version/s: 0.4.0
Assignee: Kevin Bates
Resolution: Fixed
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TOREE-506) Enable TravisCI to build using Open JDK
Luciano Resende created TOREE-506: - Summary: Enable TravisCI to build using Open JDK Key: TOREE-506 URL: https://issues.apache.org/jira/browse/TOREE-506 Project: TOREE Issue Type: Task Components: Build Reporter: Luciano Resende Recently TravisCI build on JDK 8 started failing, and running the tests on openJDK enables a safe net in case of JDK not being available on the user env. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TOREE-505) Travis CI builds are failing due to issues installing JDK8
Luciano Resende created TOREE-505: - Summary: Travis CI builds are failing due to issues installing JDK8 Key: TOREE-505 URL: https://issues.apache.org/jira/browse/TOREE-505 Project: TOREE Issue Type: Bug Components: Build Reporter: Luciano Resende Assignee: Luciano Resende $ ~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL" Ignoring license option: BCL -- using GPLv2+CE by default install-jdk.sh 2019-10-16 Expected feature release number in range of 9 to 14, but got: 8 The command "~/bin/install-jdk.sh --target "/home/travis/oraclejdk8" --workspace "/home/travis/.cache/install-jdk" --feature "8" --license "BCL"" failed and exited with 3 during . Details on the following build log: https://travis-ci.org/apache/incubator-toree?utm_medium=notification_source=email -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups
[
https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955245#comment-16955245
]
Luciano Resende commented on TOREE-421:
---
Sorry for the delay, I particularly don't have strong opinions one way or
another, but it seems that there are at least a few use cases where folks would
benefit from having a little more flexibility if we could enable different
thread groups properly, so +1 for that.
> KernelSecurityManager doesn't allow users to create their own thread groups
> ---
>
> Key: TOREE-421
> URL: https://issues.apache.org/jira/browse/TOREE-421
> Project: TOREE
> Issue Type: Bug
>Reporter: Piyush Narang
>Priority: Major
>
> I'm trying to run a Spark Scala job using Toree and I'm running into some
> issues as the code in our job calls into one of our libraries which tries to
> create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> Stack looks like:
> {code}
> Name: java.lang.SecurityException
> Message: Not allowed to modify ThreadGroups!
> StackTrace: at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> at java.lang.Thread.init(Thread.java:394)
> at java.lang.Thread.init(Thread.java:349)
> at java.lang.Thread.(Thread.java:599)
> at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> ...
> {code}
> Here's a simple repro:
> {code}
> println(Thread.currentThread().getThreadGroup) // default thread group
> val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> val hello = new Thread(group, new Runnable {
> def run() {
> println("hello world")
> }
> })
> println(hello.getThreadGroup)
> hello.start
> {code}
> Any suggestions for working around this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-504) Version missing from shutdown_reply header
[
https://issues.apache.org/jira/browse/TOREE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933050#comment-16933050
]
Luciano Resende commented on TOREE-504:
---
[~kbates] Do we need a release with this change?
> Version missing from shutdown_reply header
> --
>
> Key: TOREE-504
> URL: https://issues.apache.org/jira/browse/TOREE-504
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> The 5.0 Jupyter message protocol adds a {{version}} field to [message
> headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
> However, shutdown_reply headers are not setting the protocol version. As a
> result, clients that adhere to the protocol can have issues when {{version}}
> is present in the response header as an empty string, leading to noise and
> possible incorrect behaviors following Toree's shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Podling Report Reminder - September 2019
On Fri, Aug 30, 2019 at 6:39 PM wrote: > > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare your quarterly board report. > > The board meeting is scheduled for Wed, 18 September 2019, 10:30 am PDT. > The report for your podling will form a part of the Incubator PMC > report. The Incubator PMC requires your report to be submitted 2 weeks > before the board meeting, to allow sufficient time for review and > submission (Wed, September 04). > > Please submit your report with sufficient time to allow the Incubator > PMC, and subsequently board members to review and digest. Again, the > very latest you should submit your report is 2 weeks prior to the board > meeting. > > Candidate names should not be made public before people are actually > elected, so please do not include the names of potential committers or > PPMC members in your report. > > Thanks, > > The Apache Incubator PMC > > Submitting your Report > > -- > > Your report should contain the following: > > * Your project name > * A brief description of your project, which assumes no knowledge of > the project or necessarily of its field > * A list of the three most important issues to address in the move > towards graduation. > * Any issues that the Incubator PMC or ASF Board might wish/need to be > aware of > * How has the community developed since the last report > * How has the project developed since the last report. > * How does the podling rate their own maturity. > > This should be appended to the Incubator Wiki page at: > > https://cwiki.apache.org/confluence/display/INCUBATOR/September2019 > > Note: This is manually populated. You may need to wait a little before > this page is created from a template. > > Note: The format of the report has changed to use markdown. > > Mentors > --- > > Mentors should review reports for their project(s) and sign them off on > the Incubator wiki page. Signing off reports shows that you are > following the project - projects that are not signed may raise alarms > for the Incubator PMC. > > Incubator PMC Toree's report has been filed. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-504) Version missing from shutdown_reply header
[
https://issues.apache.org/jira/browse/TOREE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-504.
---
Fix Version/s: 0.4.0
Assignee: Kevin Bates
Resolution: Fixed
> Version missing from shutdown_reply header
> --
>
> Key: TOREE-504
> URL: https://issues.apache.org/jira/browse/TOREE-504
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
>Reporter: Kevin Bates
>Assignee: Kevin Bates
>Priority: Major
> Fix For: 0.4.0
>
>
> The 5.0 Jupyter message protocol adds a {{version}} field to [message
> headers|https://jupyter-client.readthedocs.io/en/latest/messaging.html#general-message-format].
> However, shutdown_reply headers are not setting the protocol version. As a
> result, clients that adhere to the protocol can have issues when {{version}}
> is present in the response header as an empty string, leading to noise and
> possible incorrect behaviors following Toree's shutdown.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
Re: About recent PySpark and SparkR interpreters Removal
On Mon, Jul 1, 2019 at 9:08 PM Eric Charles wrote: > > Hi, > > I have lately commented on the PySpark and SparkR interpreters Removal Pull > Request. > > https://github.com/apache/incubator-toree/pull/166#issuecomment-500356895 > > If it is too late to bring them back in the picture: I wonder > > - What are your advices to benefit from a shared spark context between > Scala/Python/R? > - What were the encountered difficulties with that integration - I wonder how > possible it is for me to maintain that in a fork? > > Thx, Eric > I would say that it mostly started as a prototype and there were issues being reported that were not being addressed by the community. In counterpart, there were other feature-rich kernel implementations such as IPython and IRKernel that were providing the necessary integration with Spark. As for shared context support, see comments below from @Gino Bustelo https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01873.html -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Apache Toree presence at ApacheCon Las Vegas
We have an accepted session at ApacheCon Las Vegas happening this September https://www.apachecon.com/acna19/s/#/scheduledEvent/1089 Please join us and come say hi... -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-502) Toree TravisCI fails installing OpenJDK
[ https://issues.apache.org/jira/browse/TOREE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-502. --- Resolution: Fixed Fix Version/s: 0.4.0 > Toree TravisCI fails installing OpenJDK > --- > > Key: TOREE-502 > URL: https://issues.apache.org/jira/browse/TOREE-502 > Project: TOREE > Issue Type: Bug > Components: Build > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.4.0 > > > Here is the logs from Travis CI > https://travis-ci.org/apache/incubator-toree/jobs/546650927 > Removing intermediate container e1ac74d99565 > Step 4/6 : RUN echo 'deb http://ftp.debian.org/debian jessie-backports main' > > /etc/apt/sources.list.d/jessie-backports.list && apt-get -y update && > apt-get install -y --no-install-recommends -t jessie-backports > openjdk-8-jre-headless && rm > /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm > -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && > update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java > ---> Running in a548a8ea42e3 > Ign http://ftp.debian.org jessie-backports InRelease > Get:1 http://security.debian.org jessie/updates InRelease [44.9 kB] > Get:2 http://repos.mesosphere.io wheezy InRelease [5,279 B] > Ign http://httpredir.debian.org jessie InRelease > Get:3 http://httpredir.debian.org jessie-updates InRelease [7,340 B] > Get:4 http://httpredir.debian.org jessie Release.gpg [2,420 B] > Get:5 http://httpredir.debian.org jessie Release [148 kB] > Ign http://ftp.debian.org jessie-backports Release.gpg > Get:6 http://security.debian.org jessie/updates/main amd64 Packages [851 kB] > Ign http://ftp.debian.org jessie-backports Release > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Get:7 http://repos.mesosphere.io wheezy/main amd64 Packages [3,625 B] > Get:8 http://httpredir.debian.org jessie/main amd64 Packages [9,098 kB] > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Err http://ftp.debian.org jessie-backports/main amd64 Packages > > Err http://ftp.debian.org jessie-backports/main amd64 Packages > 404 Not Found > W: Failed to fetch > http://httpredir.debian.org/debian/dists/jessie-updates/InRelease Unable to > find expected entry 'main/binary-amd64/Packages' in Release file (Wrong > sources.list entry or malformed file) > W: Failed to fetch > http://ftp.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages > 404 Not Found > E: Some index files failed to download. They have been ignored, or old ones > used instead. > Fetched 10.2 MB in 1s (5,361 kB/s) > The command '/bin/sh -c echo 'deb http://ftp.debian.org/debian > jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list && > apt-get -y update && apt-get install -y --no-install-recommends -t > jessie-backports openjdk-8-jre-headless && rm > /etc/apt/sources.list.d/jessie-backports.list && apt-get clean && rm > -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/* && > update-alternatives --set java > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java' returned a non-zero code: 100 > make: *** [.system-test-image] Error 100 > The command "make all-travis" exited with 2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TOREE-500) Failed to start Toree on Spark 2.4.2
[
https://issues.apache.org/jira/browse/TOREE-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853837#comment-16853837
]
Luciano Resende edited comment on TOREE-500 at 6/1/19 11:32 PM:
I will take a look at this, but I am going to be slow due to day job
workload... having said that, feel free to provide patches if you have some
cycles.
The ideal solution would be to produce scala 2.11 and 2.12 artifacts when
performing a build/release.
was (Author: luciano resende):
I will take a look at this, but I am going to be slow due to day job
workload... having said that, feel free to provide patches if you have some
cycles.
The ideal solution would be to produce scala 2.11 and 2.12 artifacts when
performing a release.
> Failed to start Toree on Spark 2.4.2
>
>
> Key: TOREE-500
> URL: https://issues.apache.org/jira/browse/TOREE-500
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: * Docker Container(openjdk8)
> * Python 3.5
>Reporter: Masayuki Takahashi
>Priority: Major
>
> {code}
> [I 04:07:48.726 NotebookApp] Kernel started:
> 1d46275c-71d6-4062-bc22-a98593d6c374
> Starting Spark Kernel with SPARK_HOME=/spark
> 19/05/05 04:07:53 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
> at org.apache.toree.Main$.(Main.scala:24)
> at org.apache.toree.Main$.(Main.scala)
> at org.apache.toree.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: scala.App$class
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
> Toree on Spark 2.4.1 works for me.
> See: https://github.com/jupyter/docker-stacks/issues/860
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-500) Failed to start Toree on Spark 2.4.2
[
https://issues.apache.org/jira/browse/TOREE-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853837#comment-16853837
]
Luciano Resende commented on TOREE-500:
---
I will take a look at this, but I am going to be slow due to day job
workload... having said that, feel free to provide patches if you have some
cycles.
The ideal solution would be to produce scala 2.11 and 2.12 artifacts when
performing a release.
> Failed to start Toree on Spark 2.4.2
>
>
> Key: TOREE-500
> URL: https://issues.apache.org/jira/browse/TOREE-500
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.3.0
> Environment: * Docker Container(openjdk8)
> * Python 3.5
>Reporter: Masayuki Takahashi
>Priority: Major
>
> {code}
> [I 04:07:48.726 NotebookApp] Kernel started:
> 1d46275c-71d6-4062-bc22-a98593d6c374
> Starting Spark Kernel with SPARK_HOME=/spark
> 19/05/05 04:07:53 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
> at org.apache.toree.Main$.(Main.scala:24)
> at org.apache.toree.Main$.(Main.scala)
> at org.apache.toree.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
> at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: scala.App$class
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
> Toree on Spark 2.4.1 works for me.
> See: https://github.com/jupyter/docker-stacks/issues/860
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Draft Incubator report
I have added a draft incubator report at https://cwiki.apache.org/confluence/display/INCUBATOR/June2019 @All, please review and provide comments and suggestions @mentors, please review and sign-off the report -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Comment Edited] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789841#comment-16789841
]
Luciano Resende edited comment on TOREE-495 at 3/11/19 9:40 PM:
The current implementation takes anything added during the notebook runtime
(e.g. %addDeps and %addJar) as transient, and it is created as a temporary file
that is deleted after the process finished independently of the location where
it is created.
It seems that, what you are looking for, is the ability to make a list of
"dependencies" available in kernels. You can accomplish that by updating the
Toree run.sh to list and submit dependencies as part of --packages or --jars
from spark-submit which will take care of making those available in the worker
nodes.
was (Author: luciano resende):
The current implementation takes anything added during the notebook runtime
(e.g. %addDeps and %addJar) as transient, and independent of the location it is
created it is created using the createTmp OS API that deletes the file after
the process is deleted. It seems that, what you need is just the ability to
have the user "dependencies" available in kernels, and you can do that by doing
the map as you mentioned together with updating the kernelspecto use additional
PYTHONPATH configuration (e.g. --conf
spark.yarn.appMasterEnv.PYTHONPATH=${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip)
the only caveat is that, permanent libraries would have to be added outside of
the notebook environment.
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-496) Toree Kernel fail to add Brunel Magic
[ https://issues.apache.org/jira/browse/TOREE-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-496. --- Resolution: Not A Bug Assignee: Luciano Resende Fix Version/s: Not Applicable The version in use is too old and incompatible with Spark 2.3. Please use version 2.6 or above > Toree Kernel fail to add Brunel Magic > - > > Key: TOREE-496 > URL: https://issues.apache.org/jira/browse/TOREE-496 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: SHOBHIT SHUKLA > Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > Attachments: Screenshot 2019-03-11 at 11.31.22 PM.png, Screenshot > 2019-03-11 at 11.31.35 PM.png, Spark 2.3.2 and Brunel 2.6 - Brunel Graph.png, > Spark 2.3.2 and Brunel 2.6.png > > > Toree Kernel fail to add Brunel Magic : > %AddJar -magic [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] > Starting download from > [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] Finished download > of spark-kernel-brunel-all-2.2.jar > Out[2]: Magic AddJar failed to execute with error: > org/apache/toree/magic/CellMagic > > ENV : Spark 2.3.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TOREE-496) Toree Kernel fail to add Brunel Magic
[ https://issues.apache.org/jira/browse/TOREE-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-496: -- Attachment: Spark 2.3.2 and Brunel 2.6.png Spark 2.3.2 and Brunel 2.6 - Brunel Graph.png > Toree Kernel fail to add Brunel Magic > - > > Key: TOREE-496 > URL: https://issues.apache.org/jira/browse/TOREE-496 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: SHOBHIT SHUKLA > Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > Attachments: Screenshot 2019-03-11 at 11.31.22 PM.png, Screenshot > 2019-03-11 at 11.31.35 PM.png, Spark 2.3.2 and Brunel 2.6 - Brunel Graph.png, > Spark 2.3.2 and Brunel 2.6.png > > > Toree Kernel fail to add Brunel Magic : > %AddJar -magic [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] > Starting download from > [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] Finished download > of spark-kernel-brunel-all-2.2.jar > Out[2]: Magic AddJar failed to execute with error: > org/apache/toree/magic/CellMagic > > ENV : Spark 2.3.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-496) Toree Kernel fail to add Brunel Magic
[ https://issues.apache.org/jira/browse/TOREE-496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789912#comment-16789912 ] Luciano Resende commented on TOREE-496: --- According to Shobhit, this is using Spark 2.3.0 > Toree Kernel fail to add Brunel Magic > - > > Key: TOREE-496 > URL: https://issues.apache.org/jira/browse/TOREE-496 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: SHOBHIT SHUKLA >Priority: Major > Attachments: Screenshot 2019-03-11 at 11.31.22 PM.png, Screenshot > 2019-03-11 at 11.31.35 PM.png > > > Toree Kernel fail to add Brunel Magic : > %AddJar -magic [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] > Starting download from > [https://brunelvis.org/jar/spark-kernel-brunel-all-2.2.jar] Finished download > of spark-kernel-brunel-all-2.2.jar > Out[2]: Magic AddJar failed to execute with error: > org/apache/toree/magic/CellMagic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789844#comment-16789844
]
Luciano Resende commented on TOREE-495:
---
BTW, we have been doing something similar on a layer between Jupyter and the
Kernel in Jupyter Enterprise Gateway for Kubernetes ([see this blog
post|https://blog.jupyter.org/on-demand-notebooks-with-jupyterhub-jupyter-enterprise-gateway-and-kubernetes-e8e423695cbf])
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789841#comment-16789841
]
Luciano Resende commented on TOREE-495:
---
The current implementation takes anything added during the notebook runtime
(e.g. %addDeps and %addJar) as transient, and independent of the location it is
created it is created using the createTmp OS API that deletes the file after
the process is deleted. It seems that, what you need is just the ability to
have the user "dependencies" available in kernels, and you can do that by doing
the map as you mentioned together with updating the kernelspecto use additional
PYTHONPATH configuration (e.g. --conf
spark.yarn.appMasterEnv.PYTHONPATH=${HOME}/.local/lib/python2.7/site-packages:/usr/hdp/current/spark2-client/python:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip)
the only caveat is that, permanent libraries would have to be added outside of
the notebook environment.
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-495) Add deps-dir option to toree configuration options
[
https://issues.apache.org/jira/browse/TOREE-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788099#comment-16788099
]
Luciano Resende commented on TOREE-495:
---
[~standa] I thought I was following your use case, untill the moment you
mentioned that the --deps-dir would be defined during the installation. To me,
%addJar and %addDeps are to add dependencies required to a specific notebook
and are transient in the sense that are not always required to everybody, or
different users might have different versions, and so on... If you sort of know
where these are, and these are always on the same location, why not add the
path to Python classpath in the kernelspec?
Please feel free to give more details on what issue you are trying to solve by
adding the --deps-dir outside from the magic parameters itself. Also, please
consider how would this work in a distributed environment (e.g. YARN cluster
mode).
> Add deps-dir option to toree configuration options
> --
>
> Key: TOREE-495
> URL: https://issues.apache.org/jira/browse/TOREE-495
> Project: TOREE
> Issue Type: Improvement
>Reporter: Stanislav Kokaisl
>Priority: Major
>
> It would be helpful to have option, where are dependencies stored with
> AddDeps magic. Currently there is only {{jar-dir option for AddJar magic.}}
> {{We create new deps-dir attribute for toree_opts. This option can be then
> used in install command:}}
> {code:java}
> jupyter toree install --toree_opts="--deps-dir=/path/to/deps"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Incubator Podling Report (Due 6th February)
Please review the initial draft report at https://wiki.apache.org/incubator/February2019 On Sat, Feb 2, 2019 at 9:26 PM Justin Mclean wrote: > > Hi, > > Is someone form PPMC working on the report? It's due in a few days. > > Thanks, > Justin -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Commented] (TOREE-493) The location for Dev snapshots of Toree is empty
[ https://issues.apache.org/jira/browse/TOREE-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738509#comment-16738509 ] Luciano Resende commented on TOREE-493: --- Agreed, please reopen and submit a patch for the docs. Thanks for catching this... > The location for Dev snapshots of Toree is empty > > > Key: TOREE-493 > URL: https://issues.apache.org/jira/browse/TOREE-493 > Project: TOREE > Issue Type: Bug > Components: Website >Affects Versions: Not Applicable >Reporter: Romulo Goncalves >Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > > The documentation to install Toree: > [https://github.com/apache/incubator-toree#install] > Tells us from where to get Dev Snapshots: > "Dev snapshots of Toree are located at > [https://dist.apache.org/repos/dist/dev/incubator/toree]. " > However, the location is empty. Was toree snapshots moved elsewhere and > documentation not updated? Or something happened to the server? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-493) The location for Dev snapshots of Toree is empty
[ https://issues.apache.org/jira/browse/TOREE-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-493. --- Resolution: Won't Fix Assignee: Luciano Resende Users should use the toree release available from pip (0.3.0) Contributors should build master. Having said that, 0.3.0 is very current. > The location for Dev snapshots of Toree is empty > > > Key: TOREE-493 > URL: https://issues.apache.org/jira/browse/TOREE-493 > Project: TOREE > Issue Type: Bug > Components: Website >Affects Versions: Not Applicable >Reporter: Romulo Goncalves >Assignee: Luciano Resende >Priority: Major > Fix For: Not Applicable > > > The documentation to install Toree: > [https://github.com/apache/incubator-toree#install] > Tells us from where to get Dev Snapshots: > "Dev snapshots of Toree are located at > [https://dist.apache.org/repos/dist/dev/incubator/toree]. " > However, the location is empty. Was toree snapshots moved elsewhere and > documentation not updated? Or something happened to the server? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [NOTICE] Mandatory migration of git repositories to gitbox.apache.org
Please see e-mail below, I will create an Infra JIRA to move us to the new gitbox. Please respond to this thread in case you have any issues and/or questions. On Thu, Jan 3, 2019 at 8:19 AM Apache Infrastructure Team wrote: > > Hello, toree folks. > As stated earlier in 2018, all git repositories must be migrated from > the git-wip-us.apache.org URL to gitbox.apache.org, as the old service > is being decommissioned. Your project is receiving this email because > you still have repositories on git-wip-us that needs to be migrated. > > The following repositories on git-wip-us belong to your project: > - incubator-toree-website.git > - incubator-toree.git > > > We are now entering the mandated (coordinated) move stage of the roadmap, > and you are asked to please coordinate migration with the Apache > Infrastructure Team before February 7th. All repositories not migrated > on February 7th will be mass migrated without warning, and we'd appreciate > it if we could work together to avoid a big mess that day :-). > > Moving to gitbox means you will get full write access on GitHub as well, > and be able to close/merge pull requests and much more. > > To have your repositories moved, please follow these steps: > > - Ensure consensus on the move (a link to a lists.apache.org thread will > suffice for us as evidence). > - Create a JIRA ticket at https://issues.apache.org/jira/browse/INFRA > > Your migration should only take a few minutes. If you wish to migrate > at a specific time of day or date, please do let us know in the ticket. > > As always, we appreciate your understanding and patience as we move > things around and work to provide better services and features for > the Apache Family. > > Should you wish to contact us with feedback or questions, please do so > at: us...@infra.apache.org. > > > With regards, > Apache Infrastructure > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[ANNOUNCE] Apache Toree 0.3.0-incubating Released
Apache Toree is a kernel for the Jupyter Notebook platform providing interactive and remote access to Apache Spark. The Apache Toree community is pleased to announce the release of Apache Toree 0.3.0-incubating which provides various bug fixes and the following enhancements. * Fix JupyterLab support after the introduction of new cell metadata information * Support for high-order functions * Fix %Showtypes and %Truncate magics * Added %ShowOutput magic to disable console output * Added support for custom resolvers for %AddDeps magic * Added support for predefined variables in SQL Statements * Removed support for PySpark and Spark R in Toree (use specific kernels) For more information about Apache Toree and go download the latest release go to: https://toree.incubator.apache.org/ For more information on how to use Apache Toree please visit our documentation page: https://toree.incubator.apache.org/docs/current/user/quick-start/ -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Updated] (TOREE-459) Failure when adding a Jar.
[
https://issues.apache.org/jira/browse/TOREE-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende updated TOREE-459:
--
Fix Version/s: (was: 0.3.0)
0.4.0
> Failure when adding a Jar.
> --
>
> Key: TOREE-459
> URL: https://issues.apache.org/jira/browse/TOREE-459
> Project: TOREE
> Issue Type: Bug
>Reporter: Naveen Swamy
> Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.4.0
>
>
> I am trying to use toree with Spark 2.2.7(latest) and Scala 2.11 however when
> I add a jar it fails with the error below
> {{
> spark-2.2.7/jars/scala-library-2.11.8.jar(scala/reflect/package.class)' is
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> error: error while loading package, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/package.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> error: error while loading Serializable, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Serializable.class)' is
> broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> res2:
> Either[org.apache.toree.magic.CellMagicOutput,org.apache.toree.magic.LineMagicOutput]
> = Right(())
> error: error while loading Predef, class file
> '/spark-2.2.7/jars/scala-library-2.11.8.jar(scala/Predef.class)' is broken
> (class java.lang.RuntimeException/Scala class file does not contain Scala
> annotation)
> }}
> The jar I am trying to add is:
> %AddJar
> https://search.maven.org/remotecontent?filepath=ml/dmlc/mxnet/mxnet-full_2.11-osx-x86_64-cpu/0.9.3a/mxnet-full_2.11-osx-x86_64-cpu-0.9.3a.jar
> Spark version 2.2.1
> Scala version 2.11.8
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[RESULT][VOTE] Apache Toree 0.3.0 RC1
On Tue, Oct 16, 2018 at 9:30 AM Luciano Resende wrote: > > Please vote to approve the release of Apache Toree 0.3.0-incubating (RC1). > > Tag: v0.3.0-incubating-rc1 (3ff5df33cc459ea214e542a4798fd9ba6e0a2ef8) > > https://github.com/apache/incubator-toree/tree/v0.3.0-incubating-rc1 > > All distribution packages, including signatures, digests, etc. can be found > at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1014 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) > > 2. Install Apache Toree via `pip install > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.3.0-incubating > [ ] -1 Do not release this package because ... > Can't seem to find the previous attempt to send the result, so sending it again The vote has passed with +1 from: Luciano Resende Marius van Niekerk Ryan Blue -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-491) Jupyter Kernel creation has bug in kernel.json and SparkR doesn't work
[ https://issues.apache.org/jira/browse/TOREE-491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-491. --- Resolution: Won't Fix Assignee: Luciano Resende Fix Version/s: 0.3.0 Toree SparkR is not supported, please see details at https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html > Jupyter Kernel creation has bug in kernel.json and SparkR doesn't work > -- > > Key: TOREE-491 > URL: https://issues.apache.org/jira/browse/TOREE-491 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Pasle Choix >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > After Toree installation for Scala and SparkR, both kernel.json files contain > an entry that is wrong: > sudo pip install --upgrade toree > sudo jupyter toree install > --spark_home=/opt/cloudera/parcels/SPARK2/lib/spark2 --interpreters=Scala > > Kernel: > "/opt/cloudera/parcels/SPARK2/lib/spark2": > "/opt/cloudera/parcels/SPARK2/lib/spark2" > > After I modified the name from "/opt/cloudera/parcels/SPARK2/lib/spark2" to > "SPARK_HOME", Scala kernel seems working now, but SparkR kernel cannot be > started still. > > Can anyone help to sort it out please? Thank you very much. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Apache Toree 0.3.0 RC1
On Tue, Oct 16, 2018 at 9:30 AM Luciano Resende wrote: > > Please vote to approve the release of Apache Toree 0.3.0-incubating (RC1). > > Tag: v0.3.0-incubating-rc1 (3ff5df33cc459ea214e542a4798fd9ba6e0a2ef8) > > https://github.com/apache/incubator-toree/tree/v0.3.0-incubating-rc1 > > All distribution packages, including signatures, digests, etc. can be found > at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1014 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) > > 2. Install Apache Toree via `pip install > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.3.0-incubating > [ ] -1 Do not release this package because ... > Off course, my +1 -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[VOTE] Apache Toree 0.3.0 RC1
Please vote to approve the release of Apache Toree 0.3.0-incubating (RC1). Tag: v0.3.0-incubating-rc1 (3ff5df33cc459ea214e542a4798fd9ba6e0a2ef8) https://github.com/apache/incubator-toree/tree/v0.3.0-incubating-rc1 All distribution packages, including signatures, digests, etc. can be found at: https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/ Staging artifacts can be found at: https://repository.apache.org/content/repositories/orgapachetoree-1014 ## Testing Instructions The fastest way to get up and running is to using Jupyter. 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) 2. Install Apache Toree via `pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz` followed by `jupyter toree install` - You need to set a valid Apache Spark 2.x home, which can be done via `jupyter toree install --spark_home=/usr/local/spark` - You may need to run with `sudo` for installation permission - For all installation options, run `jupyter toree install --help-all` 4. Run a Jupyter notebook server via `jupyter notebook` - If the notebook portion of Jupyter is not installed but Jupyter is, you can install via `pip install notebook` 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new dev 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the notebook ## Voting Instructions The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Toree 0.3.0-incubating [ ] -1 Do not release this package because ... -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Commented] (TOREE-489) Apache Toree kernel throw error when executing code in jupyter lab
[
https://issues.apache.org/jira/browse/TOREE-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642528#comment-16642528
]
Luciano Resende commented on TOREE-489:
---
Please use Toree master, as the fix has already been merged
> Apache Toree kernel throw error when executing code in jupyter lab
> --
>
> Key: TOREE-489
> URL: https://issues.apache.org/jira/browse/TOREE-489
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: Not Applicable
> Environment: OS: Ubuntu 18:10
> Jupyter lab = 0.35.0
> Python = 3.7.0
> Jupyter notebook = 5.7.0
> Spark = 2.3.2
> Java = 1.8.0_181
> Scala = 2.11.8
>Reporter: Felipe Kenji Yoshida
>Priority: Minor
> Fix For: 0.3.0
>
>
> When I try to execute any code (Ex: "_5+5_") in the Apache Toree Kernel
> inside jupyter lab, no output is visible in the notebook and the jupyter lab
> log contains the following message:
> {code:java}
> 2018-10-08 21:26:50 ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-10-08 21:26:50 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748)
> 2018-10-08 21:26:50 ERROR
> LocalActorRefProvider(akka://spark-kernel-actor-system):159 - guardian
> failed, shutting down system
> java.lang.AssertionError: assertion failed: Socket is not alive to be able to
> send messages!
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.toree.communication.socket.JeroMQSocket.send(JeroMQSocket.scala:39)
> at
> org.apache.toree.communication.actors.RouterSocketActor$$anonfun$receive$1.applyOrElse(RouterSocketActor.scala:47)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.communication.actors.RouterSocketActor.aroundReceive(RouterSocketActor.scala:31)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.sc
[jira] [Closed] (TOREE-489) Apache Toree kernel throw error when executing code in jupyter lab
[
https://issues.apache.org/jira/browse/TOREE-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende closed TOREE-489.
-
Resolution: Duplicate
Fix Version/s: 0.3.0
> Apache Toree kernel throw error when executing code in jupyter lab
> --
>
> Key: TOREE-489
> URL: https://issues.apache.org/jira/browse/TOREE-489
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: Not Applicable
> Environment: OS: Ubuntu 18:10
> Jupyter lab = 0.35.0
> Python = 3.7.0
> Jupyter notebook = 5.7.0
> Spark = 2.3.2
> Java = 1.8.0_181
> Scala = 2.11.8
>Reporter: Felipe Kenji Yoshida
>Priority: Minor
> Fix For: 0.3.0
>
>
> When I try to execute any code (Ex: "_5+5_") in the Apache Toree Kernel
> inside jupyter lab, no output is visible in the notebook and the jupyter lab
> log contains the following message:
> {code:java}
> 2018-10-08 21:26:50 ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-10-08 21:26:50 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748)
> 2018-10-08 21:26:50 ERROR
> LocalActorRefProvider(akka://spark-kernel-actor-system):159 - guardian
> failed, shutting down system
> java.lang.AssertionError: assertion failed: Socket is not alive to be able to
> send messages!
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.toree.communication.socket.JeroMQSocket.send(JeroMQSocket.scala:39)
> at
> org.apache.toree.communication.actors.RouterSocketActor$$anonfun$receive$1.applyOrElse(RouterSocketActor.scala:47)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.communication.actors.RouterSocketActor.aroundReceive(RouterSocketActor.scala:31)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dis
Release 0.3.0 ?
We have done some good progress on some cleanup and annoying bugs and I was wondering if we should look into creating our 0.3 release sometime soon. Thoughts? Any other important issues that should be addressed? -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-488) Remove Spark R intepreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-488. --- Resolution: Fixed Fix Version/s: 0.3.0 > Remove Spark R intepreter support in Toree > -- > > Key: TOREE-488 > URL: https://issues.apache.org/jira/browse/TOREE-488 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-487) Remove PySpark Interpreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-487. --- Resolution: Fixed Fix Version/s: 0.3.0 > Remove PySpark Interpreter support in Toree > --- > > Key: TOREE-487 > URL: https://issues.apache.org/jira/browse/TOREE-487 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-488) Remove Spark R intepreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-488: - Assignee: Luciano Resende > Remove Spark R intepreter support in Toree > -- > > Key: TOREE-488 > URL: https://issues.apache.org/jira/browse/TOREE-488 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-487) Remove PySpark Interpreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-487: - Assignee: Luciano Resende > Remove PySpark Interpreter support in Toree > --- > > Key: TOREE-487 > URL: https://issues.apache.org/jira/browse/TOREE-487 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-488) Remove Spark R intepreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637694#comment-16637694 ] Luciano Resende commented on TOREE-488: --- GitHub user lresende opened a pull request: [https://github.com/apache/incubator-toree/pull/166] TOREE-487TOREE-488 Remove PySpark and SparkR interpreters Instead, please use a supported kernel such IPython or IRKernel Closes #166 You can merge this pull request into a Git repository by running: $ git pull [https://github.com/lresende/incubator-toree] toree-487-488-clean-python-r Alternatively you can review and apply these changes as the patch at: [https://github.com/apache/incubator-toree/pull/166.patch] To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #166 commit 1459403fc8ad5b4bb1e956513ef4190a0fc5b6f4 Author: Luciano Resende Date: 2018-10-03T19:26:14Z TOREE-487TOREE-488 Remove PySpark and SparkR interpreters Instead, please use a supported kernel such IPython or IRKernel Closes #166 > Remove Spark R intepreter support in Toree > -- > > Key: TOREE-488 > URL: https://issues.apache.org/jira/browse/TOREE-488 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-482) Truncation magic does not work
[ https://issues.apache.org/jira/browse/TOREE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-482. --- Resolution: Fixed Fix Version/s: 0.3.0 > Truncation magic does not work > -- > > Key: TOREE-482 > URL: https://issues.apache.org/jira/browse/TOREE-482 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TOREE-487) Remove PySpark Interpreter support in Toree
[ https://issues.apache.org/jira/browse/TOREE-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende updated TOREE-487: -- Component/s: Kernel > Remove PySpark Interpreter support in Toree > --- > > Key: TOREE-487 > URL: https://issues.apache.org/jira/browse/TOREE-487 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende >Priority: Major > > Detailed discussion on the mailing-list > https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-488) Remove Spark R intepreter support in Toree
Luciano Resende created TOREE-488: - Summary: Remove Spark R intepreter support in Toree Key: TOREE-488 URL: https://issues.apache.org/jira/browse/TOREE-488 Project: TOREE Issue Type: Improvement Components: Kernel Reporter: Luciano Resende Detailed discussion on the mailing-list https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TOREE-487) Remove PySpark Interpreter support in Toree
Luciano Resende created TOREE-487: - Summary: Remove PySpark Interpreter support in Toree Key: TOREE-487 URL: https://issues.apache.org/jira/browse/TOREE-487 Project: TOREE Issue Type: Improvement Reporter: Luciano Resende Detailed discussion on the mailing-list https://www.mail-archive.com/dev@toree.incubator.apache.org/msg01865.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-486) Typo in %AddJar magic in Scala notebook ==> Strange classloader errors
[ https://issues.apache.org/jira/browse/TOREE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-486: - Assignee: Luciano Resende > Typo in %AddJar magic in Scala notebook ==> Strange classloader errors > --- > > Key: TOREE-486 > URL: https://issues.apache.org/jira/browse/TOREE-486 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.2.0 >Reporter: SHOBHIT SHUKLA >Assignee: Luciano Resende >Priority: Major > Attachments: 5cd4964a-9b1d-11e8-95fe-ec9b604cd332.png > > > If the user makes a typo while entering an %AddJar magic with an HTTP URL, > the Spark classloader ends up in a strange state that produces errors. > Attachment have the error logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34
[
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-485.
---
Resolution: Fixed
Fix Version/s: 0.3.0
> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
> Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Assignee: Luciano Resende
>Priority: Major
> Fix For: 0.3.0
>
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala
> code in Toree. The kernel starts fine, without any warnings, but once I
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the
> issue remains. Is there any way to fix this?
>
> Thanks,
> Sergey
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-475) Sql statement can't resolve predefined variables in scala cells
[
https://issues.apache.org/jira/browse/TOREE-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende resolved TOREE-475.
---
Resolution: Fixed
Fix Version/s: 0.3.0
> Sql statement can't resolve predefined variables in scala cells
> ---
>
> Key: TOREE-475
> URL: https://issues.apache.org/jira/browse/TOREE-475
> Project: TOREE
> Issue Type: Improvement
> Components: Kernel
>Reporter: Huafeng Wang
>Assignee: Huafeng Wang
>Priority: Major
> Fix For: 0.3.0
>
>
> Current the SQL cell does not support code like
> "
> {{%%sql}}
> {{select * from ${tableName} limit ${limitNum}}}
> "
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TOREE-475) Sql statement can't resolve predefined variables in scala cells
[
https://issues.apache.org/jira/browse/TOREE-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende reassigned TOREE-475:
-
Assignee: Huafeng Wang
> Sql statement can't resolve predefined variables in scala cells
> ---
>
> Key: TOREE-475
> URL: https://issues.apache.org/jira/browse/TOREE-475
> Project: TOREE
> Issue Type: Improvement
> Components: Kernel
>Reporter: Huafeng Wang
>Assignee: Huafeng Wang
>Priority: Major
>
> Current the SQL cell does not support code like
> "
> {{%%sql}}
> {{select * from ${tableName} limit ${limitNum}}}
> "
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34
[
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628981#comment-16628981
]
Luciano Resende commented on TOREE-485:
---
Ok, this seems like a recent change in Lab that is passing additional metadata
on the requests and Scala type-safe objects are not accommodating these changes:
The IDs of all deleted cells since the last run cell are now passed as part of
the cell metadata on execution. The IDs of deleted cells since the last run
cell are stored as ``deletedCells`` in ``NotebookModel``. This helps in
developing reactive kernels. (`#5037
<[https://github.com/jupyterlab/jupyterlab/pull/5037]>`__)
Contents looks like:
{"deletedCells":[],"cellId":"d01fdb74-4760-4bd2-91d7-11f85977cb43"}
The issue is that v5.Metadata does not seem to accommodate these new values.
> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
> Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Priority: Major
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala
> code in Toree. The kernel starts fine, without any warnings, but once I
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the
> issue remains. Is there any way to fix this?
>
> Thanks,
> Sergey
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TOREE-485) Not able to run Toree kernel in JupyterLab 0.34
[
https://issues.apache.org/jira/browse/TOREE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628839#comment-16628839
]
Luciano Resende commented on TOREE-485:
---
Just FYI, we have already released Toree 0.2.0 and that is already available on
[PyPi|[https://pypi.org/project/toree].] I would recommend moving to the
official release instead of using the RC.
As for JupyterLab, I have updated to version 0.34.12 and I don't seem to have
an issue during kernel start, but when I try to execute a cell then I DO get
the port already in use exception. I will need to further investigate the
issue, as I don't seem to have a similar issue using the IPython kernel.
> Not able to run Toree kernel in JupyterLab 0.34
> ---
>
> Key: TOREE-485
> URL: https://issues.apache.org/jira/browse/TOREE-485
> Project: TOREE
> Issue Type: Bug
> Environment: Ubuntu 18.04, Python 3.6.6, JupyterLab v0.34.12, Apache
> Toree 0.2.0-rc6
>Reporter: Sergey Kuznetsov
>Priority: Major
>
> Hello! I've been using Toree kernel in JupyterLab (Python 2.7) versions 0.32
> and 0.33 for some time and it worked great. Recently I updated JupyterLab to
> version 0.34 that supports only Python 3 and now I'm not ably to run Scala
> code in Toree. The kernel starts fine, without any warnings, but once I
> executed the cell with simple import I get the following error:
> {noformat}
> ERROR OneForOneStrategy:159 -
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> play.api.libs.json.JsResultException:
> JsResultException(errors:List((/deletedCells,List(ValidationError(error.expected.jsstring,WrappedArray())
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsValue$$anonfun$2.apply(JsValue.scala:67)
> at play.api.libs.json.JsResult$class.fold(JsResult.scala:77)
> at play.api.libs.json.JsError.fold(JsResult.scala:13)
> at play.api.libs.json.JsValue$class.as(JsValue.scala:65)
> at play.api.libs.json.JsObject.as(JsValue.scala:166)
> at
> org.apache.toree.kernel.protocol.v5.kernel.Utilities$.ZMQMessageToKernelMessage(Utilities.scala:69)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket$$anonfun$receive$1.applyOrElse(ZeromqKernelMessageSocket.scala:48)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at
> org.apache.toree.kernel.protocol.v5.kernel.socket.ZeromqKernelMessageSocket.aroundReceive(ZeromqKernelMessageSocket.scala:39)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-09-26 14:45:29 ERROR ZeroMQSocketRunnable:167 - Unexpected exception in
> 0mq socket runnable!
> org.zeromq.ZMQException: Errno 48 : Address already in use
> at org.zeromq.ZMQ$Socket.mayRaise(ZMQ.java:1425)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1187)
> at org.zeromq.ZMQ$Socket.bind(ZMQ.java:1148)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:88)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable$$anonfun$processOptions$2.apply(ZeroMQSocketRunnable.scala:87)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.processOptions(ZeroMQSocketRunnable.scala:87)
> at
> org.apache.toree.communication.socket.ZeroMQSocketRunnable.run(ZeroMQSocketRunnable.scala:144)
> at java.lang.Thread.run(Thread.java:748){noformat}
> I tried to update Toree: install with pip3 the latest rc6 release, but the
> issue remains. Is there any way to fix this?
>
> Thanks,
> Sergey
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-474. --- Resolution: Fixed Fix Version/s: 0.3.0 > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Assignee: Russell McEntyre >Priority: Major > Fix For: 0.3.0 > > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-474: - Assignee: Russell McEntyre > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Assignee: Russell McEntyre >Priority: Major > Fix For: 0.3.0 > > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Removing Toree specific support for Python and R
Toree currently has some support for Python and R which are a thin wrapper around what is provided in Spark. While this enables sharing the Spark session and SQL Context between Scala and Python code, it brings a huge gap on functionality when comparing with other Python kernels like IPython and IRKernel. Adding to that, there is no community member that is actively enhancing and contributing to these two areas. Based on these, I would like to suggest the following to be done for Toree 0.3.0 release timeframe: - Remove Python and R support from Toree - Document possible suggestions on Toree website (e.g. IPython, IRKernel or others more active on their respective community) Some of the benefits: - Avoid user confusion and frustration - Improve kernel startup performance - Cleaner code to maintain Please let me know your thoughts. [1] https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/sqlcontext_sharing.ipynb -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Resolved] (TOREE-483) Add %showOutput magic to disable console output
[ https://issues.apache.org/jira/browse/TOREE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-483. --- Resolution: Fixed Fix Version/s: 0.3.0 > Add %showOutput magic to disable console output > --- > > Key: TOREE-483 > URL: https://issues.apache.org/jira/browse/TOREE-483 > Project: TOREE > Issue Type: Improvement > Components: Kernel > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-484) Toree kernels keeps crashing on Spark 1.6
[ https://issues.apache.org/jira/browse/TOREE-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-484. --- Resolution: Won't Fix Fix Version/s: 0.2.0 Apache Toree 0.1.0 supports Spark 1.x while Apache Toree 0.2.0 supports Spark 2.x. > Toree kernels keeps crashing on Spark 1.6 > - > > Key: TOREE-484 > URL: https://issues.apache.org/jira/browse/TOREE-484 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Environment: RHEL 6.10(Santiago) > Anaconda2-5.2.0 > Python 2.7 > Jupyter 4.4.0 > HDP 2.5.3 >Reporter: Amit Jha >Priority: Major > Fix For: 0.2.0 > > > Have installed Jupyter using Anaconda. Installed Toree using: > pip install toree > jupyter toree uninstall --spark_home=$SPARK_HOME > --interpreters=Scala,PySpark,SparkR,SQL > kernelspec lists all the toree kernels and other jupyter kernels are > running(Python2/R) but all Toree kernels keeps restarting with below > error message: > Starting Spark Kernel with SPARK_HOME=/usr/hdp/current/spark-client > [I 07:47:20.211 NotebookApp] Kernel started: > 81f1ee05-9486-4c44-9c40-0c8557c75675 > Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at > org.apache.toree.boot.CommandLineOptions.toConfig(CommandLineOptions.scala:161) > at org.apache.toree.Main$$anon$1.(Main.scala:35) > at > org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:35) > at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24) > at scala.Function0$class.apply$mcV$sp(Function0.scala:40) > at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) > at scala.App$$anonfun$main$1.apply(App.scala:71) > at scala.App$$anonfun$main$1.apply(App.scala:71) > at scala.collection.immutable.List.foreach(List.scala:318) > at > scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32) > at scala.App$class.main(App.scala:71) > at org.apache.toree.Main$.main(Main.scala:24) > at org.apache.toree.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > [I 07:47:23.209 NotebookApp] KernelRestarter: restarting kernel (1/5), new > random ports > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609673#comment-16609673 ] Luciano Resende commented on TOREE-245: --- Ok, but we must wait for the next release to publish official versions of the other jars. Also, if you want to provide a pr to enable publish them when building a release it would be appreciated: https://github.com/apache/incubator-toree/blob/master/etc/tools/release-build.sh > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin > Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609647#comment-16609647 ] Luciano Resende commented on TOREE-245: --- Also note that, because the assembly jar is shaded, it will enable deploying the kernel in different spark distributions/mode that might bring conflicting dependencies into the classpath. > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin > Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609642#comment-16609642 ] Luciano Resende commented on TOREE-245: --- The kernel or standalone applications only need the assembly jar to work. Maybe you could give us more details about what you are trying to accomplish, etc But otherwise, we were not planning to publish any other jars. > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin > Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-245) Publish Toree jars
[ https://issues.apache.org/jira/browse/TOREE-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-245. --- Resolution: Fixed Assignee: Luciano Resende (was: Gino Bustelo) Fix Version/s: 0.2.0 This is available as of the last release: https://search.maven.org/artifact/org.apache.toree/toree-assembly/0.2.0-incubating/jar > Publish Toree jars > -- > > Key: TOREE-245 > URL: https://issues.apache.org/jira/browse/TOREE-245 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Ron Jadin > Assignee: Luciano Resende >Priority: Major > Fix For: 0.2.0 > > > Hello There- > I'm just offering this as an enhancement request. Would it possible to have > an option available for easy consumption/utilization that does not require > building from source? This is an excellent project. I just think this would > make it more accessible to a broader audience. > Thank you, > John -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TOREE-482) Truncation magic does not work
[ https://issues.apache.org/jira/browse/TOREE-482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende reassigned TOREE-482: - Assignee: Luciano Resende > Truncation magic does not work > -- > > Key: TOREE-482 > URL: https://issues.apache.org/jira/browse/TOREE-482 > Project: TOREE > Issue Type: Bug > Components: Kernel >Affects Versions: 0.2.0 > Reporter: Luciano Resende >Assignee: Luciano Resende >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-467) Showtypes magic doesn't work
[ https://issues.apache.org/jira/browse/TOREE-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-467. --- Resolution: Fixed Assignee: Luciano Resende Fix Version/s: 0.3.0 > Showtypes magic doesn't work > > > Key: TOREE-467 > URL: https://issues.apache.org/jira/browse/TOREE-467 > Project: TOREE > Issue Type: Bug >Reporter: Maciej Bryński > Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > Attachments: image-2018-02-21-12-02-56-348.png > > > !image-2018-02-21-12-02-56-348.png! > > I'm using Toree version 0.2.0-incubating-rc3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TOREE-480) Deprecate nosparkcontext in favor of spark-context-initialization-mode none
[ https://issues.apache.org/jira/browse/TOREE-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved TOREE-480. --- Resolution: Fixed Fix Version/s: 0.3.0 > Deprecate nosparkcontext in favor of spark-context-initialization-mode none > --- > > Key: TOREE-480 > URL: https://issues.apache.org/jira/browse/TOREE-480 > Project: TOREE > Issue Type: Bug > Components: Kernel > Reporter: Luciano Resende > Assignee: Luciano Resende >Priority: Major > Fix For: 0.3.0 > > > We currently have spark-context-initialization-mode that supports eager and > lazy options. This ticket is to deprecate the nonsparkcontext in favor of > adding a 'none' option to spark-context-initialization-mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[ANNOUNCE] Apache Toree 0.2.0-incubating Released
Apache Toree is a kernel for the Jupyter Notebook platform providing interactive and remote access to Apache Spark. The Apache Toree community is pleased to announce the release of Apache Toree 0.2.0-incubating which provides various bug fixes and the following enhancements. * Support Apache Spark 2.x codebase including Spark 2.2.2 * Enable Toree to run in Yarn cluster mode * Create spark context lazily to avoid long startup times for the kernel * Properly cleanup of temporary files/directories upon kernel shutdown * %AddJAR now supports HDFS file format * %AddDEP now defaults to default configuration * Cell Interrupt now cancel running Spark jobs and works in background processes * Support configurable alternative interrupt signal via --alternate-sigint command line * Interpreters now have the ability to send results other than text/plain For more information about Apache Toree and go download the latest release go to: https://toree.incubator.apache.org/ For more information on how to use Apache Toree please visit our documentation page: https://toree.incubator.apache.org/docs/current/user/quick-start/ -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Draft Podling Report - August 2018
Toree Toree provides applications with a mechanism to interactively and remotely access Apache Spark. Toree has been incubating since 2015-12-02. Three most important issues to address in the move towards graduation: 1.More discussion and engagement on the mailing list as opposed to "gitter" 3.Continue to make releases Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware of? * We have discussed couple times about inactive mentors, and we will most likely start a process to replace those with new more available ones so that we can get our release and graduate process more smoothly. How has the community developed since the last report? * The community has been working on the 0.2.0 release, and a new RC6 is waiting for IPMC approval. How has the project developed since the last report? * Regular cadence of community activity, with some increase on mailing list activity. Having said that, lots of questions and interactions are still happening in a very active fashion on the "gitter chat community" but we were able to get some of those discussions turning into JIRAs on the project. How would you assess the podling's maturity? Please feel free to add your own commentary. [ ] Initial setup [ ] Working towards first release [x] Community building [x] Nearing graduation [ ] Other: Date of last release: * 2017-02-21 When were the last committers or PPMC members elected? * Ryan Blue was added to the PMC on 2017-04-03 -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Created] (TOREE-483) Add %showOutput magic to disable console output
Luciano Resende created TOREE-483: - Summary: Add %showOutput magic to disable console output Key: TOREE-483 URL: https://issues.apache.org/jira/browse/TOREE-483 Project: TOREE Issue Type: Improvement Components: Kernel Reporter: Luciano Resende Assignee: Luciano Resende -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[RESULT][VOTE] Apache Toree 0.2.0 RC6
On Mon, Jul 30, 2018 at 10:39 AM Luciano Resende wrote: > Please vote to approve the release of Apache Toree 0.2.0-incubating > (RC6). > > Tag: v0.2.0-incubating-rc6 (eae2da61828a0c88e2ab64a7ffb9ff7647909a2c) > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc6 > > All distribution packages, including signatures, digests, etc. can be > found at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > -incubating-rc6/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1013 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html > ) > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > dist/dev/incubator/toree/0.2.0-incubating-rc6/toree-pip/toree > -0.2.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at > least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > [ ] -1 Do not release this package because ... > > Vote passed with +1 from: Marius van Niekerk Luciano Resende Ryan Blue I will submit the IPMC vote now. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Created] (TOREE-482) Truncation magic does not work
Luciano Resende created TOREE-482: - Summary: Truncation magic does not work Key: TOREE-482 URL: https://issues.apache.org/jira/browse/TOREE-482 Project: TOREE Issue Type: Bug Components: Kernel Affects Versions: 0.2.0 Reporter: Luciano Resende -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[VOTE] Apache Toree 0.2.0 RC6
Please vote to approve the release of Apache Toree 0.2.0-incubating (RC6). Tag: v0.2.0-incubating-rc6 (eae2da61828a0c88e2ab64a7ffb9ff7647909a2c) https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc6 All distribution packages, including signatures, digests, etc. can be found at: https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0-incubating-rc6/ Staging artifacts can be found at: https://repository.apache.org/content/repositories/orgapachetoree-1013 ## Testing Instructions The fastest way to get up and running is to using Jupyter. 1. Install Jupyter if you haven't already (http://jupyter.org/install.html) 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ dist/dev/incubator/toree/0.2.0-incubating-rc6/toree-pip/toree-0.2.0.tar.gz` followed by `jupyter toree install` - You need to set a valid Apache Spark 2.x home, which can be done via `jupyter toree install --spark_home=/usr/local/spark` - You may need to run with `sudo` for installation permission - For all installation options, run `jupyter toree install --help-all` 4. Run a Jupyter notebook server via `jupyter notebook` - If the notebook portion of Jupyter is not installed but Jupyter is, you can install via `pip install notebook` 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new dev 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the notebook ## Voting Instructions The vote is open for at least 72 hours and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Toree 0.2.0-incubating [ ] -1 Do not release this package because ... -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Comment Edited] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488163#comment-16488163 ] Luciano Resende edited comment on TOREE-474 at 5/23/18 10:59 PM: - Would you be willing to help port the fix from 0.1 to 0.2? We would appreciate the patch. was (Author: luciano resende): Would you be willing to port the fix from 0.1 to 0.2? We would appreciate the patch. > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Priority: Major > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TOREE-474) support for custom resolvers (for %AddDeps)
[ https://issues.apache.org/jira/browse/TOREE-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488163#comment-16488163 ] Luciano Resende commented on TOREE-474: --- Would you be willing to port the fix from 0.1 to 0.2? We would appreciate the patch. > support for custom resolvers (for %AddDeps) > --- > > Key: TOREE-474 > URL: https://issues.apache.org/jira/browse/TOREE-474 > Project: TOREE > Issue Type: Improvement > Components: Kernel >Affects Versions: 0.2.0 >Reporter: Evgeniy Tokarev >Priority: Major > > Jira has old issue like this https://issues.apache.org/jira/browse/TOREE-169 > which was fixed at 0.1.0 but seems like 0.2.x doesn't as this functionality > anymore. > Lot of libs using custom repositories like Bintray therefore seem like is a > big limitation of project > which doesn't allow to use custom repositories (Zeppelin and spark-notebooks > has that functionality) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: jar import issues
Current master, which is the one supporting Spark 2.x is currently compiled with Scala 2.11. Also, if you are using Scala 2.12, you should make sure that both Spark and Toree gets compiled properly with Scala 2.12 as I believe Spark still provide binaries compiled with Scala 2.11. On Fri, May 18, 2018 at 3:39 PM, Igor Fialko <igorfia...@gmail.com> wrote: > Hello community, > > name is Igor and I'm working in my spare time on a trading system. So far I > used jupyter notebook with python kernel for visualisation/playing around > with data. I wanted to go a step forward and reuse my existing scala code, > so I came up with toree as seemingly the only one still maintained jupyter > kernel. > > Therefore I've assembled a jar via sbt and tried to add it in the notebook. > It has failed with an exact same error as here. > > https://stackoverflow.com/questions/46403426/adding- > dependency-to-apache-toree-fails-on-import > > 2018-05-18 23:43:51 WARN MagicManager:111 - Magic AddJar failed to execute > with error: > object Predef does not have a member classOf > scala.reflect.internal.FatalError: object Predef does not have a member > classOf > > > 2018-05-18 23:43:51 ERROR OneForOneStrategy:159 - object Predef does not > have a member classOf > scala.reflect.internal.FatalError: object Predef does not have a member > classOf > > So I've tried to import the jar via toree_opts like > > sudo jupyter toree install --spark_home=/path/bin/spark-2.3.0 > --toree_opts='--jar-dir /path/target/scala-2.12' > > which didn't make any impact unfortunately (from source code I could see > that it should print a message that jar could not be loaded, neither did an > import work). > > I could assume that it's maybe a scala version issue? My project is > compiled against 2.12.3, I couldn't find scala version that toree is using. > > Did somebody face this issue? How could I debug what is wrong? > > Thanks > Best regards, > Igor > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Closed] (TOREE-436) variable assignments generate output in notebook
[ https://issues.apache.org/jira/browse/TOREE-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende closed TOREE-436. - Resolution: Duplicate Fix Version/s: Not Applicable > variable assignments generate output in notebook > > > Key: TOREE-436 > URL: https://issues.apache.org/jira/browse/TOREE-436 > Project: TOREE > Issue Type: Bug >Affects Versions: 0.2.0 > Environment: Ubuntu 16.04 > Spark 2.2 > notebook 5.0.0 >Reporter: Paulo Villegas >Priority: Minor > Fix For: Not Applicable > > Attachments: Toree.png > > > I'm having a somehow strange behaviour with the latest Toree kernel (checked > out from github master branch). Every assignment on a cell generates notebook > output, as it can be seen in the attached figure > !Toree.png|thumbnail! > The output produces the value of the variable (but apparently not at the time > of assignment, but the final value in the cell). I see no means of switching > that off (I've tried to find a %%magic to activate/deactivate it, but found > no one). Is that expected behaviour? If I try an earlier version of the > kernel on the same system, no output is generated (except the one explicitly > generated by printing, or the usual last value in the cell) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Apache Toree 0.2.0 RC5
On Mon, Apr 30, 2018 at 11:04 AM, Ryan Blue <rb...@netflix.com.invalid> wrote: > -1 > > I'm hitting an error > in integration.interpreter.scala.AddExternalJarMagicSpecForIntegration, > which I think was the same error someone on the IPMC list hit in the last > RC. Not sure what's happening yet, but I'll try to take some time to dig > into it soon. > > I'm on Ubuntu 18.04 and I ran `make test`, if you want to try to reproduce > it. > > rb > > -- > Ryan Blue > Software Engineer > Netflix > I was able to get an Ubuntu cloud VM and looks like that the test failure is reproducible in that environment. I will have to investigate, but if you (Ryan) has actually a local Ubuntu env and could help debug/fix this issue, we would appreciate. Please consider this vote CANCELED. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
Draft report posted, was Re: Podling Report Reminder - May 2018
I have posted a draft report to the Incubator Wiki. https://wiki.apache.org/incubator/May2018#preview Please feel free to use this thread for comments/questions. On Fri, Apr 27, 2018 at 8:01 PM, <johndam...@apache.org> wrote: > Dear podling, > > This email was sent by an automated system on behalf of the Apache > Incubator PMC. It is an initial reminder to give you plenty of time to > prepare your quarterly board report. > > The board meeting is scheduled for Wed, 16 May 2018, 10:30 am PDT. > The report for your podling will form a part of the Incubator PMC > report. The Incubator PMC requires your report to be submitted 2 weeks > before the board meeting, to allow sufficient time for review and > submission (Wed, May 02). > > Please submit your report with sufficient time to allow the Incubator > PMC, and subsequently board members to review and digest. Again, the > very latest you should submit your report is 2 weeks prior to the board > meeting. > > Candidate names should not be made public before people are actually > elected, so please do not include the names of potential committers or > PPMC members in your report. > > Thanks, > > The Apache Incubator PMC > > Submitting your Report > > -- > > Your report should contain the following: > > * Your project name > * A brief description of your project, which assumes no knowledge of > the project or necessarily of its field > * A list of the three most important issues to address in the move > towards graduation. > * Any issues that the Incubator PMC or ASF Board might wish/need to be > aware of > * How has the community developed since the last report > * How has the project developed since the last report. > * How does the podling rate their own maturity. > > This should be appended to the Incubator Wiki page at: > > https://wiki.apache.org/incubator/May2018 > > Note: This is manually populated. You may need to wait a little before > this page is created from a template. > > Mentors > --- > > Mentors should review reports for their project(s) and sign them off on > the Incubator wiki page. Signing off reports shows that you are > following the project - projects that are not signed may raise alarms > for the Incubator PMC. > > Incubator PMC > -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
