Re: [VOTE] Apache Toree 0.3.0 RC1
+1 Checked source build, signatures, checksums, and ran tests. On Fri, Oct 26, 2018 at 7:24 AM Marius van Niekerk < marius.v.niek...@gmail.com> wrote: > +1 > > Tested a sample notebook in a binderhub instance running toree. > > On Wed, 24 Oct 2018 at 03:27 Manu Zhang wrote: > > > +1 (non-binding) > > > > > > I downloaded the toree-0.3.0.tar.gz, installed with pip and checked the > > following functionalities with jupyterlab-0.28.2. > > > > 1. Created a SparkSession, read from a HDFS parquet file into Spark > > DataFrame and showed some rows. > > 2. Disabled console output with showOutput magic > > 3. SQL magic with variables in sql > > > > > > Thanks, > > Manu Zhang > > On Oct 18, 2018, 11:16 AM +0800, Luciano Resende , > > wrote: > > > On Tue, Oct 16, 2018 at 9:30 AM Luciano Resende > > wrote: > > > > > > > > Please vote to approve the release of Apache Toree 0.3.0-incubating > > (RC1). > > > > > > > > Tag: v0.3.0-incubating-rc1 (3ff5df33cc459ea214e542a4798fd9ba6e0a2ef8) > > > > > > > > https://github.com/apache/incubator-toree/tree/v0.3.0-incubating-rc1 > > > > > > > > All distribution packages, including signatures, digests, etc. can be > > found at: > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/ > > > > > > > > Staging artifacts can be found at: > > > > > > > > > https://repository.apache.org/content/repositories/orgapachetoree-1014 > > > > > > > > ## Testing Instructions > > > > > > > > The fastest way to get up and running is to using Jupyter. > > > > > > > > 1. Install Jupyter if you haven't already ( > > http://jupyter.org/install.html) > > > > > > > > 2. Install Apache Toree via `pip install > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz` > <https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz> > > < > https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz > > > > > > followed by `jupyter toree install` > > > > > > > > - You need to set a valid Apache Spark 2.x home, which can be done > via > > > > `jupyter toree install --spark_home=/usr/local/spark` > > > > > > > > - You may need to run with `sudo` for installation permission > > > > > > > > - For all installation options, run `jupyter toree install > --help-all` > > > > > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > > > you can install via `pip install notebook` > > > > > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown > > under new > > > > dev > > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` > > in the > > > > notebook > > > > > > > > ## Voting Instructions > > > > > > > > The vote is open for at least 72 hours and passes if a majority of at > > least > > > > 3 +1 PMC votes are cast. > > > > > > > > [ ] +1 Release this package as Apache Toree 0.3.0-incubating > > > > [ ] -1 Do not release this package because ... > > > > > > > > > > > > > Off course, my +1 > > > > > > -- > > > Luciano Resende > > > http://twitter.com/lresende1975 > > > http://lresende.blogspot.com/ > > > -- > regards > Marius van Niekerk > -- Ryan Blue Software Engineer Netflix
Re: Removing Toree specific support for Python and R
+1 After investigating the python support, we concluded that it's better to use a normal ipython kernel. I think that the python Toree kernel would take a lot of work to be on par with what already exists in Python. On Tue, Sep 25, 2018 at 9:04 AM Luciano Resende wrote: > Toree currently has some support for Python and R which are a thin > wrapper around what is provided in Spark. While this enables sharing > the Spark session and SQL Context between Scala and Python code, it > brings a huge gap on functionality when comparing with other Python > kernels like IPython and IRKernel. Adding to that, there is no > community member that is actively enhancing and contributing to these > two areas. > > Based on these, I would like to suggest the following to be done for > Toree 0.3.0 release timeframe: > - Remove Python and R support from Toree > - Document possible suggestions on Toree website (e.g. IPython, > IRKernel or others more active on their respective community) > > Some of the benefits: > - Avoid user confusion and frustration > - Improve kernel startup performance > - Cleaner code to maintain > > Please let me know your thoughts. > > [1] > https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/sqlcontext_sharing.ipynb > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: Apache Toree 0.2.0-incubating release status
Great news! Thank you for making this happen, Luciano. On Tue, Aug 14, 2018 at 3:29 AM Chip Senkbeil wrote: > Who has ownership rights? Corey and Go no? I don't think I have them, but > I'll check later today. > > On Mon, Aug 13, 2018, 11:23 PM Luciano Resende > wrote: > > > The RC6 has been approved by the IPMC and I have started the process of > > publishing the release which usually takes a day or so for replicating to > > the mirrors. After that, I will update the website with the new > artifacts. > > > > I will need permissions to push the artifacts to PyPI. Could someone with > > ownership rights to the repository please add me there as well. > > > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC6
+1 Tests succeed, signature and checksum are good. Thanks for getting this ready, Luciano! On Thu, Aug 9, 2018 at 8:18 AM Luciano Resende wrote: > Have been using RC6 internally for a few days already without any issues. > Of course, my +1 > > On Mon, Jul 30, 2018 at 10:39 AM Luciano Resende > wrote: > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > (RC6). > > > > Tag: v0.2.0-incubating-rc6 (eae2da61828a0c88e2ab64a7ffb9ff7647909a2c) > > > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc6 > > > > All distribution packages, including signatures, digests, etc. can be > > found at: > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > > -incubating-rc6/ > > > > Staging artifacts can be found at: > > > > https://repository.apache.org/content/repositories/orgapachetoree-1013 > > > > ## Testing Instructions > > > > The fastest way to get up and running is to using Jupyter. > > > > 1. Install Jupyter if you haven't already ( > http://jupyter.org/install.html > > ) > > > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > > dist/dev/incubator/toree/0.2.0-incubating-rc6/toree-pip/toree > > -0.2.0.tar.gz` > > followed by `jupyter toree install` > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > `jupyter toree install --spark_home=/usr/local/spark` > > > > - You may need to run with `sudo` for installation permission > > > > - For all installation options, run `jupyter toree install --help-all` > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > you can install via `pip install notebook` > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > > new > > dev > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in > the > > notebook > > > > ## Voting Instructions > > > > The vote is open for at least 72 hours and passes if a majority of at > > least > > 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > > [ ] -1 Do not release this package because ... > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC5
-1 I'm hitting an error in integration.interpreter.scala.AddExternalJarMagicSpecForIntegration, which I think was the same error someone on the IPMC list hit in the last RC. Not sure what's happening yet, but I'll try to take some time to dig into it soon. I'm on Ubuntu 18.04 and I ran `make test`, if you want to try to reproduce it. rb On Sun, Apr 29, 2018 at 7:59 PM, Marius van Niekerk < marius.v.niek...@gmail.com> wrote: > Tested this on binderhub and seems to work fine +1 > > On Wed, 25 Apr 2018 at 15:32 Luciano Resende <luckbr1...@gmail.com> wrote: > > > On Sat, Apr 21, 2018 at 3:58 PM, Luciano Resende <luckbr1...@gmail.com> > > wrote: > > > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > > (RC5). > > > > > > Tag: v0.2.0-incubating-rc5 (1b2d4aac2113bf59bbb4ec1cff75adbe127f221a) > > > > > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc5 > > > > > > All distribution packages, including signatures, digests, etc. can be > > > found at: > > > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > > > -incubating-rc5/ > > > > > > Staging artifacts can be found at: > > > > > > https://repository.apache.org/content/repositories/orgapachetoree-1012 > > > > > > ## Testing Instructions > > > > > > The fastest way to get up and running is to using Jupyter. > > > > > > 1. Install Jupyter if you haven't already ( > > http://jupyter.org/install.html > > > ) > > > > > > 2. Install Apache Toree via `pip install > https://dist.apache.org/repos/ > > > dist/dev/incubator/toree/0.2.0-incubating-rc5/toree-pip/toree > > > -0.2.0.tar.gz` > > > followed by `jupyter toree install` > > > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > > `jupyter toree install --spark_home=/usr/local/spark` > > > > > > - You may need to run with `sudo` for installation permission > > > > > > - For all installation options, run `jupyter toree install --help-all` > > > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > > you can install via `pip install notebook` > > > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown > under > > > new > > > dev > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in > > the > > > notebook > > > > > > ## Voting Instructions > > > > > > The vote is open for at least 72 hours and passes if a majority of at > > > least > > > 3 +1 PMC votes are cast. > > > > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > > > [ ] -1 Do not release this package because ... > > > > > > > > > > Off course, here is my +1. > > > > Other committers and mentors, please vote here. > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > -- > regards > Marius van Niekerk > -- Ryan Blue Software Engineer Netflix
Re: [VOTE] Apache Toree 0.2.0 RC4
+1 (binding) Checked licenses in src tarball, built and tested using `make test`, validated checksums and signature. Thanks for working on this, Luciano! On Sat, Mar 31, 2018 at 9:46 AM, Luciano Resende <luckbr1...@gmail.com> wrote: > My +1. > > Also, notice that the only change between RC3 and RC4 is to remove the > binary test jars from the source code, which should make the review a > little easier for those that already reviewed the previous release > candidate. > > On Fri, Mar 30, 2018 at 10:28 PM, Luciano Resende <luckbr1...@gmail.com> > wrote: > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > (RC4). > > > > Tag: v0.2.0-incubating-rc4 (8849271e28a4ea4e77c57ef8fb3da77b73469799) > > > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc4 > > > > All distribution packages, including signatures, digests, etc. can be > > found at: > > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0 > > -incubating-rc4/ > > > > Staging artifacts can be found at: > > > > https://repository.apache.org/content/repositories/orgapachetoree-1011 > > > > ## Testing Instructions > > > > The fastest way to get up and running is to using Jupyter. > > > > 1. Install Jupyter if you haven't already (http://jupyter.org/install. > html > > ) > > > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > > dist/dev/incubator/toree/0.2.0-incubating-rc4/toree-pip/toree > > -0.2.0.tar.gz` > > followed by `jupyter toree install` > > > > - You need to set a valid Apache Spark 2.x home, which can be done via > > `jupyter toree install --spark_home=/usr/local/spark` > > > > - You may need to run with `sudo` for installation permission > > > > - For all installation options, run `jupyter toree install --help-all` > > > > 4. Run a Jupyter notebook server via `jupyter notebook` > > > > - If the notebook portion of Jupyter is not installed but Jupyter is, > > you can install via `pip install notebook` > > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > > new > > dev > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in > the > > notebook > > > > ## Voting Instructions > > > > The vote is open for at least 72 hours and passes if a majority of at > > least > > 3 +1 PMC votes are cast. > > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > > [ ] -1 Do not release this package because ... > > > > -- > > Luciano Resende > > http://twitter.com/lresende1975 > > http://lresende.blogspot.com/ > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: Podling Report Reminder - February 2018
Thanks for putting that together. It looks good to me and I added my sign-off. On Thu, Feb 1, 2018 at 7:32 PM, Luciano Resende <luckbr1...@gmail.com> wrote: > I have added a draft report at > https://wiki.apache.org/incubator/February2018 > > Please feel free to provide feedback and other additions and I can update > it (or feel free to update it directly on the wiki page) > > On Thu, Feb 1, 2018 at 7:16 PM, <johndam...@apache.org> wrote: > > > Dear podling, > > > > This email was sent by an automated system on behalf of the Apache > > Incubator PMC. It is an initial reminder to give you plenty of time to > > prepare your quarterly board report. > > > > The board meeting is scheduled for Wed, 21 February 2018, 10:30 am PDT. > > The report for your podling will form a part of the Incubator PMC > > report. The Incubator PMC requires your report to be submitted 2 weeks > > before the board meeting, to allow sufficient time for review and > > submission (Wed, February 07). > > > > Please submit your report with sufficient time to allow the Incubator > > PMC, and subsequently board members to review and digest. Again, the > > very latest you should submit your report is 2 weeks prior to the board > > meeting. > > > > Thanks, > > > > The Apache Incubator PMC > > > > Submitting your Report > > > > -- > > > > Your report should contain the following: > > > > * Your project name > > * A brief description of your project, which assumes no knowledge of > > the project or necessarily of its field > > * A list of the three most important issues to address in the move > > towards graduation. > > * Any issues that the Incubator PMC or ASF Board might wish/need to be > > aware of > > * How has the community developed since the last report > > * How has the project developed since the last report. > > * How does the podling rate their own maturity. > > > > This should be appended to the Incubator Wiki page at: > > > > https://wiki.apache.org/incubator/February2018 > > > > Note: This is manually populated. You may need to wait a little before > > this page is created from a template. > > > > Mentors > > --- > > > > Mentors should review reports for their project(s) and sign them off on > > the Incubator wiki page. Signing off reports shows that you are > > following the project - projects that are not signed may raise alarms > > for the Incubator PMC. > > > > Incubator PMC > > > > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
[jira] [Commented] (TOREE-427) Unable to use display() or magics in Toree-PySpark
[
https://issues.apache.org/jira/browse/TOREE-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306482#comment-16306482
]
Ryan Blue commented on TOREE-427:
-
Magics currently get translated into Scala code that gets run in the current
interpreter. We need to move magics to get invoked directly instead of going
through the interpreter. That would also help debugging because you could have
magics that, for example, print logs when the interpreter has crashed. Patches
are welcome for this work, and I'll help review them if anyone wants to pick
this up.
> Unable to use display() or magics in Toree-PySpark
> --
>
> Key: TOREE-427
> URL: https://issues.apache.org/jira/browse/TOREE-427
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
>Affects Versions: 0.2.0
> Environment: Linux
>Reporter: Mark Gritter
>Priority: Minor
>
> I am trying to use PySpark in a Jupyter notebook using PySpark. Plain-text
> interaction is working fine. When I tried to include HTML or graphical
> output, though I get the following:
> display(sc.getConf().getAll())
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Traceback (most recent call last):
> File
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py",
> line 189, in
> eval(compiled_code)
> File "", line 1, in
> File "/usr/local/lib/python2.7/dist-packages/IPython/core/display.py", line
> 300, in display
> format = InteractiveShell.instance().display_formatter.format
> File
> "/usr/local/lib/python2.7/dist-packages/traitlets/config/configurable.py",
> line 412, in instance
> inst = cls(*args, **kwargs)
> File
> "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py",
> line 500, in __init__
> self.init_io()
> File
> "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py",
> line 660, in init_io
> io.stdout = io.IOStream(sys.stdout)
> File "/usr/local/lib/python2.7/dist-packages/IPython/utils/io.py", line 34,
> in __init__
> raise ValueError("fallback required, but not specified")
> ValueError: fallback required, but not specified
> StackTrace:
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> scala.Option.foreach(Option.scala:257)
> org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> py4j.Gateway.invoke(Gateway.java:280)
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> py4j.commands.CallCommand.execute(CallCommand.java:79)
> py4j.GatewayConnection.run(GatewayConnection.java:214)
> java.lang.Thread.run(Thread.java:748)
> I tried using magics to display raw HTML (following the example here:
> https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/magic-tutorial.ipynb)
> and get the following backtrace:
> %%dataframe
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Traceback (most recent call last):
> File
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py",
> line 189, in
> eval(compiled_code)
> File "", line 1, in
> File
> "/tmp/kernel-PySpark-7b121ed7-79a8-4222-bf15-733286d125d2/pyspark_runner.py",
> line 107, in __getattr__
> return self._jvm_kernel.__getattribute__(name)
> AttributeError: 'JavaObject' object has no attribute 'magics'
> StackTrace:
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> org.apache.toree.interpreter.broker.BrokerState$$anonfun$markFailure$1.apply(BrokerState.scala:163)
> scala.Option.foreach(Option.scala:257)
> org.apache.toree.interpreter.broker.BrokerState.markFailure(BrokerState.scala:162)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.Delegat
Re: [VOTE] Apache Toree 0.2.0 RC2
I think the tag and commit hash are incorrect. The tag is v0.2.0-incubating-rc2 and the commit hash is 32bbefa121aafd8713afab81516917234d72d690 See https://github.com/apache/incubator-toree/commit/32bbefa. rb On Sun, Nov 5, 2017 at 10:27 AM, Luciano Resende <luckbr1...@gmail.com> wrote: > Please vote to approve the release of Apache Toree 0.2.0-incubating (RC2). > > Tag: v0.2.0-incubating-rc1 (01cd97e9bad04878a8014016c154a50e2a00f21d) > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc2 > > All distribution packages, including signatures, digests, etc. can be found > at: > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2. > 0-incubating-rc2/ > > Staging artifacts can be found at: > > https://repository.apache.org/content/repositories/orgapachetoree-1008 > > ## Testing Instructions > > The fastest way to get up and running is to using Jupyter. > > 1. Install Jupyter if you haven't already (http://jupyter.org/install.html > ) > > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/ > dist/dev/incubator/toree/0.2.0-incubating-rc2/toree-pip/ > toree-0.2.0.tar.gz` > followed by `jupyter toree install` > > - You need to set a valid Apache Spark 2.x home, which can be done via > `jupyter toree install --spark_home=/usr/local/spark` > > - You may need to run with `sudo` for installation permission > > - For all installation options, run `jupyter toree install --help-all` > > 4. Run a Jupyter notebook server via `jupyter notebook` > > - If the notebook portion of Jupyter is not installed but Jupyter is, > you can install via `pip install notebook` > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under > new > dev > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the > notebook > > ## Voting Instructions > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating > [ ] -1 Do not release this package because ... > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Ryan Blue Software Engineer Netflix
Re: Removing support for Scala 2.10
+1 On Mon, Nov 6, 2017 at 6:04 AM, Marius van Niekerk < marius.v.niek...@gmail.com> wrote: > I don't think that the current support for 2.10 in master even works > reliably. Remove it as it will make the maintenance of this simpler. > > On Sat, 4 Nov 2017 at 18:56 Chip Senkbeil <chip.senkb...@gmail.com> wrote: > > > Go for it. > > > > On Sat, Nov 4, 2017, 3:08 PM Luciano Resende <luckbr1...@gmail.com> > wrote: > > > > > The current master only works with Apache Spark 2.x which is based on > > Scala > > > 2.11 and moving towards Scala 2.12. > > > > > > Anyone opposed on removing Scala 2.10 from master ? > > > > > > -- > > > Luciano Resende > > > http://twitter.com/lresende1975 > > > http://lresende.blogspot.com/ > > > > > > -- > regards > Marius van Niekerk > -- Ryan Blue Software Engineer Netflix
[jira] [Commented] (TOREE-355) java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
[
https://issues.apache.org/jira/browse/TOREE-355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180975#comment-16180975
]
Ryan Blue commented on TOREE-355:
-
Looks like the formatting was messed up. The option is -noverify.
> java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> ---
>
> Key: TOREE-355
> URL: https://issues.apache.org/jira/browse/TOREE-355
> Project: TOREE
> Issue Type: Bug
> Environment: Toree 0.2.0.dev1
> CDH 5.9's embedded spark-2.0.0-beta2
>Reporter: Adrien Lavoillotte
>Assignee: Luciano Resende
> Labels: easyfix
> Fix For: 0.2.0
>
> Attachments: toree shade.png
>
>
> Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327
> for me. Instead, I got this error:
> {code}
> Exception in thread "main" java.lang.IncompatibleClassChangeError: class
> org.clapper.classutil.asm.ASMEmptyVisitor has interface
> org.objectweb.asm.ClassVisitor as super class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
> at
> org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at
> org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
> at scala.collection.immutable.Stream.map(Stream.scala:418)
> at
> org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
> at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
> at
> org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
> at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
> at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
> at scala.util.Try$.apply(Try.scala:192)
> at
> org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
> at
> org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
> at
> org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
> at
> org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
> at
> org.apache.toree
[jira] [Resolved] (TOREE-414) Non-daemon thread pools prevent graceful shutdown
[ https://issues.apache.org/jira/browse/TOREE-414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved TOREE-414. - Resolution: Fixed Assignee: Ryan Blue Fix Version/s: 0.2.0 Merged #122. Thanks to [~senkwich] for reviewing! > Non-daemon thread pools prevent graceful shutdown > - > > Key: TOREE-414 > URL: https://issues.apache.org/jira/browse/TOREE-414 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.1.0 >Reporter: Ryan Blue > Assignee: Ryan Blue > Fix For: 0.2.0 > > > TaskManager and ScheduledTaskManager create threads without using > ThreadFactoryBuilder, which can set the thread name and whether the thread is > a daemon. Not using daemon threads prevents graceful shutdown (TOREE-413). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (TOREE-380) Interpreters should be allowed to send results other than text/plain.
[
https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue resolved TOREE-380.
-
Resolution: Fixed
Fix Version/s: 0.2.0
Merged PR #104. Thanks to [~mariusvniekerk] and [~chipsenkbeil] for reviewing!
> Interpreters should be allowed to send results other than text/plain.
> -
>
> Key: TOREE-380
> URL: https://issues.apache.org/jira/browse/TOREE-380
> Project: TOREE
> Issue Type: Improvement
> Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 0.2.0
>
>
> Jupyter allows kernels to send results using different content types, but
> this isn't allowed by Toree for its interpreters. This prevents custom
> display logic. The basic problem is that {{ExecuteOutput}} is a {{String}}
> and not a {{Map[String, String]}} like {{CellMagicOutput}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (TOREE-418) Improve use of Jupyter's jvm-repr-api
Ryan Blue created TOREE-418: --- Summary: Improve use of Jupyter's jvm-repr-api Key: TOREE-418 URL: https://issues.apache.org/jira/browse/TOREE-418 Project: TOREE Issue Type: Bug Reporter: Ryan Blue TOREE-380 uses the [Jupyter jvm-repr API|https://github.com/jupyter/jvm-repr] to show rich representations of JVM objects. Libraries can register methods to display their classes by MIME type. Currently, the use is limited to the Scala interpreter, but it should probably be used to display any JVM object that interpreters don't handle. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TOREE-417) Flaky test: ExecuteActorRelaySpec
Ryan Blue created TOREE-417:
---
Summary: Flaky test: ExecuteActorRelaySpec
Key: TOREE-417
URL: https://issues.apache.org/jira/browse/TOREE-417
Project: TOREE
Issue Type: Bug
Affects Versions: 0.1.0
Reporter: Ryan Blue
ExecuteActorRelaySpec is flaky when running in gradle. Some {{expectMsgClass}}
calls take more than the default 3s timeout.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (TOREE-416) Flaky test: KernelInputStreamSpec
Ryan Blue created TOREE-416: --- Summary: Flaky test: KernelInputStreamSpec Key: TOREE-416 URL: https://issues.apache.org/jira/browse/TOREE-416 Project: TOREE Issue Type: Bug Affects Versions: 0.1.0 Reporter: Ryan Blue When running tests in gradle, KernelInputStreamSpec is flaky. Calls to the InputStream.read method hang forever. Sleeping for ~100ms at the end of the before block seems to avoid the problem. The likely cause is an actor thread or ZMQ consumer that needs to start up before running tests, or messages get lost. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (TOREE-415) Interrupting tasks should cancel running Spark jobs.
Ryan Blue created TOREE-415: --- Summary: Interrupting tasks should cancel running Spark jobs. Key: TOREE-415 URL: https://issues.apache.org/jira/browse/TOREE-415 Project: TOREE Issue Type: Improvement Components: Kernel Affects Versions: 0.1.0 Reporter: Ryan Blue The most common use of CTRL+C in Jupyter console is to cancel the running Spark stage, but the current implementation only interrupts the monitoring thread. Interrupting a running cell should cancel running Spark jobs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-413) Add support for session timeout and exit
Ryan Blue created TOREE-413: --- Summary: Add support for session timeout and exit Key: TOREE-413 URL: https://issues.apache.org/jira/browse/TOREE-413 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue When using Toree with Jupyter console, the only way to exit is to use CTRL+D to send EOF and then confirm. Toree should include support for "exit", "quit", and ":q" in any interpreter. In addition, because Toree runs in a separate process and can start up slowly, the default Jupyter console timeout (60s) will sometimes cause the console to exit but leave a Toree kernel running. Similarly, losing a SSH session running Jupyter console with Toree leaves the kernel running. Toree should have a 24h idle timeout to clean up these zombie kernels. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-412) AddDeps should default to the default configuration, not compile
Ryan Blue created TOREE-412: --- Summary: AddDeps should default to the default configuration, not compile Key: TOREE-412 URL: https://issues.apache.org/jira/browse/TOREE-412 Project: TOREE Issue Type: Improvement Components: Kernel Reporter: Ryan Blue The AddDeps magic (using Coursier) currently requests the compile configuration instead of the default configuration. The dependencies shouldn't be those needed to compile, but should be the runtime dependencies, which are normally pulled in by the default configuration. We ran into issues with this because runtime dependencies are not included. In addition, I'd like to add --ivy-configuration and --classifier options to give users control over what gets used. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-407) Improve Branding on Site
[ https://issues.apache.org/jira/browse/TOREE-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045660#comment-16045660 ] Ryan Blue commented on TOREE-407: - +1 [~luciano resende], are there docs on how to edit the site? > Improve Branding on Site > > > Key: TOREE-407 > URL: https://issues.apache.org/jira/browse/TOREE-407 > Project: TOREE > Issue Type: Improvement >Reporter: Kyle Kelley > > I want to recommend Toree to others as a Scala kernel. The messaging on > https://toree.incubator.apache.org/ is all about "remote spark" which muddies > what it actually does, which is provide kernels that are connected to Spark. > Toree, standalone, doesn't do anything without Jupyter. > Here's what I wish it read: > ``` > Apache Toree > Spark connected kernels for Jupyter projects - Scala, Python, and R > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045657#comment-16045657
]
Ryan Blue commented on TOREE-411:
-
The solution to this problem that we use is to set the catalog implementation
to "hive", as suggested by Xin, in our Spark defaults.
Also, TOREE-390 was just committed, which changes the Spark session so that it
is lazily created when you reference {{spark}} or picks up an existing context.
That means you can either use the default session or create your own and call
{{enableHiveSupport}}, among other configuration options. This should address
the problem without needing to set the catalog implementation outside of Toree.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
> Fix For: 0.2.0
>
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[
https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15990490#comment-15990490
]
Ryan Blue commented on TOREE-390:
-
[~luciano resende], we aren't having the weird issues with requests any more,
since the commit you linked to and
[8f1775d|https://github.com/apache/incubator-toree/commit/8f1775d9dcdc56dd4f3a7c44abf0c36ae0417444].
The problem this fixes is the initial wait time is both startup for Toree and
Spark. This separates the two waits, making it clear which one is due to Spark
starting up and getting resources from YARN (in our deployment). This is better
for the user experience because people don't think Toree is slow, they
correctly attribute it to Spark. The secondary benefit is that users can
customize their Spark session before it starts by creating one in the first
cell, and Toree works just fine with that.
> Lazily start Spark sessions
> ---
>
> Key: TOREE-390
> URL: https://issues.apache.org/jira/browse/TOREE-390
> Project: TOREE
> Issue Type: Improvement
> Reporter: Ryan Blue
>
> In our deployment, more than half of the startup time for a Toree notebook is
> taken by starting a Spark session and waiting for containers. Lazily starting
> Spark sessions helps the notebook environment feel faster, even if the user
> is waiting on Spark to start up because the time waiting for Spark to start
> is clearly Spark, not Toree, and is initiated by the user.
> Also, lazily starting a Spark session allows users to change settings that
> can't be changed in a Spark context. It also enables the same startup code
> that would be used in a spark-submit application:
> {code:lang=java}
> SparkSession.builder
> .config(...)
> .setAppName(...)
> .getOrCreate()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
SQL interpreter
I've been working on improvements to the SQL interpreter, but I'm not quite sure why it works the way it currently does. Right now, the interpreter is associated with a SQL service that is a BrokerService. BrokerService says that it is a high-level interface between the JVM and another process, like R or Python, so it isn't clear why SQL uses it. The version I have working is considerably simpler and uses the lazy `kernel.sparkSession` accessor. Is there a reason why the current one doesn't use this approach? rb -- Ryan Blue Software Engineer Netflix
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936567#comment-15936567 ] Ryan Blue commented on TOREE-395: - I don't think there is a reason to keep it. When using Toree with Jupyter notebooks, the user doesn't see the second copy of results. That opens up the possibility to print sensitive data as was already pointed out, but, more importantly, it pollutes the notebook server log with results that aren't useful when you go to look at it. Is there a use case for notebooks that you can think of that makes this a useful feature? The other environment to think about is Jupyter console. When you use Toree with the console, everything gets printed as cell output (which my PR addresses) and is printed to stdout. That it produces double output is the main reason why we disabled it. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935254#comment-15935254 ] Ryan Blue commented on TOREE-395: - Yeah, please do. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala
[ https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934825#comment-15934825 ] Ryan Blue commented on TOREE-395: - Thanks for continuing the discussion on this, Kevin. A few weeks ago, I posted a [WIP pull request|https://github.com/apache/incubator-toree/pull/104] with what we're currently using that you might be interested in. That addresses a few issues, including this one. The approach we took was to get rid of multiOutputStream and just use lastResultOut. Normal stderr and stdout printing happens just fine, so I think that the only output that gets printed the writer passed to IMain is the summary of what ran (e.g. import scala...). Your idea to conditionally print the output to the logger should work fine, but is it really necessary for people to see this output in the logs? I see no reason to keep it this way. > Provide a way to disable automatic printing of results in Toree Scala > - > > Key: TOREE-395 > URL: https://issues.apache.org/jira/browse/TOREE-395 > Project: TOREE > Issue Type: Improvement >Affects Versions: 0.2.0 >Reporter: Kun Liu > > Scala REPL supports a flag, ":silent" to enable/disable automatic printing of > results (those messages begin with "res"). This is also supported in Spark > shell. But for Toree Scala kernel, this flag is not supported. Thus the > result of a cell would be always printed. > But if there is any logging mechanism, the results would be recorded in a log > file, while this may not be desirable. For instance, a user may not want any > sensitive data logged when running "dataRDD.take(5)" in a cell. > Also found the possible source codes for this: > https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala > There are three methods with silent: Boolean = false as parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-390) Lazily start Spark sessions
Ryan Blue created TOREE-390:
---
Summary: Lazily start Spark sessions
Key: TOREE-390
URL: https://issues.apache.org/jira/browse/TOREE-390
Project: TOREE
Issue Type: Improvement
Reporter: Ryan Blue
In our deployment, more than half of the startup time for a Toree notebook is
taken by starting a Spark session and waiting for containers. Lazily starting
Spark sessions helps the notebook environment feel faster, even if the user is
waiting on Spark to start up because the time waiting for Spark to start is
clearly Spark, not Toree, and is initiated by the user.
Also, lazily starting a Spark session allows users to change settings that
can't be changed in a Spark context. It also enables the same startup code that
would be used in a spark-submit application:
{code:lang=java}
SparkSession.builder
.config(...)
.setAppName(...)
.getOrCreate()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TOREE-389) Scala interpreter hangs if Spark is referenced before it is bound
Ryan Blue created TOREE-389:
---
Summary: Scala interpreter hangs if Spark is referenced before it
is bound
Key: TOREE-389
URL: https://issues.apache.org/jira/browse/TOREE-389
Project: TOREE
Issue Type: Bug
Reporter: Ryan Blue
Some users run cells in a notebook before the kernel is finished starting. The
messages are queued and run after the KernelMessageRelay is enabled, but any
messages that reference {{spark}} or bound variables cause IMain to throw an
exception. That exception causes the interpreter to hang.
It seems odd that a missing variable reference causes IMain to throw an
exception, so there may be a threading issue, a race between binding {{spark}}
and the cell that references it.
The problem can be avoided by moving {{postInit}} tasks into init.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TOREE-388) Add optional gradle build script
Ryan Blue created TOREE-388: --- Summary: Add optional gradle build script Key: TOREE-388 URL: https://issues.apache.org/jira/browse/TOREE-388 Project: TOREE Issue Type: New Feature Reporter: Ryan Blue We've been using a gradle build instead of sbt. I'll open a PR with it, in case anyone else is interested. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Interpreter outputs
Hi everyone, I've been putting together a proof-of-concept Toree deployment for our Spark users to test out. One of the things I needed to change was how interpreter outputs are displayed, allowing interpreters to show HTML or other representations of output objects. For anyone else interested, I've put up a PR with the changes for discussion: https://github.com/apache/incubator-toree/pull/104 There are also a few other PRs that fix tab completion, code completeness checks, syntax highlighting, and other usability issues. It would be great to get feedback from the community. Thanks! rb -- Ryan Blue Software Engineer Netflix
[jira] [Updated] (TOREE-380) Interpreters should be allowed to send results other than text/plain.
[
https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue updated TOREE-380:
Summary: Interpreters should be allowed to send results other than
text/plain. (was: Interpreters should be allowed to send non-text results.)
> Interpreters should be allowed to send results other than text/plain.
> -
>
> Key: TOREE-380
> URL: https://issues.apache.org/jira/browse/TOREE-380
> Project: TOREE
> Issue Type: Improvement
> Reporter: Ryan Blue
>
> Jupyter allows kernels to send results using different content types, but
> this isn't allowed by Toree for its interpreters. This prevents custom
> display logic. The basic problem is that {{ExecuteOutput}} is a {{String}}
> and not a {{Map[String, String]}} like {{CellMagicOutput}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (TOREE-380) Interpreters should be allowed to send non-text results.
[
https://issues.apache.org/jira/browse/TOREE-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue updated TOREE-380:
Description: Jupyter allows kernels to send results using different content
types, but this isn't allowed by Toree for its interpreters. This prevents
custom display logic. The basic problem is that {{ExecuteOutput}} is a
{{String}} and not a {{Map[String, String]}} like {{CellMagicOutput}}. (was:
Jupyter allows kernels to send results using different content types, but this
isn't allowed by Toree for its interpreters. This prevents custom display
logic. The basic problem is that `ExecuteOutput` is a `String` and not a
`Map[String, String]` like `CellMagicOutput`.)
> Interpreters should be allowed to send non-text results.
>
>
> Key: TOREE-380
> URL: https://issues.apache.org/jira/browse/TOREE-380
> Project: TOREE
> Issue Type: Improvement
> Reporter: Ryan Blue
>
> Jupyter allows kernels to send results using different content types, but
> this isn't allowed by Toree for its interpreters. This prevents custom
> display logic. The basic problem is that {{ExecuteOutput}} is a {{String}}
> and not a {{Map[String, String]}} like {{CellMagicOutput}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TOREE-380) Interpreters should be allowed to send non-text results.
Ryan Blue created TOREE-380: --- Summary: Interpreters should be allowed to send non-text results. Key: TOREE-380 URL: https://issues.apache.org/jira/browse/TOREE-380 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue Jupyter allows kernels to send results using different content types, but this isn't allowed by Toree for its interpreters. This prevents custom display logic. The basic problem is that `ExecuteOutput` is a `String` and not a `Map[String, String]` like `CellMagicOutput`. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-378) Usability improvements in jupyter console
Ryan Blue created TOREE-378: --- Summary: Usability improvements in jupyter console Key: TOREE-378 URL: https://issues.apache.org/jira/browse/TOREE-378 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue Using Toree with jupyter console has some bugs and unexpected behavior. These include: * IsCompleteHandler sends "incomplete" for indentation * A syntactically complete multi-line block will be run immediately, unlike python which waits for a blank line * Indentation is based on the previous line, but is not anchored -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-377) When magic fails, the error is swallowed
Ryan Blue created TOREE-377: --- Summary: When magic fails, the error is swallowed Key: TOREE-377 URL: https://issues.apache.org/jira/browse/TOREE-377 Project: TOREE Issue Type: Improvement Reporter: Ryan Blue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TOREE-376) Scala interpreter should make Spark SQL implicits available
Ryan Blue created TOREE-376:
---
Summary: Scala interpreter should make Spark SQL implicits
available
Key: TOREE-376
URL: https://issues.apache.org/jira/browse/TOREE-376
Project: TOREE
Issue Type: Improvement
Reporter: Ryan Blue
The scala interpreter doesn't define Spark SQL functions or implicits that are
commonly used and available by default in Spark's shell.
Relevant test cases:
{code}
scala> val c = $"test"
c: org.apache.spark.sql.ColumnName = test
scala> val df = Seq((1, "a"), (2, "b")).toDF("id", "data")
df: org.apache.spark.sql.DataFrame = [id: int, data: string]
scala> df.withColumn("m", map(lit("id"), $"id")).printSchema
root
|-- id: integer (nullable = false)
|-- data: string (nullable = true)
|-- m: map (nullable = false)
||-- key: string
||-- value: integer (valueContainsNull = false)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
