Re: [VOTE] Apache Toree 0.4.0 RC1

[Chip Senkbeil]

+1, LGTM. Did we want to update the release notes with the changes? I
was using 
https://github.com/apache/incubator-toree/compare/v0.3.0-incubating...v0.4.0-incubating-rc1
to get a summary of what has changed since 0.3.0's release.

On Thu, Jul 16, 2020 at 2:26 PM Luciano Resende  wrote:
>
> Please vote to approve the release of Apache Toree 0.4.0-incubating (RC1).
>
> Tag: v0.4.0-incubating-rc1 (b2374e28ce66ae28079703d725b7c0db4d4a)
>
> https://github.com/apache/incubator-toree/tree/v0.4.0-incubating-rc1
>
> All distribution packages, including signatures, digests, etc. can be found 
> at:
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachetoree-1016
>
> ## Testing Instructions
>
> The fastest way to get up and running is to using Jupyter.
>
> 1. Install Jupyter if you haven't already (http://jupyter.org/install.html)
>
> 2. Install Apache Toree via `pip install
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz`
> followed by `jupyter toree install`
>
> - You need to set a valid Apache Spark 2.x home, which can be done via
> `jupyter toree install --spark_home=/usr/local/spark`
>
> - You may need to run with `sudo` for installation permission
>
> - For all installation options, run `jupyter toree install --help-all`
>
> 4. Run a Jupyter notebook server via `jupyter notebook`
>
> - If the notebook portion of Jupyter is not installed but Jupyter is,
> you can install via `pip install notebook`
>
> 5. Create a notebook using "Apache Toree - Scala" from the dropdown under new
> dev
> 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the
> notebook
>
> ## Voting Instructions
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.4.0-incubating
> [ ] -1 Do not release this package because ...
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/

Re: [jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups

[Chip Senkbeil]

We only ever exposed the thread killing functionality through capturing the
interrupt signal on the kernel process. This was to support the stop button
in Jupyter notebooks.

I can't remember if that code still exists as it was a bit flakey, relying
on a Sun API to capture the signal and respond.

If we remove the support in security manager, we should make sure that the
signal capture code is also removed. If we add a flag to disable support,
we'd want to update the signal capture code to not register since it would
serve no purpose.

On Sun, Oct 20, 2019, 9:50 AM Kevin Bates (Jira)  wrote:

>
> [
> https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955516#comment-16955516
> ]
>
> Kevin Bates commented on TOREE-421:
> ---
>
> bq. enable different thread groups properly
>
> I don't think anything has to happen to enable proper thread group
> creation other than NOT preventing its creation via the SecurityManager.
>
> Although I'd prefer to not have to support an additional CLI option, I'm
> wondering if a boolean option such as {{--prevent-new-thread-groups}} that
> defaults to {{false}} would be beneficial.  This would open up thread group
> creation by default, but any applications that require their prevention
> could then add the CLI option.  I realize this could be considered a
> "regression", but I suspect most won't be needing this, and applications
> that require its prevention have probably already been "trained" to not
> attempt thread group creation in the first place (due to its previous
> prevention).
>
> If others feel we don't need a CLI option, we can simply remove this
> portion of the SecurityManager and move on.
>
> Thoughts?
>
> > KernelSecurityManager doesn't allow users to create their own thread
> groups
> >
> ---
> >
> > Key: TOREE-421
> > URL: https://issues.apache.org/jira/browse/TOREE-421
> > Project: TOREE
> >  Issue Type: Bug
> >Reporter: Piyush Narang
> >Priority: Major
> >
> > I'm trying to run a Spark Scala job using Toree and I'm running into
> some issues as the code in our job calls into one of our libraries which
> tries to create threads in its own ThreadGroup:
> https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> > This seems to cause this check in Toree's KernelSecurityManager to trip:
> https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> > Stack looks like:
> > {code}
> > Name: java.lang.SecurityException
> > Message: Not allowed to modify ThreadGroups!
> > StackTrace:   at
> org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> >   at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> >   at java.lang.Thread.init(Thread.java:394)
> >   at java.lang.Thread.init(Thread.java:349)
> >   at java.lang.Thread.(Thread.java:599)
> >   at
> com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> > ...
> > {code}
> > Here's a simple repro:
> > {code}
> > println(Thread.currentThread().getThreadGroup) // default thread group
> > val group: ThreadGroup = new
> ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> > val hello = new Thread(group, new Runnable {
> > def run() {
> > println("hello world")
> >   }
> > })
> > println(hello.getThreadGroup)
> > hello.start
> > {code}
> > Any suggestions for working around this?
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

Re: [jira] [Commented] (TOREE-421) KernelSecurityManager doesn't allow users to create their own thread groups

[Chip Senkbeil]

This was an old implementation geared towards allowing us to manage all 
launched threads so we could interrupt/kill them on request.

I'm not sure how useful this feature has been to folks, so my thought is that 
we either remove it entirely or make it a feature that can be enabled/disabled 
through a CLI option.

Luciano, do you have any thoughts about which direction you'd like to go here?

On Tue, Oct 15, 2019 at 02:06:00PM +, Pablo J. Villacorta (Jira) wrote:
> 
> [ 
> https://issues.apache.org/jira/browse/TOREE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951949#comment-16951949
>  ] 
> 
> Pablo J. Villacorta commented on TOREE-421:
> ---
> 
> Any updates on this? I am having the same issue.
> 
>  
> 
> Did you manage to find a workaround?
> 
> > KernelSecurityManager doesn't allow users to create their own thread groups
> > ---
> >
> > Key: TOREE-421
> > URL: https://issues.apache.org/jira/browse/TOREE-421
> > Project: TOREE
> >  Issue Type: Bug
> >Reporter: Piyush Narang
> >Priority: Major
> >
> > I'm trying to run a Spark Scala job using Toree and I'm running into some 
> > issues as the code in our job calls into one of our libraries which tries 
> > to create threads in its own ThreadGroup: 
> > https://github.com/twitter/util/blob/develop/util-core/src/main/scala/com/twitter/concurrent/NamedPoolThreadFactory.scala#L28
> > This seems to cause this check in Toree's KernelSecurityManager to trip: 
> > https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/security/KernelSecurityManager.scala#L121
> > Stack looks like:
> > {code}
> > Name: java.lang.SecurityException
> > Message: Not allowed to modify ThreadGroups!
> > StackTrace:   at 
> > org.apache.toree.security.KernelSecurityManager.checkAccess(KernelSecurityManager.scala:114)
> >   at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
> >   at java.lang.Thread.init(Thread.java:394)
> >   at java.lang.Thread.init(Thread.java:349)
> >   at java.lang.Thread.(Thread.java:599)
> >   at 
> > com.twitter.concurrent.NamedPoolThreadFactory.newThread(NamedPoolThreadFactory.scala:32)
> > ...
> > {code}
> > Here's a simple repro:
> > {code}
> > println(Thread.currentThread().getThreadGroup) // default thread group
> > val group: ThreadGroup = new 
> > ThreadGroup(Thread.currentThread().getThreadGroup(), "name")
> > val hello = new Thread(group, new Runnable {
> > def run() {
> > println("hello world")
> >   }
> > })
> > println(hello.getThreadGroup)
> > hello.start
> > {code}
> > Any suggestions for working around this? 
> 
> 
> 
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)


signature.asc
Description: PGP signature

Re: Removing Toree specific support for Python and R

[Chip Senkbeil]

They served as experiments in a time where the functionality may have been
useful, but that time has long since passed.

+1

On Tue, Sep 25, 2018, 9:19 PM Gino Bustelo  wrote:

> +1
>
> Spark’s support to share temp tables across sessions voids any need for
> this multi-Lang support.
>
> Gino B.
>
> > On Sep 25, 2018, at 2:20 PM, Corey Stubbs  wrote:
> >
> > +1
> >
> > On Tue, Sep 25, 2018, 13:10 Marius van Niekerk <
> marius.v.niek...@gmail.com>
> > wrote:
> >
> >> +1
> >>
> >> This would simplify usage, and remove a large class of python questions
> and
> >> confusion on our mailing list.  Several users assume that our R and
> Python
> >> featureset has parity with the primary kernels.
> >>
> >>> On Tue, 25 Sep 2018 at 12:46 kbat...@gmail.com 
> wrote:
> >>>
> >>> +1
> >>> The benefits that Luciano points out far outweigh the, probably
> >>> seldom-used, context sharing capabilities with limited features, IMHO.
> >>> This will eliminate an entire class of issues and questions while
> >> allowing
> >>> contributors to narrow their focus only on scala functionality and
> >>> improvements.
> >>>
>  On 2018/09/25 16:04:33, Luciano Resende  wrote:
>  Toree currently has some support for Python and R which are a thin
>  wrapper around what is provided in Spark. While this enables sharing
>  the Spark session and SQL Context between Scala and Python code, it
>  brings a huge gap on functionality when comparing with other Python
>  kernels like IPython and IRKernel. Adding to that, there is no
>  community member that is actively enhancing and contributing to these
>  two areas.
> 
>  Based on these, I would like to suggest the following to be done for
>  Toree 0.3.0 release timeframe:
>  - Remove Python and R support from Toree
>  - Document possible suggestions on Toree website (e.g. IPython,
>  IRKernel or others more active on their respective community)
> 
>  Some of the benefits:
>  - Avoid user confusion and frustration
>  - Improve kernel startup performance
>  - Cleaner code to maintain
> 
>  Please let me know your thoughts.
> 
>  [1]
> >>>
> >>
> https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/sqlcontext_sharing.ipynb
> 
>  --
>  Luciano Resende
>  http://twitter.com/lresende1975
>  http://lresende.blogspot.com/
> 
> >>>
> >> --
> >> regards
> >> Marius van Niekerk
> >>
>

Re: Apache Toree 0.2.0-incubating release status

[Chip Senkbeil]

You should be added to it. There's only a 0.1.0 release in there presently,
but I believe that is the correct PyPi project.

On Tue, Aug 14, 2018 at 12:15 PM Luciano Resende 
wrote:

> My PyPi username is lresende and I believe it’s associated with my Apache
> email.
>
> On Tue, Aug 14, 2018 at 08:54 Chip Senkbeil 
> wrote:
>
>> Oh, I think I do have access to PyPi. Luciano, what email address do you
>> want associated? Your gmail one? Your apache one?
>>
>> On Tue, Aug 14, 2018 at 10:21 AM Ryan Blue 
>> wrote:
>>
>>> Great news! Thank you for making this happen, Luciano.
>>>
>>> On Tue, Aug 14, 2018 at 3:29 AM Chip Senkbeil 
>>> wrote:
>>>
>>> > Who has ownership rights? Corey and Go no? I don't think I have them,
>>> but
>>> > I'll check later today.
>>> >
>>> > On Mon, Aug 13, 2018, 11:23 PM Luciano Resende 
>>> > wrote:
>>> >
>>> > > The RC6 has been approved by the IPMC and I have started the process
>>> of
>>> > > publishing the release which usually takes a day or so for
>>> replicating to
>>> > > the mirrors. After that, I will update the website with the new
>>> > artifacts.
>>> > >
>>> > > I will need permissions to push the artifacts to PyPI. Could someone
>>> with
>>> > > ownership rights to the repository please add me there as well.
>>> > >
>>> > >
>>> > > --
>>> > > Luciano Resende
>>> > > http://twitter.com/lresende1975
>>> > > http://lresende.blogspot.com/
>>> > >
>>> >
>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Netflix
>>>
>> --
> Sent from my Mobile device
>

Re: Apache Toree 0.2.0-incubating release status

[Chip Senkbeil]

Oh, I think I do have access to PyPi. Luciano, what email address do you
want associated? Your gmail one? Your apache one?

On Tue, Aug 14, 2018 at 10:21 AM Ryan Blue 
wrote:

> Great news! Thank you for making this happen, Luciano.
>
> On Tue, Aug 14, 2018 at 3:29 AM Chip Senkbeil 
> wrote:
>
> > Who has ownership rights? Corey and Go no? I don't think I have them, but
> > I'll check later today.
> >
> > On Mon, Aug 13, 2018, 11:23 PM Luciano Resende 
> > wrote:
> >
> > > The RC6 has been approved by the IPMC and I have started the process of
> > > publishing the release which usually takes a day or so for replicating
> to
> > > the mirrors. After that, I will update the website with the new
> > artifacts.
> > >
> > > I will need permissions to push the artifacts to PyPI. Could someone
> with
> > > ownership rights to the repository please add me there as well.
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Apache Toree 0.2.0-incubating release status

[Chip Senkbeil]

Who has ownership rights? Corey and Go no? I don't think I have them, but
I'll check later today.

On Mon, Aug 13, 2018, 11:23 PM Luciano Resende  wrote:

> The RC6 has been approved by the IPMC and I have started the process of
> publishing the release which usually takes a day or so for replicating to
> the mirrors. After that, I will update the website with the new artifacts.
>
> I will need permissions to push the artifacts to PyPI. Could someone with
> ownership rights to the repository please add me there as well.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [VOTE] Apache Toree 0.2.0 RC3

[Chip Senkbeil]

I just ran through the usual basic stuff and it looked fine. +1

On Thu, Jan 18, 2018 at 11:03 AM Luciano Resende 
wrote:

> Off course, my +1
>
> Any others?
>
> On Sun, Jan 14, 2018 at 6:30 PM, Luciano Resende 
> wrote:
>
> > Please vote to approve the release of Apache Toree 0.2.0-incubating
> >  (RC3).
> >
> > Tag: v0.2.0-incubating-rc3 (32bbefa121aafd8713afab81516917234d72d690)
> >
> > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc3
> >
> > All distribution packages, including signatures, digests, etc. can be
> > found at:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.
> > 0-incubating-rc3/
> >
> > Staging artifacts can be found at:
> >
> > https://repository.apache.org/content/repositories/orgapachetoree-1010
> >
> > ## Testing Instructions
> >
> > The fastest way to get up and running is to using Jupyter.
> >
> > 1. Install Jupyter if you haven't already (
> http://jupyter.org/install.html
> > )
> >
> > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/
> > dist/dev/incubator/toree/0.2.0-incubating-rc3/toree-
> > pip/toree-0.2.0.tar.gz`
> > followed by `jupyter toree install`
> >
> > - You need to set a valid Apache Spark 2.x home, which can be done via
> > `jupyter toree install --spark_home=/usr/local/spark`
> >
> > - You may need to run with `sudo` for installation permission
> >
> > - For all installation options, run `jupyter toree install --help-all`
> >
> > 4. Run a Jupyter notebook server via `jupyter notebook`
> >
> > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > you can install via `pip install notebook`
> >
> > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> > new
> > dev
> > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> the
> > notebook
> >
> > ## Voting Instructions
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Toree 0.2.0-incubating
> > [ ] -1 Do not release this package because ...
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Speeding up Toree Startup

[Chip Senkbeil]

At one point, we were going to make Toree highly pluggable using our
then-new plugin framework. Things like an interpreter and a spark context
would be initialized via plugins and anything needing them would be plugins
dependent on those objects. The scala interpreter wasn't necessary for the
spark context or even the kernel object to be instantiated. It just needed
to be created early for scala 2.10 because the repl class server was
created inside it and was needed when creating the spark context.

You could break up initialization and make a lot of pieces optional. Would
also make it easier to do other work like running interpreters in a
separate process that communicate with a remote context (environment?)
easier for a more robust system.

On Thu, Jan 4, 2018, 4:10 PM Corey Stubbs  wrote:

> When we originally started we chose to use Akka because we thought there
> might be the need to do remote interpretation of the code. If I remember
> correctly from some earlier performance evaluations, the Akka actor system
> initialization did take some time.
>
> We might be able to save some startup time and potentially reduce code
> complexity by removing the actor system since there really is no need for
> it. This would be a major refactor though. I would think we would want to
> have a more thorough discussion about this before we consider it.
>
> Kind Regards,
> Corey Stubbs
>
> On Thu, Jan 4, 2018 at 3:26 PM Luciano Resende 
> wrote:
>
> > On Thu, Jan 4, 2018 at 1:09 PM, Bradley Kaiser  >
> > wrote:
> >
> > > Hey Guys,
> > >
> > > I've been looking at ways to speed up Toree's startup and came up with
> a
> > > quick proof of concept that took the startup time from around 6.5
> seconds
> > > to 5.5 seconds on my laptop. I'd like to check committer interest in
> this
> > > before I go any further. This change will require rearranging the
> > > KernelBootstrap class.
> > >
> > >
> > What are you considering for the startup time ? To me, measuring it via
> > yourkit as well, the Main.scala was consuming 13 secs.
> >
> >
> >
> > > My proof of concept is simple, I saw that the slowest part of startup
> was
> > > starting the Scala interpreter, so I ripped that out, moved it to its
> own
> > > thread, and started it very early in the bootstrap process. This way
> > while
> > > it runs the rest of the kernel bootstrap can continue. I did this in a
> > > hacky way just to see if it would work. It would need to be implemented
> > for
> > > real.
> > >
> > > I tested the startup time by setting up a test harness that just runs
> the
> > > kernel bootstrap and times it. I used YourKit to profile the
> application
> > > and see what was taking the most time.
> > >
> > > Here are the results from my tests. Time is in seconds.
> > >
> > > run originalchanged
> > > 1   6.876   5.2
> > > 2   6.445   5.8
> > > 3   6.527   5.7
> > > 4   6.676   5.4
> > > avg 6.631   5.525
> > >
> >
> > Interesting, as my measurings was telling me that some big portions of
> the
> > time was being spent on performing initialization tasks which required
> the
> > scala interpreter to be ready. Consolidating the multiple calls with some
> > other minor tweaks got about 2 secs of performance improvement on the
> > kernel initialization.
> >
> > Another area of possible improvement might be configuring a flag to
> disable
> > auto-loading of all different plugins, which would save us another 1 sec
> > (e.g. not loading other interpreter plugins which are not used when toree
> > is mainly used as a scala kernel)
> >
> >
> > >
> > > If you guys think this is a reasonable approach let me know and i will
> > > make a ticket and work on implementing it for real. I think this
> speedup
> > > will work nicely with the changes proposed in TOREE-444 also.
> > >
> > >
> > I have attached my initial findings at
> > https://issues.apache.org/jira/browse/TOREE-444
> >
> >
> > Others, any possible additions here ?
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>

Re: Removing support for Scala 2.10

[Chip Senkbeil]

Go for it.

On Sat, Nov 4, 2017, 3:08 PM Luciano Resende  wrote:

> The current master only works with Apache Spark 2.x which is based on Scala
> 2.11 and moving towards Scala 2.12.
>
> Anyone opposed on removing Scala 2.10 from master ?
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Uploading PIP for 0.1.0 release

[Chip Senkbeil]

I just changed everything to use toree again. >_>

https://pypi.python.org/pypi?:action=display&name=toree&version=0.1.0

I can delete this and undo all of it so we can use apache-toree if that's
what you'd rather do.

On Mon, Apr 17, 2017 at 2:24 PM Luciano Resende 
wrote:

> Just to clarify, we are going to upload:
>
> https://dist.apache.org/repos/dist/release/incubator/toree/0.1.0-incubating/toree-pip/
>
>
> I don't have strong opinions for the name, "apache-toree" is even better
> than just toree... but we will need to update docs and other stuff, right ?
>
> On Mon, Apr 17, 2017 at 11:57 AM, Chip Senkbeil 
> wrote:
>
>> It looks like all of our makefile work was updated awhile ago to use
>> apache-toree. This must have been from an earlier conversation. I can
>> update our release artifacts for pip to use toree, but we'll have to go
>> through the makefile and update it to produce toree in the future if that's
>> the route we want to go.
>>
>> On Mon, Apr 17, 2017 at 1:27 PM Luciano Resende 
>> wrote:
>>
>>> I looked into some other Apache projects, I believe we could follow the
>>> same pattern and use "toree". but describe it as "Apache Toree
>>> (incubating)". in the description field.
>>>
>>> On Mon, Apr 17, 2017 at 9:50 AM, Chip Senkbeil 
>>> wrote:
>>>
>>>> What is the exact name that you want for the release?
>>>>
>>>> Previously, we were using just "toree", but do you want "apache-toree"
>>>> or "apache-toree-incubating"?
>>>>
>>>> On Mon, Apr 17, 2017 at 10:47 AM Luciano Resende 
>>>> wrote:
>>>>
>>>>> On Mon, Apr 17, 2017 at 8:40 AM, Chip Senkbeil <
>>>>> chip.senkb...@gmail.com> wrote:
>>>>>
>>>>>> I've gotten mixed messages regarding where the pip artifacts should
>>>>>> go. Originally, we were going to publish them on
>>>>>> https://pypi.python.org/pypi (which we were previously doing for dev
>>>>>> snapshots), but there has been some issues with hosting externally from
>>>>>> Apache? Is that still the case? Can we host official releases on Pypi?
>>>>>>
>>>>>
>>>>> Official releases are ok to go there.
>>>>>
>>>>>
>>>>>
>>>>>> If not, we can just indicate that they install via `pip install
>>>>>> https://dist.apache.org/repos/dist/release/incubator/toree/0.1.0-incubating/toree-pip/apache-toree-0.1.0.tar.gz`
>>>>>> or whatever the command is that Gino was mentioning.
>>>>>>
>>>>>> If we want and are allowed to publish on Pypi, I'll do that soon.
>>>>>> Just let me know.
>>>>>>
>>>>>>
>>>>> Yes, please.
>>>>>
>>>>> --
>>>>> Luciano Resende
>>>>> http://twitter.com/lresende1975
>>>>> http://lresende.blogspot.com/
>>>>>
>>>>
>>>
>>>
>>> --
>>> Luciano Resende
>>> http://twitter.com/lresende1975
>>> http://lresende.blogspot.com/
>>>
>>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Uploading PIP for 0.1.0 release

[Chip Senkbeil]

It looks like all of our makefile work was updated awhile ago to use
apache-toree. This must have been from an earlier conversation. I can
update our release artifacts for pip to use toree, but we'll have to go
through the makefile and update it to produce toree in the future if that's
the route we want to go.

On Mon, Apr 17, 2017 at 1:27 PM Luciano Resende 
wrote:

> I looked into some other Apache projects, I believe we could follow the
> same pattern and use "toree". but describe it as "Apache Toree
> (incubating)". in the description field.
>
> On Mon, Apr 17, 2017 at 9:50 AM, Chip Senkbeil 
> wrote:
>
>> What is the exact name that you want for the release?
>>
>> Previously, we were using just "toree", but do you want "apache-toree" or
>> "apache-toree-incubating"?
>>
>> On Mon, Apr 17, 2017 at 10:47 AM Luciano Resende 
>> wrote:
>>
>>> On Mon, Apr 17, 2017 at 8:40 AM, Chip Senkbeil 
>>> wrote:
>>>
>>>> I've gotten mixed messages regarding where the pip artifacts should go.
>>>> Originally, we were going to publish them on
>>>> https://pypi.python.org/pypi (which we were previously doing for dev
>>>> snapshots), but there has been some issues with hosting externally from
>>>> Apache? Is that still the case? Can we host official releases on Pypi?
>>>>
>>>
>>> Official releases are ok to go there.
>>>
>>>
>>>
>>>> If not, we can just indicate that they install via `pip install
>>>> https://dist.apache.org/repos/dist/release/incubator/toree/0.1.0-incubating/toree-pip/apache-toree-0.1.0.tar.gz`
>>>> or whatever the command is that Gino was mentioning.
>>>>
>>>> If we want and are allowed to publish on Pypi, I'll do that soon. Just
>>>> let me know.
>>>>
>>>>
>>> Yes, please.
>>>
>>> --
>>> Luciano Resende
>>> http://twitter.com/lresende1975
>>> http://lresende.blogspot.com/
>>>
>>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Uploading PIP for 0.1.0 release

[Chip Senkbeil]

What is the exact name that you want for the release?

Previously, we were using just "toree", but do you want "apache-toree" or
"apache-toree-incubating"?

On Mon, Apr 17, 2017 at 10:47 AM Luciano Resende 
wrote:

> On Mon, Apr 17, 2017 at 8:40 AM, Chip Senkbeil 
> wrote:
>
>> I've gotten mixed messages regarding where the pip artifacts should go.
>> Originally, we were going to publish them on https://pypi.python.org/pypi 
>> (which
>> we were previously doing for dev snapshots), but there has been some issues
>> with hosting externally from Apache? Is that still the case? Can we host
>> official releases on Pypi?
>>
>
> Official releases are ok to go there.
>
>
>
>> If not, we can just indicate that they install via `pip install
>> https://dist.apache.org/repos/dist/release/incubator/toree/0.1.0-incubating/toree-pip/apache-toree-0.1.0.tar.gz`
>> or whatever the command is that Gino was mentioning.
>>
>> If we want and are allowed to publish on Pypi, I'll do that soon. Just
>> let me know.
>>
>>
> Yes, please.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Uploading PIP for 0.1.0 release

[Chip Senkbeil]

I've gotten mixed messages regarding where the pip artifacts should go.
Originally, we were going to publish them on
https://pypi.python.org/pypi (which
we were previously doing for dev snapshots), but there has been some issues
with hosting externally from Apache? Is that still the case? Can we host
official releases on Pypi? If not, we can just indicate that they install
via `pip install
https://dist.apache.org/repos/dist/release/incubator/toree/0.1.0-incubating/toree-pip/apache-toree-0.1.0.tar.gz`
or whatever the command is that Gino was mentioning.

If we want and are allowed to publish on Pypi, I'll do that soon. Just let
me know.

On Sun, Apr 16, 2017 at 2:28 PM Luciano Resende 
wrote:

> Hi Chip,
>
> Could you please handle the upload of the PIP install artifacts and
> document how to handle that for next releases.
>
> After that, I think we can send the announce e-mail.
>
> Thanks
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Toree website

[Chip Senkbeil]

The branch
https://github.com/apache/incubator-toree-website/tree/OverhaulSite was
meant to replace the current master and is closer to the current website
than the master branch.

I know that Gino/Corey were working on content that was based on the
branch, but they may have lost it.

On Fri, Mar 31, 2017 at 9:15 PM Luciano Resende 
wrote:

> How are we building the Toree website ? The current source at [1] seems to
> be very different from the current live website. Hopefully I am missing
> something and we are not updating the html site directly.
>
>
> [1] https://github.com/apache/incubator-toree-website
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: SQL interpreter

[Chip Senkbeil]

When refactoring our original interpreter codebase to support new languages
such as Python and R, I wrote the broker service to abstract the process of
communicating with another language's interpreter. When SQL was added, it
was easier to add a stubbed-out broker service such that the interpreter
followed the same structure and interfaces as the others.

The other intention was that we might fork the broker service to run in a
separate process along with the interpreter itself.

Go ahead and put your code as a WIP PR and I can take a look at it later.

On Mon, Apr 3, 2017 at 12:05 PM Ryan Blue  wrote:

> I've been working on improvements to the SQL interpreter, but I'm not quite
> sure why it works the way it currently does. Right now, the interpreter is
> associated with a SQL service that is a BrokerService. BrokerService says
> that it is a high-level interface between the JVM and another process, like
> R or Python, so it isn't clear why SQL uses it.
>
> The version I have working is considerably simpler and uses the lazy
> `kernel.sparkSession` accessor. Is there a reason why the current one
> doesn't use this approach?
>
> rb
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Lazy consensus for 0.1.0 release?

[Chip Senkbeil]

Hurray! Glad to hear from you again, Hitesh!

On Tue, Mar 28, 2017 at 4:01 PM Hitesh Shah  wrote:

> Hi folks
>
> Apologies for the delay in this. I will take a look at the release later
> today and comment on the vote.
>
> -- Hitesh
>
> On Wed, Mar 22, 2017 at 7:40 AM, Chip Senkbeil 
> wrote:
>
> > I've emailed all of our mentors and it's been a week with no response.
> > Luciano, how do we go about getting additional, active mentors? We're
> being
> > blocked on the release front purely from lack of votes.
> >
> > On Wed, Mar 15, 2017 at 4:04 PM Chip Senkbeil 
> > wrote:
> >
> > > Hitesh was our only other active mentor, and he seems to have
> > disappeared.
> > > Xin backed out because of too many commitments, IIRC. I've never
> > > seen Julien Le Dem involved in our project.
> > >
> > > On Wed, Mar 15, 2017 at 4:02 PM Luciano Resende 
> > > wrote:
> > >
> > > No, we do need 3 binding votes... I was mostly trying to provoque
> others
> > to
> > > review/vote.
> > >
> > > How about you send direct email to each mentor requesting their vote ?
> > >
> > > Also, it might be time to receuit more active mentors.
> > >
> > >
> > > On Wed, Mar 15, 2017 at 1:40 PM Chip Senkbeil  >
> > > wrote:
> > >
> > > > Given that we have only 1 vote (+1) in three weeks, should I go ahead
> > and
> > > > publish the release? I know Luciano was asking about that on @general
> > > > incubator. Just want to get confirmation from mentors before doing
> so.
> > > >
> > > --
> > > Sent from my Mobile device
> > >
> > >
> >
>

[jira] [Commented] (TOREE-396) SparkR interpeter does not connect to Spark 2.1

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936457#comment-15936457
 ] 

Chip Senkbeil commented on TOREE-396:
-

Thanks for pointing this out. Looks like the timeout was previously set to 6000.

> SparkR interpeter does not connect to Spark 2.1
> ---
>
> Key: TOREE-396
> URL: https://issues.apache.org/jira/browse/TOREE-396
> Project: TOREE
>  Issue Type: Bug
>Reporter: Mario Briggs
>
> I was trying the SparkR interpreter and trying to connect to a Spark 2.1 
> cluster. I get the following failure when execute any cell
> "Failed to connect JVM: Error in socketConnection(host = hostname, port = 
> port, server = FALSE, : argument "timeout" is missing, with no default
> Calls: sparkR.connect ... tryCatch -> tryCatchList -> tryCatchOne -> 
> 
> "
> Looks like in Spark2.1, the 
> [timeout|https://github.com/apache/spark/blob/branch-2.1/R/pkg/R/client.R#L22]
>  param to sparkRConnect no longer has a default value. 
> So this needs to fixed in 
> [here|https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/resources/kernelR/sparkr_runner.R#L65]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936446#comment-15936446
 ] 

Chip Senkbeil commented on TOREE-395:
-

I commented on the PR, but yeah I'm in favor of leaving it as an option with 
the default condition being disabled UNLESS one of you has a reason you believe 
that Console.out should be removed entirely.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Lazy consensus for 0.1.0 release?

[Chip Senkbeil]

I've emailed all of our mentors and it's been a week with no response.
Luciano, how do we go about getting additional, active mentors? We're being
blocked on the release front purely from lack of votes.

On Wed, Mar 15, 2017 at 4:04 PM Chip Senkbeil 
wrote:

> Hitesh was our only other active mentor, and he seems to have disappeared.
> Xin backed out because of too many commitments, IIRC. I've never
> seen Julien Le Dem involved in our project.
>
> On Wed, Mar 15, 2017 at 4:02 PM Luciano Resende 
> wrote:
>
> No, we do need 3 binding votes... I was mostly trying to provoque others to
> review/vote.
>
> How about you send direct email to each mentor requesting their vote ?
>
> Also, it might be time to receuit more active mentors.
>
>
> On Wed, Mar 15, 2017 at 1:40 PM Chip Senkbeil 
> wrote:
>
> > Given that we have only 1 vote (+1) in three weeks, should I go ahead and
> > publish the release? I know Luciano was asking about that on @general
> > incubator. Just want to get confirmation from mentors before doing so.
> >
> --
> Sent from my Mobile device
>
>

Re: Github Integration for Toree repositories

[Chip Senkbeil]

That's great news! Thanks for getting that set up! I'll take a look on
Monday to see how ghee integration works.

On Sat, Mar 18, 2017, 3:59 PM Luciano Resende  wrote:

> I have worked with infra team to enhance the issue workflow for Toree. We
> now have enabled Github integration that will update JIRA issues with the
> link for the Pull Request when they are created (please use [JIRA-xxx] on
> the PR header). This has been added to both toree and toree-website
> repositories.
>
> Also, comments to the PR are now being archived on the newly created
> reviews@ mailing list.
>
> Please let me know if you have questions or comments
>
> Thanks
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Lazy consensus for 0.1.0 release?

[Chip Senkbeil]

Hitesh was our only other active mentor, and he seems to have disappeared.
Xin backed out because of too many commitments, IIRC. I've never
seen Julien Le Dem involved in our project.

On Wed, Mar 15, 2017 at 4:02 PM Luciano Resende 
wrote:

> No, we do need 3 binding votes... I was mostly trying to provoque others to
> review/vote.
>
> How about you send direct email to each mentor requesting their vote ?
>
> Also, it might be time to receuit more active mentors.
>
>
> On Wed, Mar 15, 2017 at 1:40 PM Chip Senkbeil 
> wrote:
>
> > Given that we have only 1 vote (+1) in three weeks, should I go ahead and
> > publish the release? I know Luciano was asking about that on @general
> > incubator. Just want to get confirmation from mentors before doing so.
> >
> --
> Sent from my Mobile device
>

Lazy consensus for 0.1.0 release?

[Chip Senkbeil]

Given that we have only 1 vote (+1) in three weeks, should I go ahead and
publish the release? I know Luciano was asking about that on @general
incubator. Just want to get confirmation from mentors before doing so.

Re: Stale incubator vote thread

[Chip Senkbeil]

It's been over two weeks since the vote started. We only have Luciano's
vote. Is there anything we can do at this point?

On Wed, Mar 1, 2017 at 6:07 AM Chip Senkbeil 
wrote:

> Thanks, appreciate it!
>
> On Wed, Mar 1, 2017, 6:41 AM Luciano Resende  wrote:
>
> On Fri, Feb 24, 2017 at 7:25 AM, Chip Senkbeil 
> wrote:
>
> > It's nearly been three days since the thread has been opened. I've seen
> > other vote threads made after ours get some votes; so, I'm a little
> > concerned that our thread has become a bit stale.
> >
> > Any mentor available to take a look at the thread and - if it looks good
> -
> > provide a +1 to get the spotlight back on the thread? Or at least comment
> > on it to get some attention? I suppose I could try to bump the thread
> > myself.
> >
> > https://lists.apache.org/thread.html/ecb5e40078f01c488ada52e776d57c
> > d80e130164c1e31d8ded564ba2@
> > 
> >
> > Luciano? Hitesh?
> >
>
> Sorry, I was traveling and without access to gmail. I have reviewed and
> voted on the general thread.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
>

Re: Best Practices for Creating and Merging PRs

[Chip Senkbeil]

We don't have access to the GitHub merge PR button, do we? That I'm aware
of, Gino, Corey, and I have been manually merging fully-rebased PRs to keep
a clean commit history.

If the GitHub merge PR button was available, we could use the
squash-and-rebase option, which provides a very clean way to both reduce
the amount of commits and ensure the history is ordered properly. To me, it
is the most modern and sensible approach. I would do that manually if I had
a way to mark the PRs on Github as merged afterward.

As for providing guidelines for PRs, I'm not sure if there is much of a
point until we get a higher volume of traffic where we need to increase
structure. We could at least encourage PRs that have the JIRA issue number
in the title. That makes sense.

On Mon, Mar 6, 2017 at 10:10 AM Luciano Resende 
wrote:

> Do we have any set of recommendations on creating and Merging PRs for the
> Toree Project ?
>
> At least on the Merging side, I would like to recommend that we avoid using
> the GitHub merge PR button as that causes a bunch of not useful commits to
> be displayed in the commit history such as: "Merging pr #101" which does
> not provide any information about what the change is, etc...
>
> If folks are ok, I would also like to propose we start using the Spark
> recommendation for creating PRs or commits which I summarized in [1]
>
> [1]
>
> https://github.com/SparkTC/development-guidelines/blob/master/project-contribution-guidelines.md#creating-a-pull-request
>
> Thoughts ?
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Stale incubator vote thread

[Chip Senkbeil]

Thanks, appreciate it!

On Wed, Mar 1, 2017, 6:41 AM Luciano Resende  wrote:

> On Fri, Feb 24, 2017 at 7:25 AM, Chip Senkbeil 
> wrote:
>
> > It's nearly been three days since the thread has been opened. I've seen
> > other vote threads made after ours get some votes; so, I'm a little
> > concerned that our thread has become a bit stale.
> >
> > Any mentor available to take a look at the thread and - if it looks good
> -
> > provide a +1 to get the spotlight back on the thread? Or at least comment
> > on it to get some attention? I suppose I could try to bump the thread
> > myself.
> >
> > https://lists.apache.org/thread.html/ecb5e40078f01c488ada52e776d57c
> > d80e130164c1e31d8ded564ba2@
> > 
> >
> > Luciano? Hitesh?
> >
>
> Sorry, I was traveling and without access to gmail. I have reviewed and
> voted on the general thread.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Stale incubator vote thread

[Chip Senkbeil]

It's nearly been three days since the thread has been opened. I've seen
other vote threads made after ours get some votes; so, I'm a little
concerned that our thread has become a bit stale.

Any mentor available to take a look at the thread and - if it looks good -
provide a +1 to get the spotlight back on the thread? Or at least comment
on it to get some attention? I suppose I could try to bump the thread
myself.

https://lists.apache.org/thread.html/ecb5e40078f01c488ada52e776d57cd80e130164c1e31d8ded564ba2@


Luciano? Hitesh?

Re: [RESULT] [VOTE] Apache Toree 0.1.0 RC6

[Chip Senkbeil]

Vote thread on the incubator @general channel can be found here:

https://lists.apache.org/thread.html/ecb5e40078f01c488ada52e776d57cd80e130164c1e31d8ded564ba2@


On Tue, Feb 21, 2017 at 2:05 PM Chip Senkbeil 
wrote:

> Voting is now closed. This vote has PASSED with 4 +1 votes and no 0 or -1
> votes. This vote will now go to the IPMC list for a second 72-hour minimum
> vote. Toree developers are encouraged to comment on the IPMC vote as well.
>
> +1:
> Chip Senkbeil
> Marius van Niekerk
> Gino Bustelo
> Corey Stubbs
>
> 0:
>
> -1:
>

[RESULT] [VOTE] Apache Toree 0.1.0 RC6

[Chip Senkbeil]

Voting is now closed. This vote has PASSED with 4 +1 votes and no 0 or -1
votes. This vote will now go to the IPMC list for a second 72-hour minimum
vote. Toree developers are encouraged to comment on the IPMC vote as well.

+1:
Chip Senkbeil
Marius van Niekerk
Gino Bustelo
Corey Stubbs

0:

-1:

Re: [VOTE] Apache Toree 0.1.0 RC6

[Chip Senkbeil]

It's been over 72 hours (not including weekend), so going to close this and
start the vote in incubutor general.

On Mon, Feb 20, 2017 at 9:33 AM Gino Bustelo  wrote:

> +1
>
> I installed it locally on my machine and it works
>
> On Mon, Feb 20, 2017 at 9:32 AM Corey Stubbs  wrote:
>
> > +1 Ran and installed the pip dist with Apache Spark 1.6.3.
> >
> > On Thu, Feb 16, 2017 at 11:24 AM Marius van Niekerk <
> > marius.v.niek...@gmail.com> wrote:
> >
> > > +1
> > >
> > > Going to +1 this since I assisted with getting rid of some of the
> bundled
> > > pieces that were blocking for RC5.
> > >
> > > On Thu, Feb 16, 2017, 12:14 Chip Senkbeil 
> > wrote:
> > >
> > > I'll go ahead and give a +1 since I've done a bit of testing using
> > > Scala/SparkR to get this back up to speed.
> > >
> > > Also resolved the remaining issues we had from the incubator general
> > vote.
> > > We're still bundling additional license information in the binary
> > release,
> > > but the source release no longer has the extra license info tacked onto
> > the
> > > LICENSE file. I've also fixed the source release such that it is no
> > longer
> > > dependent on being in a Git repository to be built nor does it require
> > you
> > > to have sbt installed.
> > >
> > > On Thu, Feb 16, 2017 at 11:09 AM Chip Senkbeil <
> chip.senkb...@gmail.com>
> > > wrote:
> > >
> > > > Please vote to approve the release of the following candidate as
> Apache
> > > > Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE
> > > files
> > > > since this is our first release.
> > > >
> > > > ## Information and Artifacts
> > > >
> > > > The tag to be voted on is v0.1.0-rc6
> > > > (51fa49cb5898e0c5b7824f986382436b969cabc7), located here:
> > > >
> > > >
> > > >
> > >
> > >
> >
> https://github.com/apache/incubator-toree/commit/51fa49cb5898e0c5b7824f986382436b969cabc7
> > > >
> > > > All distribution packages, including signatures, digests, etc. can be
> > > found
> > > > at:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/
> > > >
> > > > Staging artifacts can be found at:
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapachetoree-1006
> > > >
> > > > ## Testing Instructions
> > > >
> > > > The fastest way to get up and running is to using Jupyter.
> > > >
> > > > 1. Install Jupyter if you haven't already (
> > > http://jupyter.org/install.html
> > > > )
> > > >
> > > > 2. Grab the Apache Toree archive from
> > > >
> > > >
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/toree-pip/apache-toree-0.1.0.tar.gz
> > > >
> > > > 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> > > > followed by `jupyter toree install`
> > > >
> > > > - You need to set a valid Apache Spark 1.6.x home, which can be done
> > via
> > > > `jupyter
> > > > toree install --spark_home=/usr/local/spark`
> > > >
> > > > - You may need to run with `sudo` for installation permission
> > > >
> > > > - For all installation options, run `jupyter toree install
> --help-all`
> > > >
> > > > 4. Run a Jupyter notebook server via `jupyter notebook`
> > > >
> > > > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > > > you can install via `pip install notebook`
> > > >
> > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
> > under
> > > > new
> > > >
> > > > - If you want other interpreter profiles than Scala, you can change
> the
> > > > interpreters via `jupyter toree install --interpreters=PySpark,SQL`
> > > >
> > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()`
> in
> > > the
> > > > notebook
> > > >
> > > > ## Voting Instructions
> > > >
> > > > The vote is open for at least 72 hours and passes if a majority of at
> > > least
> > > > 3 +1 PMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Toree 0.1.0
> > > > [ ] -1 Do not release this package because ...
> > > >
> > >
> > > --
> > > regards
> > > Marius van Niekerk
> > >
> >
>

Re: Supported Scala versions

[Chip Senkbeil]

The 0.2.x branch (currently master) is for Scala 2.11 and Spark 2.x.

The 0.1.x branch is for Scala 2.10 and Spark 1.x.

On Thu, Feb 16, 2017 at 2:41 PM Jakob Odersky  wrote:

> Hi everyone,
>
> during a recent discussion with Marius
> https://github.com/apache/incubator-toree/pull/93, I found out that
> Toree does not support being built with Scala 2.10 anymore.
>
> I am all in favor of dropping the EOL scala version, however
> considering that there are still various references to the old version
> and, more importantly, version specific files such as the the
> interpreters, I wanted to check on the mailing list what Toree's
> official standpoint on Scala support is.
>
> best,
> --Jakob
>

Re: Interpreter outputs

[Chip Senkbeil]

I'll take a look at them soon and give you some feedback. Was preparing the
next vote for the 0.1.0 release, which is why I've been a bit distracted.

On Thu, Feb 16, 2017 at 12:05 PM Ryan Blue 
wrote:

> Hi everyone,
>
> I've been putting together a proof-of-concept Toree deployment for our
> Spark users to test out. One of the things I needed to change was how
> interpreter outputs are displayed, allowing interpreters to show HTML or
> other representations of output objects. For anyone else interested, I've
> put up a PR with the changes for discussion:
>
>   https://github.com/apache/incubator-toree/pull/104
>
> There are also a few other PRs that fix tab completion, code completeness
> checks, syntax highlighting, and other usability issues. It would be great
> to get feedback from the community.
>
> Thanks!
>
> rb
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: [VOTE] Apache Toree 0.1.0 RC6

[Chip Senkbeil]

I'll go ahead and give a +1 since I've done a bit of testing using
Scala/SparkR to get this back up to speed.

Also resolved the remaining issues we had from the incubator general vote.
We're still bundling additional license information in the binary release,
but the source release no longer has the extra license info tacked onto the
LICENSE file. I've also fixed the source release such that it is no longer
dependent on being in a Git repository to be built nor does it require you
to have sbt installed.

On Thu, Feb 16, 2017 at 11:09 AM Chip Senkbeil 
wrote:

> Please vote to approve the release of the following candidate as Apache
> Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE files
> since this is our first release.
>
> ## Information and Artifacts
>
> The tag to be voted on is v0.1.0-rc6
> (51fa49cb5898e0c5b7824f986382436b969cabc7), located here:
>
>
> https://github.com/apache/incubator-toree/commit/51fa49cb5898e0c5b7824f986382436b969cabc7
>
> All distribution packages, including signatures, digests, etc. can be found
> at:
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachetoree-1006
>
> ## Testing Instructions
>
> The fastest way to get up and running is to using Jupyter.
>
> 1. Install Jupyter if you haven't already (http://jupyter.org/install.html
> )
>
> 2. Grab the Apache Toree archive from
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/toree-pip/apache-toree-0.1.0.tar.gz
>
> 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> followed by `jupyter toree install`
>
> - You need to set a valid Apache Spark 1.6.x home, which can be done via
> `jupyter
> toree install --spark_home=/usr/local/spark`
>
> - You may need to run with `sudo` for installation permission
>
> - For all installation options, run `jupyter toree install --help-all`
>
> 4. Run a Jupyter notebook server via `jupyter notebook`
>
> - If the notebook portion of Jupyter is not installed but Jupyter is,
> you can install via `pip install notebook`
>
> 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> new
>
> - If you want other interpreter profiles than Scala, you can change the
> interpreters via `jupyter toree install --interpreters=PySpark,SQL`
>
> 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the
> notebook
>
> ## Voting Instructions
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.1.0
> [ ] -1 Do not release this package because ...
>

[VOTE] Apache Toree 0.1.0 RC6

[Chip Senkbeil]

Please vote to approve the release of the following candidate as Apache
Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE files
since this is our first release.

## Information and Artifacts

The tag to be voted on is v0.1.0-rc6
(51fa49cb5898e0c5b7824f986382436b969cabc7), located here:

https://github.com/apache/incubator-toree/commit/51fa49cb5898e0c5b7824f986382436b969cabc7

All distribution packages, including signatures, digests, etc. can be found
at:

https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachetoree-1006

## Testing Instructions

The fastest way to get up and running is to using Jupyter.

1. Install Jupyter if you haven't already (http://jupyter.org/install.html)

2. Grab the Apache Toree archive from
https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/toree-pip/apache-toree-0.1.0.tar.gz

3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
followed by `jupyter toree install`

- You need to set a valid Apache Spark 1.6.x home, which can be done via
`jupyter
toree install --spark_home=/usr/local/spark`

- You may need to run with `sudo` for installation permission

- For all installation options, run `jupyter toree install --help-all`

4. Run a Jupyter notebook server via `jupyter notebook`

- If the notebook portion of Jupyter is not installed but Jupyter is,
you can install via `pip install notebook`

5. Create a notebook using "Apache Toree - Scala" from the dropdown under
new

- If you want other interpreter profiles than Scala, you can change the
interpreters via `jupyter toree install --interpreters=PySpark,SQL`

6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the
notebook

## Voting Instructions

The vote is open for at least 72 hours and passes if a majority of at least
3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Toree 0.1.0
[ ] -1 Do not release this package because ...

Re: Toree for 1.6.x is broken

[Chip Senkbeil]

Looks like there was a missing SparkR::: for connExists. Maybe it became
public in 2.0, but it was hidden in 1.6. Testing  now before starting vote
for RC6.

On Thu, Feb 16, 2017 at 10:54 AM Marius van Niekerk <
marius.v.niek...@gmail.com> wrote:

> The sparkR error might be something i did wrong when backporting the spark
> 2.0 R things
>
> On Thu, 16 Feb 2017 at 10:18 Chip Senkbeil 
> wrote:
>
> > Turns out I needed to set my hostname on my machine to localhost. Doing
> so
> > fixed that error.
> >
> > As for the SparkR error, I'll take a look.
> >
> > On Thu, Feb 16, 2017 at 9:07 AM Gino Bustelo  wrote:
> >
> > > I tried your pip package at
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/
> > > using
> > > the `make test-release` target and it worked fine. I ran the magics
> test
> > > notebook and everything worked except there is an issue when calling
> the
> > > %SparkR magic. It ends up in an infinite loop printing
> > >
> > > Warning message:
> > >
> > > In rm(".sparkRcon", envir = .sparkREnv) : object '.sparkRcon' not found
> > >
> > > Error in sparkR.connect() : could not find function "connExists"
> > >
> > > Execution halted
> > >
> > > 17/02/15 23:15:51 [ERROR] o.a.t.k.i.s.SparkRProcessHandler - null
> process
> > > exited: 1
> > >
> > > Loading required package: methods
> > >
> > >
> > > Attaching package: ‘SparkR’
> > >
> > >
> > > The following objects are masked from ‘package:stats’:
> > >
> > >
> > > cov, filter, lag, na.omit, predict, sd, var
> > >
> > >
> > > The following objects are masked from ‘package:base’:
> > >
> > >
> > > colnames, colnames<-, endsWith, intersect, rank, rbind, sample,
> > >
> > > startsWith, subset, summary, table, transform
> > >
> > >
> > > On Thu, Feb 16, 2017 at 9:03 AM Corey Stubbs 
> wrote:
> > >
> > > > I was able to run from the pip package just fine.
> > > >
> > > > On Wed, Feb 15, 2017 at 4:50 PM Chip Senkbeil <
> chip.senkb...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > By master, I mean the 0.1.x branch. Was trying to get the next vote
> > > > > started.
> > > > >
> > > > > On Wed, Feb 15, 2017, 4:44 PM Chip Senkbeil <
> chip.senkb...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Just built master, got all of the artifacts ready, blah blah
> blah.
> > > > Tested
> > > > > > by installing the artifact from
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/
> > > > > and
> > > > > > now it's failing with not being able to bind to an ephemeral port
> > for
> > > > the
> > > > > > sparkDriver. Can someone help me take a look at this? I just
> > > installed
> > > > > like
> > > > > > usual via `pip install apache-toree-0.1.0.tar.gz` and pointed to
> > > spark
> > > > > > distributions I downloaded (1.6.1 and 1.6.3) when running
> `jupyter
> > > > toree
> > > > > > install --spark_home=...`. When launching a kernel, it fails
> > with...
> > > > > >
> > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> > could
> > > > not
> > > > > > bind on port 0. Attempting port 1.
> > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> > could
> > > > not
> > > > > > bind on port 0. Attempting port 1.
> > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> > could
> > > > not
> > > > > > bind on port 0. Attempting port 1.
> > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> > could
> > > > not
> > > > > > bind on port 0. Attempting port 1.
> > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> > could
> > > > not
> > > > > > bind on port 0. Attempting 

Re: Toree for 1.6.x is broken

[Chip Senkbeil]

Turns out I needed to set my hostname on my machine to localhost. Doing so
fixed that error.

As for the SparkR error, I'll take a look.

On Thu, Feb 16, 2017 at 9:07 AM Gino Bustelo  wrote:

> I tried your pip package at
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/
> using
> the `make test-release` target and it worked fine. I ran the magics test
> notebook and everything worked except there is an issue when calling the
> %SparkR magic. It ends up in an infinite loop printing
>
> Warning message:
>
> In rm(".sparkRcon", envir = .sparkREnv) : object '.sparkRcon' not found
>
> Error in sparkR.connect() : could not find function "connExists"
>
> Execution halted
>
> 17/02/15 23:15:51 [ERROR] o.a.t.k.i.s.SparkRProcessHandler - null process
> exited: 1
>
> Loading required package: methods
>
>
> Attaching package: ‘SparkR’
>
>
> The following objects are masked from ‘package:stats’:
>
>
> cov, filter, lag, na.omit, predict, sd, var
>
>
> The following objects are masked from ‘package:base’:
>
>
> colnames, colnames<-, endsWith, intersect, rank, rbind, sample,
>
> startsWith, subset, summary, table, transform
>
>
> On Thu, Feb 16, 2017 at 9:03 AM Corey Stubbs  wrote:
>
> > I was able to run from the pip package just fine.
> >
> > On Wed, Feb 15, 2017 at 4:50 PM Chip Senkbeil 
> > wrote:
> >
> > > By master, I mean the 0.1.x branch. Was trying to get the next vote
> > > started.
> > >
> > > On Wed, Feb 15, 2017, 4:44 PM Chip Senkbeil 
> > > wrote:
> > >
> > > > Just built master, got all of the artifacts ready, blah blah blah.
> > Tested
> > > > by installing the artifact from
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/
> > > and
> > > > now it's failing with not being able to bind to an ephemeral port for
> > the
> > > > sparkDriver. Can someone help me take a look at this? I just
> installed
> > > like
> > > > usual via `pip install apache-toree-0.1.0.tar.gz` and pointed to
> spark
> > > > distributions I downloaded (1.6.1 and 1.6.3) when running `jupyter
> > toree
> > > > install --spark_home=...`. When launching a kernel, it fails with...
> > > >
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could
> > not
> > > > bind on port 0. Attempting port 1.
> > > > 17/02/15 16:41:29 [WARN] o.a.s.

Re: Toree for 1.6.x is broken

[Chip Senkbeil]

By master, I mean the 0.1.x branch. Was trying to get the next vote started.

On Wed, Feb 15, 2017, 4:44 PM Chip Senkbeil  wrote:

> Just built master, got all of the artifacts ready, blah blah blah. Tested
> by installing the artifact from
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/ 
> and
> now it's failing with not being able to bind to an ephemeral port for the
> sparkDriver. Can someone help me take a look at this? I just installed like
> usual via `pip install apache-toree-0.1.0.tar.gz` and pointed to spark
> distributions I downloaded (1.6.1 and 1.6.3) when running `jupyter toree
> install --spark_home=...`. When launching a kernel, it fails with...
>
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
> bind on port 0. Attempting port 1.
> 17/02/15 16:41:29 [ERROR] o.a.s.SparkContext - Error initializing
> SparkContext.
> java.net.BindException: Can't assign requested address: Service
> 'sparkDriver' failed after 16 retries!
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
> at
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
> at
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
> at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
> at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
>

Toree for 1.6.x is broken

[Chip Senkbeil]

Just built master, got all of the artifacts ready, blah blah blah. Tested
by installing the artifact from
https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/ and
now it's failing with not being able to bind to an ephemeral port for the
sparkDriver. Can someone help me take a look at this? I just installed like
usual via `pip install apache-toree-0.1.0.tar.gz` and pointed to spark
distributions I downloaded (1.6.1 and 1.6.3) when running `jupyter toree
install --spark_home=...`. When launching a kernel, it fails with...

17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver' could not
bind on port 0. Attempting port 1.
17/02/15 16:41:29 [ERROR] o.a.s.SparkContext - Error initializing
SparkContext.
java.net.BindException: Can't assign requested address: Service
'sparkDriver' failed after 16 retries!
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
at
io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
at
io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
at
io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)

[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856197#comment-15856197
 ] 

Chip Senkbeil commented on TOREE-374:
-

Here's what I used to do. It still works with our 0.1.x branch. Assuming it'll 
work on master using Scala 2.11's REPL implementation.

{code}
val x = 3
println(x) // show
{code}

Just tack on a {code}// show{code} at the end of your code, the space between 
the forward slash and show being required.

> Variables declared on the Notebook are not garbage collected
> 
>
> Key: TOREE-374
> URL: https://issues.apache.org/jira/browse/TOREE-374
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: David Taieb
>
> I'm not sure if it's a bug or a limitation of the underlying scala REPL.
> As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) 
> auto-visualization feature within Scala gateway, I have implemented a weak 
> hashmap that tracks objects declared on the Scala REPL. However, I have found 
> that objects are not correctly gc'ed when the object is declared in a cell 
> with a val or var keyword and then the cell is ran again. One would expect 
> that the original object has no more references and should be gc'ed but it's 
> not. 
> However, when the object is declare with var keyword and then set to null in 
> another cell, then it is correctly gc'ed.
> I'm concerned that users who run the same cell multiple times would 
> unwittingly have memory leaks which can eventually lead to OOM errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Podling Report

[Chip Senkbeil]

Thanks :)

On Thu, Feb 2, 2017 at 9:56 AM Luciano Resende  wrote:

> I have uploaded the report to the wiki and signed off as a mentor, and will
> take the blame if anyone complains about it :)
>
> On Thu, Feb 2, 2017 at 4:28 AM, Chip Senkbeil 
> wrote:
>
> > Guess we missed this one since it was due yesterday. Considering they
> > really didn't like when we edited after the deadline earlier, we should
> > just be listed as missing the report.
> >
> > On Wed, Feb 1, 2017, 11:12 PM Marius van Niekerk <
> > marius.v.niek...@gmail.com>
> > wrote:
> >
> > > Pretty sure the following is correct, my wiki account is broken somehow
> > >
> > > 
> > >
> > > Toree
> > >
> > > Toree provides applications with a mechanism to interactively and
> > remotely
> > > access Apache Spark. It enables interactive workloads between
> > applications
> > > and
> > > a Spark cluster. As a Jupyter Notebook extension, it provides the user
> > > with a
> > > preconfigured environment for interacting with Spark using Scala,
> Python,
> > > R or
> > > SQL.
> > >
> > > Toree has been incubating since 2015-12-02.
> > >
> > > Three most important issues to address in the move towards graduation:
> > >
> > >  1. Make a release: The community is working on RC from 0.1.x branch.
> > > Master
> > >has moved to start support for Spark 2.0. Currently working on RC5.
> > >  2. Grow a diverse community: We should put some emphasis on growing
> the
> > > community and making it diverse (the rule is at least three
> > independent
> > > contributors) In progress.
> > >
> > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > aware
> > > of?
> > >
> > >  NONE
> > >
> > > How has the community developed since the last report?
> > >
> > >  1. Active communication in mailing list and gitter with early adopters
> > >  2. Migrated gitter to away from ibm-et/spark-kernel to apache/toree
> > >  3. More external contributions being made. Mainly focused on master to
> > > stabilize Spark 2.0 support
> > >
> > > How has the project developed since the last report?
> > >
> > >  1. Still working on 1st release.  Removed several legacy binaries from
> > > release builds that were not in compliance with Apache release
> processes.
> > > Got an RC5 ready soon
> > >  2. Addressing issues opened by community
> > >
> > > Date of last release:
> > >
> > >  None since incubation.
> > >
> > > When were the last committers or PMC members elected?
> > >
> > >  No new additions since incubation
> > >
> > > Signed-off-by:
> > >
> > >  [ ](toree) Luciano Resende
> > >  [ ](toree) Reynold Xin
> > >  [ ](toree) Hitesh Shah
> > >  [ ](toree) Julien Le Dem
> > >
> > > Shepherd/Mentor notes:
> > >
> > > 
> > >
> > > --
> > > regards
> > > Marius van Niekerk
> > >
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Podling Report

[Chip Senkbeil]

Guess we missed this one since it was due yesterday. Considering they
really didn't like when we edited after the deadline earlier, we should
just be listed as missing the report.

On Wed, Feb 1, 2017, 11:12 PM Marius van Niekerk 
wrote:

> Pretty sure the following is correct, my wiki account is broken somehow
>
> 
>
> Toree
>
> Toree provides applications with a mechanism to interactively and remotely
> access Apache Spark. It enables interactive workloads between applications
> and
> a Spark cluster. As a Jupyter Notebook extension, it provides the user
> with a
> preconfigured environment for interacting with Spark using Scala, Python,
> R or
> SQL.
>
> Toree has been incubating since 2015-12-02.
>
> Three most important issues to address in the move towards graduation:
>
>  1. Make a release: The community is working on RC from 0.1.x branch.
> Master
>has moved to start support for Spark 2.0. Currently working on RC5.
>  2. Grow a diverse community: We should put some emphasis on growing the
> community and making it diverse (the rule is at least three independent
> contributors) In progress.
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be aware
> of?
>
>  NONE
>
> How has the community developed since the last report?
>
>  1. Active communication in mailing list and gitter with early adopters
>  2. Migrated gitter to away from ibm-et/spark-kernel to apache/toree
>  3. More external contributions being made. Mainly focused on master to
> stabilize Spark 2.0 support
>
> How has the project developed since the last report?
>
>  1. Still working on 1st release.  Removed several legacy binaries from
> release builds that were not in compliance with Apache release processes.
> Got an RC5 ready soon
>  2. Addressing issues opened by community
>
> Date of last release:
>
>  None since incubation.
>
> When were the last committers or PMC members elected?
>
>  No new additions since incubation
>
> Signed-off-by:
>
>  [ ](toree) Luciano Resende
>  [ ](toree) Reynold Xin
>  [ ](toree) Hitesh Shah
>  [ ](toree) Julien Le Dem
>
> Shepherd/Mentor notes:
>
> 
>
> --
> regards
> Marius van Niekerk
>

Podling Report

[Chip Senkbeil]

Looks like the report is due today. Marius, are you available to take care
of it this month? I don't have time today.

Re: Calling a vote again?

[Chip Senkbeil]

Sorry, had a deadline at work today. I need to clear up a couple of things
before I cut new artefacts for a vote.

On Wed, Feb 1, 2017, 10:14 AM Marius van Niekerk 
wrote:

> I think we've cleared out the required static jar pieces.  Should we call a
> vote again?
>
> -Marius
> --
> regards
> Marius van Niekerk
>

Close incubator vote

[Chip Senkbeil]

Do I need to add a reply closing the existing vote on general@incubator if
we are preparing a new release? Marius and I have been addressing the
issues they've mentioned on general@incubator in preparation for a new vote
here followed by another vote on incubator. Just not sure if I need to do
anything about the old vote thread on incubator.

Re: [RESULT] [VOTE] Apache Toree 0.1.0 RC4

[Chip Senkbeil]

Okay, as long as it is acceptable to leave it running for longer than the
72 hour period, I guess we're in good shape. Thanks!

On Mon, Jan 16, 2017 at 9:33 AM Luciano Resende 
wrote:

> Thanks for getting this moving Chip, couple comments:
>
> When the 72hrs vote goes mostly over the weekend, it's usually good
> practice to let it run for couple more days to make sure folks had time to
> vote.
>
> Another thing, IPMC votes usually will actually take longer, as we have to
> wait for 3 binding votes from IPMC, so untill that the vote continues open.
>
>
>
> On Mon, Jan 16, 2017 at 7:18 AM, Chip Senkbeil 
> wrote:
>
> > No votes as of yet and it closes later today. Should some of us comment
> on
> > the thread to get activity again?
> >
> > On Fri, Jan 13, 2017 at 4:45 PM Chip Senkbeil 
> > wrote:
> >
> > > The IPMC vote thread is here:
> > > https://lists.apache.org/thread.html/f891ecb9452b6b34b27e0f2879efb5
> > 70e3768f8d09943a57bdd6e9d8@
> > >
> > > On Fri, Jan 13, 2017 at 4:26 PM Chip Senkbeil  >
> > > wrote:
> > >
> > > Voting is now closed. This vote has PASSED with 5 +1 votes and no 0 or
> -1
> > > votes. This vote will now go to the IPMC list for a second 72-hour
> vote.
> > > Toree developers are encouraged to comment on the IPMC
> > > vote as well.
> > >
> > > +1:
> > > Chip Senkbeil
> > > Luciano Resende
> > > Marius van Niekerk
> > > Gino Bustelo
> > > Corey Stubbs
> > >
> > > 0:
> > >
> > > -1:
> > >
> > >
> > >
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [RESULT] [VOTE] Apache Toree 0.1.0 RC4

[Chip Senkbeil]

No votes as of yet and it closes later today. Should some of us comment on
the thread to get activity again?

On Fri, Jan 13, 2017 at 4:45 PM Chip Senkbeil 
wrote:

> The IPMC vote thread is here:
> https://lists.apache.org/thread.html/f891ecb9452b6b34b27e0f2879efb570e3768f8d09943a57bdd6e9d8@
>
> On Fri, Jan 13, 2017 at 4:26 PM Chip Senkbeil 
> wrote:
>
> Voting is now closed. This vote has PASSED with 5 +1 votes and no 0 or -1
> votes. This vote will now go to the IPMC list for a second 72-hour vote.
> Toree developers are encouraged to comment on the IPMC
> vote as well.
>
> +1:
> Chip Senkbeil
> Luciano Resende
> Marius van Niekerk
> Gino Bustelo
> Corey Stubbs
>
> 0:
>
> -1:
>
>
>

Re: [RESULT] [VOTE] Apache Toree 0.1.0 RC4

[Chip Senkbeil]

The IPMC vote thread is here:
https://lists.apache.org/thread.html/f891ecb9452b6b34b27e0f2879efb570e3768f8d09943a57bdd6e9d8@

On Fri, Jan 13, 2017 at 4:26 PM Chip Senkbeil 
wrote:

> Voting is now closed. This vote has PASSED with 5 +1 votes and no 0 or -1
> votes. This vote will now go to the IPMC list for a second 72-hour vote.
> Toree developers are encouraged to comment on the IPMC
> vote as well.
>
> +1:
> Chip Senkbeil
> Luciano Resende
> Marius van Niekerk
> Gino Bustelo
> Corey Stubbs
>
> 0:
>
> -1:
>
>
>

[RESULT] [VOTE] Apache Toree 0.1.0 RC4

[Chip Senkbeil]

Voting is now closed. This vote has PASSED with 5 +1 votes and no 0 or -1
votes. This vote will now go to the IPMC list for a second 72-hour vote.
Toree developers are encouraged to comment on the IPMC
vote as well.

+1:
Chip Senkbeil
Luciano Resende
Marius van Niekerk
Gino Bustelo
Corey Stubbs

0:

-1:

Re: issue running toree kernel on mesos cluster

[Chip Senkbeil]

For some reason, I'm missing the first half of this thread. I only have
Paco Nathan's email. I don't have one from Eszti and I don't see one in our
mailing archive, either. Can someone repost his part?

As for the runtime errors in Docker, we'd be happy to accept any and all
contributions back into the main project.

In terms of related reports, I don't remember any with regard to Docker,
but I'd need to look back through our JIRAs to be certain.

On Wed, Jan 11, 2017 at 6:25 PM Paco Nathan  wrote:

> I'll chime in on this, too -
>
> We run Jupyter kernels inside Docker containers, running on an Apache Mesos
> cluster. This is used for online publishing
> .
>
> Spark and related projects are popular among our authors at O'Reilly, so
> we've got a big need to be using Toree.  Works great when run on laptops,
> though with an Apache 2.x version of Toree inside a container we encounter
> errors -- as Eszti described above,
>
> Some of our ops team at O'Reilly can rewrite the build, to work around
> these runtime errors in Docker. We'd like to contribute that back to the
> Toree project, rather than maintain separately.
>
> First we wanted to check with Toree devs, to see if that fits, doesn't
> conflict with your plans, etc. ?  Also, have there been any related bug
> reports -- might be useful.
>
> Thanks,
> Paco
>
> PS: we may have some history here, not 100% sure: saw the commit from Osh
> -- over a year ago we had Jeremy Freeman's team (mostly Osh
> ) consult to develop a tutorial based on
> Spark in a container, and I have a hunch some of that code may have gone
> into Toree?
>

Re: [VOTE] Apache Toree 0.1.0 RC4

[Chip Senkbeil]

Forgot to mention that the required Apache Spark version to use with Toree
for the 0.1.x branch is Apache Spark 1.6.x.

I've tested the release artifacts myself using the provided testing method
and everything checks out. Have run Rat against the artifacts and they look
good.

So, going to give it a +1.

On Tue, Jan 10, 2017 at 4:15 PM Chip Senkbeil 
wrote:

> Please vote to approve the release of the following candidate as Apache
> Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE files
> since this is our first release.
>
> ## Information and Artifacts
>
> The tag to be voted on is v0.1.0-rc4
> (1d526954ecaba1d5dc0f40ec555dd598b2b11df7), located here:
>
>
> https://github.com/apache/incubator-toree/commit/1d526954ecaba1d5dc0f40ec555dd598b2b11df7
>
> All distribution packages, including signatures, digests, etc. can be
> found at:
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc4/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachetoree-1002/
>
> ## Testing Instructions
>
> The fastest way to get up and running is to using Jupyter.
>
> 1. Install Jupyter if you haven't already (http://jupyter.org/install.html
> )
>
> 2. Grab the Apache Toree archive from
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc4/toree-pip/apache-toree-0.1.0.tar.gz
>
> 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> followed by `jupyter toree install`
>
> - You need to set a valid Spark home, which can be done via `jupyter
> toree install --spark_home=/usr/local/spark`
>
> - You may need to run with `sudo` for installation permission
>
> - For all installation options, run `jupyter toree install --help-all`
>
> 4. Run a Jupyter notebook server via `jupyter notebook`
>
> - If the notebook portion of Jupyter is not installed but Jupyter is,
> you can install via `pip install notebook`
>
> 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> new
>
> - If you want other interpreter profiles than Scala, you can change
> the interpreters via `jupyter toree install --interpreters=PySpark,SQL`
>
> 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> the notebook
>
> ## Voting Instructions
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.1.0
> [ ] -1 Do not release this package because ...
>

[VOTE] Apache Toree 0.1.0 RC4

[Chip Senkbeil]

Please vote to approve the release of the following candidate as Apache
Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE files
since this is our first release.

## Information and Artifacts

The tag to be voted on is v0.1.0-rc4
(1d526954ecaba1d5dc0f40ec555dd598b2b11df7), located here:

https://github.com/apache/incubator-toree/commit/1d526954ecaba1d5dc0f40ec555dd598b2b11df7

All distribution packages, including signatures, digests, etc. can be found
at:

https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc4/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachetoree-1002/

## Testing Instructions

The fastest way to get up and running is to using Jupyter.

1. Install Jupyter if you haven't already (http://jupyter.org/install.html)

2. Grab the Apache Toree archive from
https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc4/toree-pip/apache-toree-0.1.0.tar.gz

3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
followed by `jupyter toree install`

- You need to set a valid Spark home, which can be done via `jupyter
toree install --spark_home=/usr/local/spark`

- You may need to run with `sudo` for installation permission

- For all installation options, run `jupyter toree install --help-all`

4. Run a Jupyter notebook server via `jupyter notebook`

- If the notebook portion of Jupyter is not installed but Jupyter is,
you can install via `pip install notebook`

5. Create a notebook using "Apache Toree - Scala" from the dropdown under
new

- If you want other interpreter profiles than Scala, you can change the
interpreters via `jupyter toree install --interpreters=PySpark,SQL`

6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the
notebook

## Voting Instructions

The vote is open for at least 72 hours and passes if a majority of at least
3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Toree 0.1.0
[ ] -1 Do not release this package because ...

Re: Where do the staging artifacts go?

[Chip Senkbeil]

Great! Closing did indeed publish to staging (from what I can see)! Thanks
for all the help, Hitesh! I'll be opening another vote tomorrow.

On Mon, Jan 9, 2017 at 5:36 PM Hitesh Shah  wrote:

> If you look at https://repository.apache.org/#stagingRepositories, I see
> an
> open repo for toree.  You need to select it and hit "Close" I believe to
> publish to staging. On close, it will provide a maven repo url that someone
> else can use as a repo url for trying it out. After the final vote, the
> repo will need to be "released"
>
> thanks
> -- Hitesh
>
> On Fri, Jan 6, 2017 at 12:09 PM, Chip Senkbeil 
> wrote:
>
> > I just published 0.1.0 artifacts to staging in preparation for an RC4
> vote.
> > I cannot tell where they have gone to include a link in the description.
> >
> > The root of the staging deployment is here:
> > https://repository.apache.org/service/local/staging/deploy/
> > maven2/org/apache/toree/
> >
> > And I see successfully publish messages including "published
> > toree-kernel_2.10 to
> > https://repository.apache.org/service/local/staging/deploy/
> > maven2/org/apache/toree/toree-kernel_2.10/0.1.0-incubating/
> > toree-kernel_2.10-0.1.0-incubating.jar
> > "
> >
> > When I visit the root of the deployment, I see "Staging of Repository
> > within profile ID='1285d3cb661df6' is not yet started!". And I don't see
> > anything under the Nexus staging listing for Apache Toree under
> > https://repository.apache.org#view-repositories;staging~browsestorage
> >
> > Did I miss a step with getting the artifacts into staging? And, for
> > staging, should the artifacts have an RC4 appended to them? Or is it okay
> > for me to overwrite staging with a new 0.1.0 artifact until we have the
> > final RC?
> >
>

Where do the staging artifacts go?

[Chip Senkbeil]

I just published 0.1.0 artifacts to staging in preparation for an RC4 vote.
I cannot tell where they have gone to include a link in the description.

The root of the staging deployment is here:
https://repository.apache.org/service/local/staging/deploy/maven2/org/apache/toree/

And I see successfully publish messages including "published
toree-kernel_2.10 to
https://repository.apache.org/service/local/staging/deploy/maven2/org/apache/toree/toree-kernel_2.10/0.1.0-incubating/toree-kernel_2.10-0.1.0-incubating.jar
"

When I visit the root of the deployment, I see "Staging of Repository
within profile ID='1285d3cb661df6' is not yet started!". And I don't see
anything under the Nexus staging listing for Apache Toree under
https://repository.apache.org#view-repositories;staging~browsestorage

Did I miss a step with getting the artifacts into staging? And, for
staging, should the artifacts have an RC4 appended to them? Or is it okay
for me to overwrite staging with a new 0.1.0 artifact until we have the
final RC?

Re: [VOTE] Apache Toree 0.1.0 RC3

[Chip Senkbeil]

Closing this vote. Going to start a vote for an RC4 once I get all of the
artifacts in place on staging and dist.

On Fri, Dec 2, 2016 at 8:59 AM Gino Bustelo  wrote:

> Go for it Chip.
>
> On Thu, Dec 1, 2016 at 6:21 PM Luciano Resende 
> wrote:
>
> > I would check with Gino if he is ok with somebody else continuing this,
> but
> > if you know he is way or something, then it should be ok for you to
> > continue,
> >
> > On Tue, Nov 29, 2016 at 7:10 AM, Chip Senkbeil 
> > wrote:
> >
> > > Does Gino need to be the one to do this? Or can I get it started by
> > closing
> > > the vote thread and opening an approval request in general for the
> > release?
> > >
> > > On Fri, Nov 18, 2016 at 5:30 PM Luciano Resende 
> > > wrote:
> > >
> > > > I haven't seen an e-mail closing the release vote, and neither the
> IPMC
> > > > vote thread started. Are we still waiting for something specific
> here ?
> > > > Anything one of the mentors can help ?
> > > >
> > > > On Fri, Nov 11, 2016 at 7:31 PM, Luciano Resende <
> luckbr1...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > First you need to close the vote. Then the release needs to be
> > approved
> > > > in
> > > > > general, check the archives there for an example of the vote
> thread,
> > as
> > > > you
> > > > > need to add link to vote thread, cote result
> > > > >
> > > > > On Fri, Nov 11, 2016 at 7:03 PM Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> So, with three +1s, what do we need to do? Are there still things
> to
> > > > >> address? Does this need to go to general Apache? This has been
> open
> > > for
> > > > >> awhile.
> > > > >>
> > > > >> On Tue, Nov 8, 2016 at 2:31 PM Gino Bustelo 
> > > wrote:
> > > > >>
> > > > >> > +1
> > > > >> >
> > > > >> > We need to get this thing moving.
> > > > >> >
> > > > >> > On Fri, Nov 4, 2016 at 5:17 PM Hitesh Shah 
> > > wrote:
> > > > >> >
> > > > >> > > Sorry for the delay in reviewing the bits.
> > > > >> > >
> > > > >> > > Comments:
> > > > >> > >
> > > > >> > > KEYS file still missing.
> > > > >> > > Did signature/checksum checks for binary, source and pip
> > bundles.
> > > > >> > > For the binary release, the assembly jar makes it a bit tough
> to
> > > > >> validate
> > > > >> > > all dependencies. Not completed this fully though. Would be
> good
> > > if
> > > > >> other
> > > > >> > > folks who are reviewing can help.
> > > > >> > > For source release, there are a bunch of files without valid
> > > > headers.
> > > > >> > Also
> > > > >> > > some files like .example-image, etc are part of the release
> > > tarball
> > > > >> when
> > > > >> > > they should not be. ( used `find . -type f | xargs grep -L
> > > "Licensed
> > > > >> to
> > > > >> > the
> > > > >> > > Apache Software”` to find the ones in question). These should
> be
> > > > fixed
> > > > >> > > before the next release.
> > > > >> > > The License/Notice file for the source bundle needs to be
> fixed.
> > > > They
> > > > >> > > contain entries for items that are not part of the source
> > bundle.
> > > > >> > > PKG-INFO for the pip package may need to be fixed. The license
> > for
> > > > it
> > > > >> is
> > > > >> > > not only ALv2 but a combination of all the other bits bundled
> > into
> > > > the
> > > > >> > > assembly jar.
> > > > >> > >
> > > > >> > > License/Notice files:
> > > > >> > > There are a bunch of copyright lines in the LICENSE file
> without
> > > any
> > > > >> > > indic

Should we start another release vote?

[Chip Senkbeil]

This is long overdue, but since the artifacts weren't in staging at the
time of the vote for RC3, should we have another release vote?

Re: toree install issue - No module named pyspark

[Chip Senkbeil]

It's showing your PYTHONPATH as
/disk3/local/filecache/103/spark-assembly.jar. Toree is looking for pyspark
on your PYTHONPATH.

https://github.com/apache/incubator-toree/blob/master/pyspark-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala#L78

That code is showing us augmenting the existing PYTHONPATH to include
$SPARK_HOME/python/, where we are searching for your pyspark distribution.

Your PYTHONPATH isn't even showing us adding the $SPARK_HOME/python/, which
is also troubling.

On Wed, Dec 14, 2016 at 9:41 AM chris snow  wrote:

> I'm trying to setup toree as follows:
>
> CLUSTER_NAME=$(curl -s -k -u $BI_USER:$BI_PASS  -X GET
> https://${BI_HOST}:9443/api/v1/clusters
> | python -c 'import sys, json;
> print(json.load(sys.stdin)["items"][0]["Clusters"]["cluster_name"]);')
> echo Cluster Name: $CLUSTER_NAME
>
> CLUSTER_HOSTS=$(curl -s -k -u $BI_USER:$BI_PASS  -X GET
> https://${BI_HOST}:9443/api/v1/clusters/${CLUSTER_NAME}/hosts
> | python -c 'import sys, json; items = json.load(sys.stdin)["items"]; hosts
> = [ item["Hosts"]["host_name"] for item in items ]; print("
> ".join(hosts));')
> echo Cluster Hosts: $CLUSTER_HOSTS
>
> wget -c
> https://repo.continuum.io/archive/Anaconda2-4.1.1-Linux-x86_64.sh
>
> # Install anaconda if it isn't already installed
> [[ -d anaconda2 ]] || bash Anaconda2-4.1.1-Linux-x86_64.sh -b
>
> # check toree is available, if not install it
> ./anaconda2/bin/python -c 'import toree' || ./anaconda2/bin/pip install
> toree
>
> # Install toree
> ./anaconda2/bin/jupyter toree install \
> --spark_home=/usr/iop/current/spark-client/ \
> --user --interpreters Scala,PySpark,SparkR  \
> --spark_opts="--master yarn" \
> --python_exec=${HOME}/anaconda2/bin/python2.7
>
> # Install anaconda on all of the cluster nodes
> for CLUSTER_HOST in ${CLUSTER_HOSTS};
> do
>if [[ "$CLUSTER_HOST" != "$BI_HOST" ]];
>then
>   echo "*** Processing $CLUSTER_HOST ***"
>   ssh $BI_USER@$CLUSTER_HOST "wget -q -c
> https://repo.continuum.io/archive/Anaconda2-4.1.1-Linux-x86_64.sh";
>   ssh $BI_USER@$CLUSTER_HOST "[[ -d anaconda2 ]] || bash
> Anaconda2-4.1.1-Linux-x86_64.sh -b"
>
>   # You can install your pip modules on each node using something
> like this:
>   # ssh $BI_USER@$CLUSTER_HOST "${HOME}/anaconda2/bin/python -c
> 'import yourlibrary' || ${HOME}/anaconda2/pip install yourlibrary"
>fi
> done
>
> echo 'Finished installing'
>
> However, when I try to run a pyspark job I get the following error:
>
> Name: org.apache.toree.interpreter.broker.BrokerException
> Message: Py4JJavaError: An error occurred while calling
> z:org.apache.spark.api.python.PythonRDD.collectAndServe.
> : org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in
> stage 0.0 (TID 6, bi4c--data-3.bi.services.bluemix.net):
> org.apache.spark.SparkException:
> Error from python worker:
>   /home/biadmin/anaconda2/bin/python2.7: No module named pyspark
> PYTHONPATH was:
>   /disk3/local/filecache/103/spark-assembly.jar
> java.io.EOFException
>
> Any ideas what is going wrong?
>

Re: Evaluating Toree

[Chip Senkbeil]

If you want REST functionality, using something like Cloudera's Livy might
be more useful: https://github.com/cloudera/livy

Apache Toree is geared more towards dynamic applications that have a need
to run a variety of jobs against Spark that are configured on the fly such
as custom filtering and transformations that a LoB user might do in a
dashboard to get more insight into his or her data. Toree works well with
Jupyter notebooks and you can use it as the backend to the
https://github.com/jupyter/kernel_gateway project, exposing websocket
access to Apache Toree.

If your algorithms are static, you are not collaborating with others to
tweak those algorithms, and you just want a RESTful API, Apache Toree might
not be the best fit.

We have demonstrated in the past using Apache Toree (before it was brougnt
into Apache) to power RESTful servers, interactive applications, and other
Spark-related tools:
https://www.youtube.com/watch?v=2AX6g0tK-us

On Thu, Dec 15, 2016 at 10:44 AM Namit Kabra  wrote:

>
>
> Hi,
>
> Our team is creating a framework that would be a repository of Data
> Analysis algorithms. These algorithms should eventually be executed on
> Spark. So I am creating a REST API that would trigger these algorithms on
> Spark and get the results.
>
> There are several solutions that I am looking at, including...
>
>   Apache Spark Hidden REST API
>   Spark feature - Provide a stable application submission gateway in
>   standalone cluster mode
>
>
> While doing this I was exporing Toree. Do you think Toree will be a better
> option? If yes, how do I get started with creating a simplistic REST call
> to Spark using Toree.
>
>
> Thanks and Regards,
> - Namit Kabra
> 
> IBM India Software Labs,
> http://namitkabra.wordpress.com/
>

Re: Forbidden publishing to staging

[Chip Senkbeil]

Nothing from infrastructure or PredictionIO's PMC. Is there anyone else we
can turn to with regard to this problem? Can mentors raise the level of
priority somewhere in Apache? This is blocking me from closing the vote (or
opening another one if we need to for staging artifacts) and moving on with
the release.

On Fri, Dec 9, 2016 at 8:26 AM Chip Senkbeil 
wrote:

> Already created an issue -
> https://issues.apache.org/jira/browse/INFRA-13061 - so, hopefully I'll
> hear back soon. ;)
>
> Also, asked on the PredictionIO dev mailing list since it's a fairly
> popular Apache incubator project that is - from all appearances - using
> just sbt to publish jars. I asked around the same time as I opened that
> issue and haven't gotten a reply, so not going to hold my breath on that
> one.
>
> On Wed, Dec 7, 2016 at 5:50 PM Hitesh Shah  wrote:
>
> I think there is a way to publish jars manually (I believe the Ambari folks
> had done it once - not sure how though). You may wish to talk to the Infra
> folks via the hipchat channel and/or file an infra ticket to get guidance
> for both the questions.
>
> -- Hitesh
>
>
>
> On Wed, Dec 7, 2016 at 7:53 AM, Chip Senkbeil 
> wrote:
>
> > Should we have
> > https://repository.apache.org/content/repositories/staging/
> > org/apache/toree/
> > exist before publishing to staging? I don't know if that gets created the
> > first time we publish to staging, or if that is something we need to get
> > made through infrastructure or some other means.
> >
> > On Wed, Dec 7, 2016 at 9:52 AM Chip Senkbeil 
> > wrote:
> >
> > > Yep, verified that the signatures are generated each jar, its sources
> > jar,
> > > its pom, and its javadoc jar. All credentials are in place, gpg
> > passphrase
> > > is being used correctly to sign the jars, and I'm still getting
> forbidden
> > > errors such as the following:
> > >
> > > (toree-sql-interpreter/*:publishSigned) java.io.IOException: Access to
> > URL
> > > https://repository.apache.org/content/repositories/staging/
> > org/apache/toree/kernel/toree-sql-interpreter_2.10/0.1.0-
> > incubating/toree-sql-interpreter_2.10-0.1.0-incubating-sources.jar
> > > was refused by the server: Forbidden
> > >
> > >
> > > On Wed, Dec 7, 2016 at 7:27 AM Chip Senkbeil 
> > > wrote:
> > >
> > > I believe they are. We're using an sbt plugin that I've also used to
> > > publish signed jars to maven central. I'll try to inspect later today
> to
> > > make sure.
> > >
> > > Is there no way to upload signed jars by hand just to test my
> > permissions?
> > >
> > > On Tue, Dec 6, 2016, 9:25 PM Hitesh Shah  wrote:
> > >
> > > Are the jars being signed with your gpg key? For releases, unsigned
> jars
> > > are not allowed so that might be something to check too.
> > >
> > > -- Hitesh
> > >
> > >
> > >
> > > On Tue, Dec 6, 2016 at 7:01 PM, Chip Senkbeil  >
> > > wrote:
> > >
> > > > I've got my credentials configured, although we're using sbt instead
> of
> > > > Maven to publish our artifacts. Before I had my credentials added, I
> > was
> > > > getting another error about not having credentials; so, I'm pretty
> sure
> > > > they're being applied when publishing the jars. Is there a way to
> > > manually
> > > > add jars to staging through the Nexus UI just to see if my account
> has
> > > > permission?
> > > >
> > > > On Tue, Dec 6, 2016 at 6:44 PM Luciano Resende  >
> > > > wrote:
> > > >
> > > > > Looks like the staging profile is set for toree... have you
> > configured
> > > to
> > > > > pass your credentials when trying to deploy ? In maven I just
> > configure
> > > > the
> > > > > settings.xml with credentials and it works... see some details on
> > > > > http://maven.apache.org/developers/committer-settings.html
> > > > >
> > > > > On Tue, Dec 6, 2016 at 10:25 AM, Chip Senkbeil <
> > > chip.senkb...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Moved to dev.
> > > > > >
> > > > > > On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah 
> > > wrote:
> > > > > >
> > > > > > > Lets move this conversation to dev@.
> &

Re: Forbidden publishing to staging

[Chip Senkbeil]

Already created an issue - https://issues.apache.org/jira/browse/INFRA-13061
- so, hopefully I'll hear back soon. ;)

Also, asked on the PredictionIO dev mailing list since it's a fairly
popular Apache incubator project that is - from all appearances - using
just sbt to publish jars. I asked around the same time as I opened that
issue and haven't gotten a reply, so not going to hold my breath on that
one.

On Wed, Dec 7, 2016 at 5:50 PM Hitesh Shah  wrote:

> I think there is a way to publish jars manually (I believe the Ambari folks
> had done it once - not sure how though). You may wish to talk to the Infra
> folks via the hipchat channel and/or file an infra ticket to get guidance
> for both the questions.
>
> -- Hitesh
>
>
>
> On Wed, Dec 7, 2016 at 7:53 AM, Chip Senkbeil 
> wrote:
>
> > Should we have
> > https://repository.apache.org/content/repositories/staging/
> > org/apache/toree/
> > exist before publishing to staging? I don't know if that gets created the
> > first time we publish to staging, or if that is something we need to get
> > made through infrastructure or some other means.
> >
> > On Wed, Dec 7, 2016 at 9:52 AM Chip Senkbeil 
> > wrote:
> >
> > > Yep, verified that the signatures are generated each jar, its sources
> > jar,
> > > its pom, and its javadoc jar. All credentials are in place, gpg
> > passphrase
> > > is being used correctly to sign the jars, and I'm still getting
> forbidden
> > > errors such as the following:
> > >
> > > (toree-sql-interpreter/*:publishSigned) java.io.IOException: Access to
> > URL
> > > https://repository.apache.org/content/repositories/staging/
> > org/apache/toree/kernel/toree-sql-interpreter_2.10/0.1.0-
> > incubating/toree-sql-interpreter_2.10-0.1.0-incubating-sources.jar
> > > was refused by the server: Forbidden
> > >
> > >
> > > On Wed, Dec 7, 2016 at 7:27 AM Chip Senkbeil 
> > > wrote:
> > >
> > > I believe they are. We're using an sbt plugin that I've also used to
> > > publish signed jars to maven central. I'll try to inspect later today
> to
> > > make sure.
> > >
> > > Is there no way to upload signed jars by hand just to test my
> > permissions?
> > >
> > > On Tue, Dec 6, 2016, 9:25 PM Hitesh Shah  wrote:
> > >
> > > Are the jars being signed with your gpg key? For releases, unsigned
> jars
> > > are not allowed so that might be something to check too.
> > >
> > > -- Hitesh
> > >
> > >
> > >
> > > On Tue, Dec 6, 2016 at 7:01 PM, Chip Senkbeil  >
> > > wrote:
> > >
> > > > I've got my credentials configured, although we're using sbt instead
> of
> > > > Maven to publish our artifacts. Before I had my credentials added, I
> > was
> > > > getting another error about not having credentials; so, I'm pretty
> sure
> > > > they're being applied when publishing the jars. Is there a way to
> > > manually
> > > > add jars to staging through the Nexus UI just to see if my account
> has
> > > > permission?
> > > >
> > > > On Tue, Dec 6, 2016 at 6:44 PM Luciano Resende  >
> > > > wrote:
> > > >
> > > > > Looks like the staging profile is set for toree... have you
> > configured
> > > to
> > > > > pass your credentials when trying to deploy ? In maven I just
> > configure
> > > > the
> > > > > settings.xml with credentials and it works... see some details on
> > > > > http://maven.apache.org/developers/committer-settings.html
> > > > >
> > > > > On Tue, Dec 6, 2016 at 10:25 AM, Chip Senkbeil <
> > > chip.senkb...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Moved to dev.
> > > > > >
> > > > > > On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah 
> > > wrote:
> > > > > >
> > > > > > > Lets move this conversation to dev@.
> > > > > > >
> > > > > > > thanks
> > > > > > > -- Hitesh
> > > > > > >
> > > > > > > On Tue, Dec 6, 2016 at 8:52 AM, Chip Senkbeil <
> > > > chip.senkb...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Both Gino and I have g

Re: Forbidden publishing to staging

[Chip Senkbeil]

Should we have
https://repository.apache.org/content/repositories/staging/org/apache/toree/
exist before publishing to staging? I don't know if that gets created the
first time we publish to staging, or if that is something we need to get
made through infrastructure or some other means.

On Wed, Dec 7, 2016 at 9:52 AM Chip Senkbeil 
wrote:

> Yep, verified that the signatures are generated each jar, its sources jar,
> its pom, and its javadoc jar. All credentials are in place, gpg passphrase
> is being used correctly to sign the jars, and I'm still getting forbidden
> errors such as the following:
>
> (toree-sql-interpreter/*:publishSigned) java.io.IOException: Access to URL
> https://repository.apache.org/content/repositories/staging/org/apache/toree/kernel/toree-sql-interpreter_2.10/0.1.0-incubating/toree-sql-interpreter_2.10-0.1.0-incubating-sources.jar
> was refused by the server: Forbidden
>
>
> On Wed, Dec 7, 2016 at 7:27 AM Chip Senkbeil 
> wrote:
>
> I believe they are. We're using an sbt plugin that I've also used to
> publish signed jars to maven central. I'll try to inspect later today to
> make sure.
>
> Is there no way to upload signed jars by hand just to test my permissions?
>
> On Tue, Dec 6, 2016, 9:25 PM Hitesh Shah  wrote:
>
> Are the jars being signed with your gpg key? For releases, unsigned jars
> are not allowed so that might be something to check too.
>
> -- Hitesh
>
>
>
> On Tue, Dec 6, 2016 at 7:01 PM, Chip Senkbeil 
> wrote:
>
> > I've got my credentials configured, although we're using sbt instead of
> > Maven to publish our artifacts. Before I had my credentials added, I was
> > getting another error about not having credentials; so, I'm pretty sure
> > they're being applied when publishing the jars. Is there a way to
> manually
> > add jars to staging through the Nexus UI just to see if my account has
> > permission?
> >
> > On Tue, Dec 6, 2016 at 6:44 PM Luciano Resende 
> > wrote:
> >
> > > Looks like the staging profile is set for toree... have you configured
> to
> > > pass your credentials when trying to deploy ? In maven I just configure
> > the
> > > settings.xml with credentials and it works... see some details on
> > > http://maven.apache.org/developers/committer-settings.html
> > >
> > > On Tue, Dec 6, 2016 at 10:25 AM, Chip Senkbeil <
> chip.senkb...@gmail.com>
> > > wrote:
> > >
> > > > Moved to dev.
> > > >
> > > > On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah 
> wrote:
> > > >
> > > > > Lets move this conversation to dev@.
> > > > >
> > > > > thanks
> > > > > -- Hitesh
> > > > >
> > > > > On Tue, Dec 6, 2016 at 8:52 AM, Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > Both Gino and I have gotten forbidden errors when trying to publish
> > > Toree
> > > > > jars to staging. I'm able to login to nexus via
> > > > > https://repository.apache.org with my apache credentials just
> fine.
> > > Gino
> > > > > was able to publish to snapshot okay.
> > > > >
> > > > > I checked out an example of an Apache Spark release vote sent to
> > IPMC,
> > > > and
> > > > > their staging has an interesting name associated with it:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1006/
> > > > > (from
> > > > > https://lists.apache.org/api/source.lua/
> > d1e547d49a43f837c0ad13364ac82a
> > > > 4b84801b755c4a364b8ab42c6d@1390809763@%
> 3Cgeneral.incubator.apache.org
> > %3E
> > > > > )
> > > > >
> > > > > How do we get staging working? Do we need to go to infrastructure
> > > again?
> > > > > Is it necessary if we have the content under snapshots?
> > > > >
> > > > > I wanted to close the Toree dev vote thread and start the vote
> thread
> > > on
> > > > > IPMC.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>

Re: Forbidden publishing to staging

[Chip Senkbeil]

Yep, verified that the signatures are generated each jar, its sources jar,
its pom, and its javadoc jar. All credentials are in place, gpg passphrase
is being used correctly to sign the jars, and I'm still getting forbidden
errors such as the following:

(toree-sql-interpreter/*:publishSigned) java.io.IOException: Access to URL
https://repository.apache.org/content/repositories/staging/org/apache/toree/kernel/toree-sql-interpreter_2.10/0.1.0-incubating/toree-sql-interpreter_2.10-0.1.0-incubating-sources.jar
was refused by the server: Forbidden


On Wed, Dec 7, 2016 at 7:27 AM Chip Senkbeil 
wrote:

> I believe they are. We're using an sbt plugin that I've also used to
> publish signed jars to maven central. I'll try to inspect later today to
> make sure.
>
> Is there no way to upload signed jars by hand just to test my permissions?
>
> On Tue, Dec 6, 2016, 9:25 PM Hitesh Shah  wrote:
>
> Are the jars being signed with your gpg key? For releases, unsigned jars
> are not allowed so that might be something to check too.
>
> -- Hitesh
>
>
>
> On Tue, Dec 6, 2016 at 7:01 PM, Chip Senkbeil 
> wrote:
>
> > I've got my credentials configured, although we're using sbt instead of
> > Maven to publish our artifacts. Before I had my credentials added, I was
> > getting another error about not having credentials; so, I'm pretty sure
> > they're being applied when publishing the jars. Is there a way to
> manually
> > add jars to staging through the Nexus UI just to see if my account has
> > permission?
> >
> > On Tue, Dec 6, 2016 at 6:44 PM Luciano Resende 
> > wrote:
> >
> > > Looks like the staging profile is set for toree... have you configured
> to
> > > pass your credentials when trying to deploy ? In maven I just configure
> > the
> > > settings.xml with credentials and it works... see some details on
> > > http://maven.apache.org/developers/committer-settings.html
> > >
> > > On Tue, Dec 6, 2016 at 10:25 AM, Chip Senkbeil <
> chip.senkb...@gmail.com>
> > > wrote:
> > >
> > > > Moved to dev.
> > > >
> > > > On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah 
> wrote:
> > > >
> > > > > Lets move this conversation to dev@.
> > > > >
> > > > > thanks
> > > > > -- Hitesh
> > > > >
> > > > > On Tue, Dec 6, 2016 at 8:52 AM, Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > Both Gino and I have gotten forbidden errors when trying to publish
> > > Toree
> > > > > jars to staging. I'm able to login to nexus via
> > > > > https://repository.apache.org with my apache credentials just
> fine.
> > > Gino
> > > > > was able to publish to snapshot okay.
> > > > >
> > > > > I checked out an example of an Apache Spark release vote sent to
> > IPMC,
> > > > and
> > > > > their staging has an interesting name associated with it:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1006/
> > > > > (from
> > > > > https://lists.apache.org/api/source.lua/
> > d1e547d49a43f837c0ad13364ac82a
> > > > 4b84801b755c4a364b8ab42c6d@1390809763@%
> 3Cgeneral.incubator.apache.org
> > %3E
> > > > > )
> > > > >
> > > > > How do we get staging working? Do we need to go to infrastructure
> > > again?
> > > > > Is it necessary if we have the content under snapshots?
> > > > >
> > > > > I wanted to close the Toree dev vote thread and start the vote
> thread
> > > on
> > > > > IPMC.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>

Re: Forbidden publishing to staging

[Chip Senkbeil]

I believe they are. We're using an sbt plugin that I've also used to
publish signed jars to maven central. I'll try to inspect later today to
make sure.

Is there no way to upload signed jars by hand just to test my permissions?

On Tue, Dec 6, 2016, 9:25 PM Hitesh Shah  wrote:

> Are the jars being signed with your gpg key? For releases, unsigned jars
> are not allowed so that might be something to check too.
>
> -- Hitesh
>
>
>
> On Tue, Dec 6, 2016 at 7:01 PM, Chip Senkbeil 
> wrote:
>
> > I've got my credentials configured, although we're using sbt instead of
> > Maven to publish our artifacts. Before I had my credentials added, I was
> > getting another error about not having credentials; so, I'm pretty sure
> > they're being applied when publishing the jars. Is there a way to
> manually
> > add jars to staging through the Nexus UI just to see if my account has
> > permission?
> >
> > On Tue, Dec 6, 2016 at 6:44 PM Luciano Resende 
> > wrote:
> >
> > > Looks like the staging profile is set for toree... have you configured
> to
> > > pass your credentials when trying to deploy ? In maven I just configure
> > the
> > > settings.xml with credentials and it works... see some details on
> > > http://maven.apache.org/developers/committer-settings.html
> > >
> > > On Tue, Dec 6, 2016 at 10:25 AM, Chip Senkbeil <
> chip.senkb...@gmail.com>
> > > wrote:
> > >
> > > > Moved to dev.
> > > >
> > > > On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah 
> wrote:
> > > >
> > > > > Lets move this conversation to dev@.
> > > > >
> > > > > thanks
> > > > > -- Hitesh
> > > > >
> > > > > On Tue, Dec 6, 2016 at 8:52 AM, Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > Both Gino and I have gotten forbidden errors when trying to publish
> > > Toree
> > > > > jars to staging. I'm able to login to nexus via
> > > > > https://repository.apache.org with my apache credentials just
> fine.
> > > Gino
> > > > > was able to publish to snapshot okay.
> > > > >
> > > > > I checked out an example of an Apache Spark release vote sent to
> > IPMC,
> > > > and
> > > > > their staging has an interesting name associated with it:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1006/
> > > > > (from
> > > > > https://lists.apache.org/api/source.lua/
> > d1e547d49a43f837c0ad13364ac82a
> > > > 4b84801b755c4a364b8ab42c6d@1390809763@%
> 3Cgeneral.incubator.apache.org
> > %3E
> > > > > )
> > > > >
> > > > > How do we get staging working? Do we need to go to infrastructure
> > > again?
> > > > > Is it necessary if we have the content under snapshots?
> > > > >
> > > > > I wanted to close the Toree dev vote thread and start the vote
> thread
> > > on
> > > > > IPMC.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>

Re: Forbidden publishing to staging

[Chip Senkbeil]

I've got my credentials configured, although we're using sbt instead of
Maven to publish our artifacts. Before I had my credentials added, I was
getting another error about not having credentials; so, I'm pretty sure
they're being applied when publishing the jars. Is there a way to manually
add jars to staging through the Nexus UI just to see if my account has
permission?

On Tue, Dec 6, 2016 at 6:44 PM Luciano Resende  wrote:

> Looks like the staging profile is set for toree... have you configured to
> pass your credentials when trying to deploy ? In maven I just configure the
> settings.xml with credentials and it works... see some details on
> http://maven.apache.org/developers/committer-settings.html
>
> On Tue, Dec 6, 2016 at 10:25 AM, Chip Senkbeil 
> wrote:
>
> > Moved to dev.
> >
> > On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah  wrote:
> >
> > > Lets move this conversation to dev@.
> > >
> > > thanks
> > > -- Hitesh
> > >
> > > On Tue, Dec 6, 2016 at 8:52 AM, Chip Senkbeil  >
> > > wrote:
> > >
> > > Both Gino and I have gotten forbidden errors when trying to publish
> Toree
> > > jars to staging. I'm able to login to nexus via
> > > https://repository.apache.org with my apache credentials just fine.
> Gino
> > > was able to publish to snapshot okay.
> > >
> > > I checked out an example of an Apache Spark release vote sent to IPMC,
> > and
> > > their staging has an interesting name associated with it:
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1006/
> > > (from
> > > https://lists.apache.org/api/source.lua/d1e547d49a43f837c0ad13364ac82a
> > 4b84801b755c4a364b8ab42c6d@1390809763@%3Cgeneral.incubator.apache.org%3E
> > > )
> > >
> > > How do we get staging working? Do we need to go to infrastructure
> again?
> > > Is it necessary if we have the content under snapshots?
> > >
> > > I wanted to close the Toree dev vote thread and start the vote thread
> on
> > > IPMC.
> > >
> > >
> > >
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Forbidden publishing to staging

[Chip Senkbeil]

Moved to dev.

On Tue, Dec 6, 2016 at 12:23 PM Hitesh Shah  wrote:

> Lets move this conversation to dev@.
>
> thanks
> -- Hitesh
>
> On Tue, Dec 6, 2016 at 8:52 AM, Chip Senkbeil 
> wrote:
>
> Both Gino and I have gotten forbidden errors when trying to publish Toree
> jars to staging. I'm able to login to nexus via
> https://repository.apache.org with my apache credentials just fine. Gino
> was able to publish to snapshot okay.
>
> I checked out an example of an Apache Spark release vote sent to IPMC, and
> their staging has an interesting name associated with it:
> https://repository.apache.org/content/repositories/orgapachespark-1006/
> (from
> https://lists.apache.org/api/source.lua/d1e547d49a43f837c0ad13364ac82a4b84801b755c4a364b8ab42c6d@1390809763@%3Cgeneral.incubator.apache.org%3E
> )
>
> How do we get staging working? Do we need to go to infrastructure again?
> Is it necessary if we have the content under snapshots?
>
> I wanted to close the Toree dev vote thread and start the vote thread on
> IPMC.
>
>
>

Re: [VOTE] Apache Toree 0.1.0 RC3

[Chip Senkbeil]

Does Gino need to be the one to do this? Or can I get it started by closing
the vote thread and opening an approval request in general for the release?

On Fri, Nov 18, 2016 at 5:30 PM Luciano Resende 
wrote:

> I haven't seen an e-mail closing the release vote, and neither the IPMC
> vote thread started. Are we still waiting for something specific here ?
> Anything one of the mentors can help ?
>
> On Fri, Nov 11, 2016 at 7:31 PM, Luciano Resende 
> wrote:
>
> > First you need to close the vote. Then the release needs to be approved
> in
> > general, check the archives there for an example of the vote thread, as
> you
> > need to add link to vote thread, cote result
> >
> > On Fri, Nov 11, 2016 at 7:03 PM Chip Senkbeil 
> > wrote:
> >
> >> So, with three +1s, what do we need to do? Are there still things to
> >> address? Does this need to go to general Apache? This has been open for
> >> awhile.
> >>
> >> On Tue, Nov 8, 2016 at 2:31 PM Gino Bustelo  wrote:
> >>
> >> > +1
> >> >
> >> > We need to get this thing moving.
> >> >
> >> > On Fri, Nov 4, 2016 at 5:17 PM Hitesh Shah  wrote:
> >> >
> >> > > Sorry for the delay in reviewing the bits.
> >> > >
> >> > > Comments:
> >> > >
> >> > > KEYS file still missing.
> >> > > Did signature/checksum checks for binary, source and pip bundles.
> >> > > For the binary release, the assembly jar makes it a bit tough to
> >> validate
> >> > > all dependencies. Not completed this fully though. Would be good if
> >> other
> >> > > folks who are reviewing can help.
> >> > > For source release, there are a bunch of files without valid
> headers.
> >> > Also
> >> > > some files like .example-image, etc are part of the release tarball
> >> when
> >> > > they should not be. ( used `find . -type f | xargs grep -L "Licensed
> >> to
> >> > the
> >> > > Apache Software”` to find the ones in question). These should be
> fixed
> >> > > before the next release.
> >> > > The License/Notice file for the source bundle needs to be fixed.
> They
> >> > > contain entries for items that are not part of the source bundle.
> >> > > PKG-INFO for the pip package may need to be fixed. The license for
> it
> >> is
> >> > > not only ALv2 but a combination of all the other bits bundled into
> the
> >> > > assembly jar.
> >> > >
> >> > > License/Notice files:
> >> > > There are a bunch of copyright lines in the LICENSE file without any
> >> > > indication of why they are there. At the moment, it seems to
> indicate
> >> > they
> >> > > are for all of the toree codebase.
> >> > >
> >> > > +0 for this release.
> >> > >
> >> > > I think that there has been enough work done on this initial release
> >> that
> >> > > it will be useful to get some more eyes from the larger Incubator
> >> group
> >> > to
> >> > > provide more useful feedback.
> >> > >
> >> > > Please note that this vote does need atleast 3 PPMC +1s to pass.
> >> > >
> >> > > thanks
> >> > > — Hitesh
> >> > >
> >> > >
> >> > > > On Oct 27, 2016, at 7:34 AM, Gino Bustelo 
> >> wrote:
> >> > > >
> >> > > > Please vote to approve the release of the following candidate as
> >> > > > Apache Toree version 0.1.0. Pay special attention to the LICENSE
> and
> >> > > > NOTICE files since this is our first release.
> >> > > >
> >> > > >
> >> > > > The tag to be voted on is v0.1.0-rc3
> >> > > > (5bef39490daeda69d98a1e4cf4096103e80be44d)
> >> > > >
> >> > > > *
> >> > >
> >> > https://github.com/apache/incubator-toree/commit/
> >> 5bef39490daeda69d98a1e4cf4096103e80be44d
> >> > > > <
> >> > >
> >> > https://github.com/apache/incubator-toree/commit/
> >> 5bef39490daeda69d98a1e4cf4096103e80be44d
> >> > > >*
> >> > > >
> >> > > > All distribution packages, including signatures, digests, etc. can
> >> be
> >> > > found at:
> >> > > >
> >> > > > *
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/
> >> > > > <
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/
> >> >*
> >> > > >
> >> > > > The vote is open for at least 72 hours and passes if a majority of
> >> at
> >> > > least
> >> > > > 3 +1 PMC votes are cast.
> >> > > >
> >> > > > [ ] +1 Release this package as Apache Toree 0.1.0
> >> > > > [ ] -1 Do not release this package because ...
> >> > >
> >> > >
> >> >
> >>
> > --
> > Sent from my Mobile device
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [VOTE] Apache Toree 0.1.0 RC3

[Chip Senkbeil]

So, with three +1s, what do we need to do? Are there still things to
address? Does this need to go to general Apache? This has been open for
awhile.

On Tue, Nov 8, 2016 at 2:31 PM Gino Bustelo  wrote:

> +1
>
> We need to get this thing moving.
>
> On Fri, Nov 4, 2016 at 5:17 PM Hitesh Shah  wrote:
>
> > Sorry for the delay in reviewing the bits.
> >
> > Comments:
> >
> > KEYS file still missing.
> > Did signature/checksum checks for binary, source and pip bundles.
> > For the binary release, the assembly jar makes it a bit tough to validate
> > all dependencies. Not completed this fully though. Would be good if other
> > folks who are reviewing can help.
> > For source release, there are a bunch of files without valid headers.
> Also
> > some files like .example-image, etc are part of the release tarball when
> > they should not be. ( used `find . -type f | xargs grep -L "Licensed to
> the
> > Apache Software”` to find the ones in question). These should be fixed
> > before the next release.
> > The License/Notice file for the source bundle needs to be fixed. They
> > contain entries for items that are not part of the source bundle.
> > PKG-INFO for the pip package may need to be fixed. The license for it is
> > not only ALv2 but a combination of all the other bits bundled into the
> > assembly jar.
> >
> > License/Notice files:
> > There are a bunch of copyright lines in the LICENSE file without any
> > indication of why they are there. At the moment, it seems to indicate
> they
> > are for all of the toree codebase.
> >
> > +0 for this release.
> >
> > I think that there has been enough work done on this initial release that
> > it will be useful to get some more eyes from the larger Incubator group
> to
> > provide more useful feedback.
> >
> > Please note that this vote does need atleast 3 PPMC +1s to pass.
> >
> > thanks
> > — Hitesh
> >
> >
> > > On Oct 27, 2016, at 7:34 AM, Gino Bustelo  wrote:
> > >
> > > Please vote to approve the release of the following candidate as
> > > Apache Toree version 0.1.0. Pay special attention to the LICENSE and
> > > NOTICE files since this is our first release.
> > >
> > >
> > > The tag to be voted on is v0.1.0-rc3
> > > (5bef39490daeda69d98a1e4cf4096103e80be44d)
> > >
> > > *
> >
> https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> > > <
> >
> https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> > >*
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > found at:
> > >
> > > *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/
> > > *
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.1.0
> > > [ ] -1 Do not release this package because ...
> >
> >
>

Re: Share Object across interpreters

[Chip Senkbeil]

I just did that using the RC3 version of Toree for the 0.1.x branch. If
you're on master, maybe it doesn't require _jvm_kernel. I just saw that was
needed for our RC3.

On Wed, Nov 2, 2016 at 12:12 PM  wrote:

> That is not working for me in the release I have 0.1.0…
>
> %%pyspark
> print dir(kernel._jvm_kernel)
>
>
> ['__call__', '__class__', '__delattr__', '__dict__', '__doc__',
> '__format__', '__getattribute__', '__hash__', '__init__', '__module__',
> '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
> '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_get_args',
> 'command_header', 'container', 'converters', 'gateway_client', 'name',
> 'pool', 'target_id’]
>
>
> Any ideas?
>
> Thanks,
> Ian
>
>
>
>
>
>
>
> On 11/2/16, 12:38 PM, "Chip Senkbeil"  wrote:
>
> >When running Scala as the default interpreter in a notebook:
> >
> >Cell 1:
> >val x = 3
> >kernel.data.put("x", x)
> >
> >Cell2:
> >%%pyspark
> >x = kernel._jvm_kernel.data().get("x")
> >kernel._jvm_kernel.data().put("x", x + 1)
> >
> >Cell3:
> >println(kernel.data.get("x"))
> >
> >On Wed, Nov 2, 2016 at 11:27 AM  wrote:
> >
> >> Thanks Chip, now, I understand how to work with it from the JVM side.
> >>Any
> >> chance you have a snippet of how to get a value from the map in python?
> >>
> >> Ian
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 11/2/16, 11:39 AM, "Chip Senkbeil"  wrote:
> >>
> >> >While it isn't supported (we don't test its use in this case), you can
> >> >store objects in a shared hashmap under the kernel object that is made
> >> >available in each interpreter. The map is exposed as `kernel.data`, but
> >> >the
> >> >way you access and store data is different per language.
> >> >
> >> >The signature of the data map on the kernel is `val data:
> >>java.util.Map[
> >> >String, Any]` and we use a concurrent hashmap, so it can handle being
> >> >accessed from different threads.
> >> >
> >> >On Wed, Nov 2, 2016 at 10:28 AM  wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I¹m working primarily using the default scala/spark interpreter. It
> >> >>works
> >> >> great, except when I need to plot something. Is there a way I can
> >>take a
> >> >> scala object or spark data frame I¹ve created in a scala cell and
> >>pass
> >> >>it
> >> >> off to a pyspark cell for plotting?
> >> >>
> >> >> This documentation issue, might be related. I¹d be happy to try to
> >> >> document this once I know how :)
> >> >>
> >> >>
> >> >>
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_ji
> >>
> >>>>ra_browse_TOREE-2D286&d=DQIFaQ&c=nulvIAQnC0yOOjC0e0NVa8TOcyq9jNhjZ156R-
> >>>>JJ
> >>
> >>>>U10&r=CxpqDYMuQy-1uNI-UOyUbaX6BMPCZXH8d8evuCoP_OA&m=R6uBtgqaKfK_uE0gD6e
> >>>>Dj
> >>
> >>>>TZkYHrLjqtC0H66BkHvmVs&s=bRCGzWGCFGO54j-onzac_6v61jjY41QMDA1QQ_qLySQ&e=
> >> >>
> >> >> Thanks!
> >> >>
> >> >> Ian
> >> >>
> >> >>
> >>
> >>
>
>

Re: [VOTE] Apache Toree 0.1.0 RC3

[Chip Senkbeil]

+1.

- Manually installed using the pip package; verified that the install
worked and the Scala option was available in the notebook.
- Ran Scala code on the notebook.
- Ran some simple parallelization, reduces, etc. using a local Spark
cluster.
- Verified that PySpark interpreter worked correctly using same logic as
Scala.

On Wed, Nov 2, 2016 at 8:40 AM Gino Bustelo  wrote:

@hitesh Where would be an good place to put release instructions?

On Tue, Nov 1, 2016 at 6:21 PM Marius van Niekerk 
wrote:

>
>
> On 2016-10-28 08:49 (-0400), Marius van Niekerk <
> marius.v.niek...@gmail.com> wrote:
> > +1
> > On Thu, Oct 27, 2016 at 10:35 Gino Bustelo  wrote:
> >
> > > Please vote to approve the release of the following candidate as
> > > Apache Toree version 0.1.0. Pay special attention to the LICENSE and
> > > NOTICE files since this is our first release.
> > >
> > >
> > > The tag to be voted on is v0.1.0-rc3
> > > (5bef39490daeda69d98a1e4cf4096103e80be44d)
> > >
> > > *
> > >
>
https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> > > <
> > >
>
https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> > > >*
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > > found at:
> > >
> > > *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/
> > > *
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.1.0
> > > [ ] -1 Do not release this package because ...
> > >
> >
>
> Tested with
>
> make system-test
>
> Deployed to my data science team internally.
>

Re: Share Object across interpreters

[Chip Senkbeil]

When running Scala as the default interpreter in a notebook:

Cell 1:
val x = 3
kernel.data.put("x", x)

Cell2:
%%pyspark
x = kernel._jvm_kernel.data().get("x")
kernel._jvm_kernel.data().put("x", x + 1)

Cell3:
println(kernel.data.get("x"))

On Wed, Nov 2, 2016 at 11:27 AM  wrote:

> Thanks Chip, now, I understand how to work with it from the JVM side. Any
> chance you have a snippet of how to get a value from the map in python?
>
> Ian Maloney
> Platform Architect
> Advanced Analytics
> Internal: 828716
> Office: (734) 623-8716
> Mobile: (313) 910-9272
>
>
>
>
>
>
>
>
> On 11/2/16, 11:39 AM, "Chip Senkbeil"  wrote:
>
> >While it isn't supported (we don't test its use in this case), you can
> >store objects in a shared hashmap under the kernel object that is made
> >available in each interpreter. The map is exposed as `kernel.data`, but
> >the
> >way you access and store data is different per language.
> >
> >The signature of the data map on the kernel is `val data: java.util.Map[
> >String, Any]` and we use a concurrent hashmap, so it can handle being
> >accessed from different threads.
> >
> >On Wed, Nov 2, 2016 at 10:28 AM  wrote:
> >
> >> Hi,
> >>
> >> I¹m working primarily using the default scala/spark interpreter. It
> >>works
> >> great, except when I need to plot something. Is there a way I can take a
> >> scala object or spark data frame I¹ve created in a scala cell and pass
> >>it
> >> off to a pyspark cell for plotting?
> >>
> >> This documentation issue, might be related. I¹d be happy to try to
> >> document this once I know how :)
> >>
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_ji
> >>ra_browse_TOREE-2D286&d=DQIFaQ&c=nulvIAQnC0yOOjC0e0NVa8TOcyq9jNhjZ156R-JJ
> >>U10&r=CxpqDYMuQy-1uNI-UOyUbaX6BMPCZXH8d8evuCoP_OA&m=R6uBtgqaKfK_uE0gD6eDj
> >>TZkYHrLjqtC0H66BkHvmVs&s=bRCGzWGCFGO54j-onzac_6v61jjY41QMDA1QQ_qLySQ&e=
> >>
> >> Thanks!
> >>
> >> Ian
> >>
> >>
>
>

Re: Share Object across interpreters

[Chip Senkbeil]

While it isn't supported (we don't test its use in this case), you can
store objects in a shared hashmap under the kernel object that is made
available in each interpreter. The map is exposed as `kernel.data`, but the
way you access and store data is different per language.

The signature of the data map on the kernel is `val data: java.util.Map[
String, Any]` and we use a concurrent hashmap, so it can handle being
accessed from different threads.

On Wed, Nov 2, 2016 at 10:28 AM  wrote:

> Hi,
>
> I’m working primarily using the default scala/spark interpreter. It works
> great, except when I need to plot something. Is there a way I can take a
> scala object or spark data frame I’ve created in a scala cell and pass it
> off to a pyspark cell for plotting?
>
> This documentation issue, might be related. I’d be happy to try to
> document this once I know how :)
>
> https://issues.apache.org/jira/browse/TOREE-286
>
> Thanks!
>
> Ian
>
>

Re: [VOTE] Apache Toree 0.1.0 RC2

[Chip Senkbeil]

+1

On Wed, Oct 12, 2016 at 12:47 PM Marius van Niekerk <
marius.v.niek...@gmail.com> wrote:

> +1
>
> On Tue, 11 Oct 2016 at 16:22 Gino Bustelo  wrote:
>
> > Tagged as v0.1.0-rc2 and points to the commit. I was trying to avoid
> overly
> > tagging and reserving them for actual releases. Instead... I was placing
> a
> > commit to mark the RC.
> >
> > On Tue, Oct 11, 2016 at 3:14 PM Luciano Resende 
> > wrote:
> >
> > > On Tue, Oct 11, 2016 at 12:16 PM, Gino Bustelo 
> > wrote:
> > >
> > > > Please vote to approve the release of the following candidate as
> > > > Apache Toree version 0.1.0
> > > >
> > > >
> > > > The commit to be voted on is 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> > > >  > > > 119bf3e2d1d16986f55802cf2323e8629ea25ef8>
> > > >
> > > > https://github.com/apache/incubator-toree/tree/
> > > > 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> > > >  > > > 9e478555f7>
> > > >
> > > > All distribution packages, including signatures, digests, etc. can be
> > > > found at:
> > > >
> > > > *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc2/
> > > > *
> > > >
> > > > The vote is open for at least 72 hours and passes if a majority of at
> > > least
> > > > 3 +1 PMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Toree 0.1.0
> > > > [ ] -1 Do not release this package because ...
> > > >
> > >
> > >
> > > Note there should be a tag associated with the vote. I am assuming that
> > the
> > > tag will be based on 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>

[jira] [Commented] (TOREE-345) Spark 2.0.0 Fails when trying to use spark-avro

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552803#comment-15552803
 ] 

Chip Senkbeil commented on TOREE-345:
-

Yeah, that is what the Ivy files generated in the resource directories were 
for. In fact, we're still supposedly loading up the base dependencies and 
excluding them here: 
https://github.com/apache/incubator-toree/blob/a0f75a1c0e5c66d2d5787a78668d31d44ffb1be0/kernel-api/src/main/scala/org/apache/toree/dependencies/CoursierDependencyDownloader.scala#L82

Either the Ivy files are not being loaded 
(https://github.com/apache/incubator-toree/blob/a0f75a1c0e5c66d2d5787a78668d31d44ffb1be0/kernel-api/src/main/scala/org/apache/toree/dependencies/CoursierDependencyDownloader.scala#L286)
 or they are not being generated with up-to-date info. Or Coursier is not 
excluding properly.

> Spark 2.0.0 Fails when trying to use spark-avro
> ---
>
> Key: TOREE-345
> URL: https://issues.apache.org/jira/browse/TOREE-345
> Project: TOREE
>  Issue Type: Bug
> Environment: Spark 2.0.0, Scala 2.11, Hadoop 2.7, Toree Git SHA: 
> 7c1bfb6df7130477c558e69bbb518b0af364e06a
>Reporter: Hollin Wilkins
>
> When trying to use the spark-avro project to load Avro files from Jupyter, we 
> get errors.
> First:
> {code}
> %AddDeps com.databricks spark-avro_2.11 3.0.1 --transitive --trace
> {code}
> Then try to load an Avro file and show it:
> {code}
> spark.sqlContext.read.format("com.databricks.spark.avro").load("/tmp/test.avro").show()
> {code}
> And we get an error. I will attach the trace as a file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Discussion regarding our first release vote

[Chip Senkbeil]

Sure.

On Wed, Oct 5, 2016 at 12:46 AM Luciano Resende 
wrote:

> Can we move this discussion to public ?
>
> On Tue, Oct 4, 2016 at 10:27 PM, Chip Senkbeil 
> wrote:
>
> Was starting to read through this link:
> http://incubator.apache.org/guides/releasemanagement.html#best-practice-incubator-release-vote
>
> A lot of the content has "TODO: Blah blah link" in place of actual links
> to content, so was hoping to get a rough outline from a mentor, if
> possible. Since we're wanting to release for two different versions of
> Toree (one supporting Spark 1.x and the other for 2.x), I'm assuming we
> need two separate vote threads with artifacts for each, right?
>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

[jira] [Resolved] (TOREE-317) Need system test for validating loading and using external plugins

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-317.
-
Resolution: Fixed

> Need system test for validating loading and using external plugins
> --
>
> Key: TOREE-317
> URL: https://issues.apache.org/jira/browse/TOREE-317
> Project: TOREE
>  Issue Type: Test
>Affects Versions: 0.1.0
>Reporter: Chip Senkbeil
>Priority: Minor
>
> Related to https://issues.apache.org/jira/browse/TOREE-314.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (TOREE-317) Need system test for validating loading and using external plugins

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil closed TOREE-317.
---

> Need system test for validating loading and using external plugins
> --
>
> Key: TOREE-317
> URL: https://issues.apache.org/jira/browse/TOREE-317
> Project: TOREE
>  Issue Type: Test
>Affects Versions: 0.1.0
>Reporter: Chip Senkbeil
>Priority: Minor
>
> Related to https://issues.apache.org/jira/browse/TOREE-314.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (TOREE-342) Switch tab completer to the default used for scala 2.11 console

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil closed TOREE-342.
---

> Switch tab completer to the default used for scala 2.11 console
> ---
>
> Key: TOREE-342
> URL: https://issues.apache.org/jira/browse/TOREE-342
> Project: TOREE
>  Issue Type: Improvement
> Environment: Scala 2.11
>Reporter: Marius Van Niekerk
>Priority: Minor
>
> Scala 2.11 adds a new tab completion engine (PresentationCompilerCompleter) 
> which should allow much richer tab completion for Toree



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TOREE-343) Added support for jupyter message `is_complete_request`

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-343.
-
Resolution: Fixed

> Added support for jupyter message `is_complete_request`
> ---
>
> Key: TOREE-343
> URL: https://issues.apache.org/jira/browse/TOREE-343
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Marius Van Niekerk
>
> According to the spec here we need to implement support for 
> is_complete_request in order to get some jupyter clients working properly 
> (notably jupyter console)
> https://github.com/jupyter/jupyter_client/blob/4.4.0/docs/messaging.rst#code-completeness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-343) Added support for jupyter message `is_complete_request`

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526840#comment-15526840
 ] 

Chip Senkbeil commented on TOREE-343:
-

Tackled by [~mariusvniekerk].

> Added support for jupyter message `is_complete_request`
> ---
>
> Key: TOREE-343
> URL: https://issues.apache.org/jira/browse/TOREE-343
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Marius Van Niekerk
>
> According to the spec here we need to implement support for 
> is_complete_request in order to get some jupyter clients working properly 
> (notably jupyter console)
> https://github.com/jupyter/jupyter_client/blob/4.4.0/docs/messaging.rst#code-completeness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (TOREE-343) Added support for jupyter message `is_complete_request`

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil closed TOREE-343.
---

> Added support for jupyter message `is_complete_request`
> ---
>
> Key: TOREE-343
> URL: https://issues.apache.org/jira/browse/TOREE-343
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Marius Van Niekerk
>
> According to the spec here we need to implement support for 
> is_complete_request in order to get some jupyter clients working properly 
> (notably jupyter console)
> https://github.com/jupyter/jupyter_client/blob/4.4.0/docs/messaging.rst#code-completeness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TOREE-342) Switch tab completer to the default used for scala 2.11 console

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-342.
-
Resolution: Fixed

> Switch tab completer to the default used for scala 2.11 console
> ---
>
> Key: TOREE-342
> URL: https://issues.apache.org/jira/browse/TOREE-342
> Project: TOREE
>  Issue Type: Improvement
> Environment: Scala 2.11
>Reporter: Marius Van Niekerk
>Priority: Minor
>
> Scala 2.11 adds a new tab completion engine (PresentationCompilerCompleter) 
> which should allow much richer tab completion for Toree



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-342) Switch tab completer to the default used for scala 2.11 console

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526839#comment-15526839
 ] 

Chip Senkbeil commented on TOREE-342:
-

Tackled by [~mariusvniekerk].

> Switch tab completer to the default used for scala 2.11 console
> ---
>
> Key: TOREE-342
> URL: https://issues.apache.org/jira/browse/TOREE-342
> Project: TOREE
>  Issue Type: Improvement
> Environment: Scala 2.11
>Reporter: Marius Van Niekerk
>Priority: Minor
>
> Scala 2.11 adds a new tab completion engine (PresentationCompilerCompleter) 
> which should allow much richer tab completion for Toree



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TOREE-341) Toree scala kernel doesn't handle exceptions correctly

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-341.
-
Resolution: Fixed

> Toree scala kernel doesn't handle exceptions correctly
> --
>
> Key: TOREE-341
> URL: https://issues.apache.org/jira/browse/TOREE-341
> Project: TOREE
>  Issue Type: Bug
>Reporter: Marius Van Niekerk
>
> Currently in master (2016-09-20) toree will swallow exceptions instead of 
> doing the right thing and returning them to the notebook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (TOREE-341) Toree scala kernel doesn't handle exceptions correctly

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil closed TOREE-341.
---

> Toree scala kernel doesn't handle exceptions correctly
> --
>
> Key: TOREE-341
> URL: https://issues.apache.org/jira/browse/TOREE-341
> Project: TOREE
>  Issue Type: Bug
>Reporter: Marius Van Niekerk
>
> Currently in master (2016-09-20) toree will swallow exceptions instead of 
> doing the right thing and returning them to the notebook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-341) Toree scala kernel doesn't handle exceptions correctly

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526355#comment-15526355
 ] 

Chip Senkbeil commented on TOREE-341:
-

Resolved by [~mariusvniekerk]. Can't seem to add him as the assignee, though.

> Toree scala kernel doesn't handle exceptions correctly
> --
>
> Key: TOREE-341
> URL: https://issues.apache.org/jira/browse/TOREE-341
> Project: TOREE
>  Issue Type: Bug
>Reporter: Marius Van Niekerk
>
> Currently in master (2016-09-20) toree will swallow exceptions instead of 
> doing the right thing and returning them to the notebook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-336) Toree not working with Apache Spark 2.0.0

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461425#comment-15461425
 ] 

Chip Senkbeil commented on TOREE-336:
-

There is a PR for this: https://github.com/apache/incubator-toree/pull/56

Until that gets merged, Toree only supports scala 2.10 and spark 1.6+ but not 
2.x.

> Toree not working with Apache Spark 2.0.0
> -
>
> Key: TOREE-336
> URL: https://issues.apache.org/jira/browse/TOREE-336
> Project: TOREE
>  Issue Type: Bug
> Environment: OSX and ubuntu-14.04, both running scala 2.10.4 and 
> spark 2.0.0
>Reporter: Tianhui Li
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Following the instructions on 
> https://github.com/apache/incubator-toree/blob/master/README.md, I run
> ```
> pip install --pre toree
> jupyter toree install --spark-home=$SPARK_HOME
> ```
> I'm able to build fine.  But upon starting the server and a new scala (or any 
> other type of notebook), I an error (provided below).  This seems related to 
> using scala 2.10 rather than 2.11 (see 
> http://stackoverflow.com/questions/29339005/run-main-0-java-lang-nosuchmethoderror-scala-collection-immutable-hashset-emp
>  and 
> http://stackoverflow.com/questions/30536759/running-a-spark-application-in-intellij-14-1-3).
>   Below is the error:
> $ jupyter notebook
> [I 12:11:59.464 NotebookApp] Serving notebooks from local directory: 
> /Users/tianhui
> [I 12:11:59.464 NotebookApp] 0 active kernels 
> [I 12:11:59.465 NotebookApp] The Jupyter Notebook is running at: 
> http://localhost:/
> [I 12:11:59.465 NotebookApp] Use Control-C to stop this server and shut down 
> all kernels (twice to skip confirmation).
> [I 12:12:06.847 NotebookApp] 302 GET / (::1) 0.47ms
> [I 12:12:10.591 NotebookApp] Creating new notebook in 
> [I 12:12:11.600 NotebookApp] Kernel started: 
> 20ca2e71-781b-4208-ad88-bc04c1ca37d6
> Starting Spark Kernel with 
> SPARK_HOME=/usr/local/Cellar/apache-spark/2.0.0/libexec/
> 16/09/03 12:12:12 [INFO] o.a.t.Main$$anon$1 - Kernel version: 
> 0.1.0.dev9-incubating-SNAPSHOT
> 16/09/03 12:12:12 [INFO] o.a.t.Main$$anon$1 - Scala version: Some(2.10.4)
> 16/09/03 12:12:12 [INFO] o.a.t.Main$$anon$1 - ZeroMQ (JeroMQ) version: 3.2.2
> 16/09/03 12:12:12 [INFO] o.a.t.Main$$anon$1 - Initializing internal actor 
> system
> Exception in thread "main" java.lang.NoSuchMethodError: 
> scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;
>   at akka.actor.ActorCell$.(ActorCell.scala:336)
>   at akka.actor.ActorCell$.(ActorCell.scala)
>   at akka.actor.RootActorPath.$div(ActorPath.scala:185)
>   at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:465)
>   at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:453)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
>   at scala.util.Try$.apply(Try.scala:192)
>   at 
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
>   at 
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>   at 
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>   at scala.util.Success.flatMap(Try.scala:231)
>   at 
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
>   at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585)
>   at akka.actor.ActorSystemImpl.(ActorSystem.scala:578)
>   at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
>   at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)
>   at 
> org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71)
>   at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35)
>   at 
> org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60)
>   at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35)
>   at 
> org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:70)
>   at org.apache.toree.Main$delayedInit$body.ap

Re: Contributing to Toree

[Chip Senkbeil]

To get involved with the project, all you need to do is pick up a JIRA.

Here's the list of open JIRAs for 0.1.0:
https://issues.apache.org/jira/browse/TOREE-299?jql=project%20%3D%20TOREE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%200.1.0%20ORDER%20BY%20priority%20DESC

Here's a list of all of our JIRAs:
https://issues.apache.org/jira/browse/TOREE-299?jql=project%20%3D%20TOREE%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20fixVersion%20ASC%2C%20priority%20DESC

We have others working on PRs such as adding Spark 2.0 support, Scala 2.11
support, etc. at https://github.com/apache/incubator-toree/pulls

Essentially, we're open to any contribution. That also includes
documentation, which can be found at
https://github.com/apache/incubator-toree-website

If there's something you would like to do or try, just mention it on the
mailing list and we can provide advice, where to look, etc.

On Thu, Jul 21, 2016 at 12:13 AM Ryan Brown  wrote:

> All,
>
> I was on the site and cant navigate to a "contributing" page, as indicated.
> Is there a proper way to become involved with this project?
>
> I ask as an IBMer. We are developing an application that will submit
> dynamic spark jobs. We would rather build for continuous deployment, robust
> testing, etc. That often raises the challenges that you enumerate: jars on
> the cluster, system constraints, etc.
>
> Let me know how I can help. This project seems like something I would like
> to work on, both for our project and the community!
>
> R
>
> --
> *Ryan **Brown *
> Sr. Software Engineer
> ryan.br...@weather.com
>

[jira] [Commented] (TOREE-314) External plugins using indirect traits not detected

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300108#comment-15300108
 ] 

Chip Senkbeil commented on TOREE-314:
-

Oh, forgot to close this one. It's already been fixed by 
https://github.com/apache/incubator-toree/pull/41.

> External plugins using indirect traits not detected
> ---
>
> Key: TOREE-314
> URL: https://issues.apache.org/jira/browse/TOREE-314
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Chip Senkbeil
>Assignee: Chip Senkbeil
>
> When loading external plugins, any class that indirectly inherits plugin 
> (such as throw LineMagic/CellMagic) is not detected. This is because we 
> recursively check all interfaces/superclasses using the class information 
> from the current jar rather than the current jar and any internal classes to 
> the kernel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-316) Generate Scala file from Jupyter Notebook

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300106#comment-15300106
 ] 

Chip Senkbeil commented on TOREE-316:
-

[~lbustelo], is this to be embedded in the kernel.json? Or is this part of the 
kernel info request message that is made when first connecting to the kernel? 
Either way, can't we just have hard-coded metadata for whatever the default 
interpreter is? If it's Scala, return that language info. Do the same for other 
languages we support based on the specified default interpreter (either when 
installing kernel.json files or when getting the kernel info request). If it 
needs to be in a message sent by the kernel, we can have a plugin per language 
that provides that information and just ask for the kernel information from 
whatever the default is? I'm doing something similar when converting 
interpreters to plugins.

> Generate Scala file from Jupyter Notebook
> -
>
> Key: TOREE-316
> URL: https://issues.apache.org/jira/browse/TOREE-316
> Project: TOREE
>  Issue Type: New Feature
> Environment: Toree 0.1.0.dev7
> Jupyter Notebook 4.2
> Scala 2.10
> Python 3.5.1
>Reporter: Leonardo Noleto
>Priority: Minor
> Fix For: 0.1.0
>
>
> Hello all,
> I am unable to generate a Scala file from a Jupyter Notebook with the good 
> file extension (.scala) through NbConvert:
> {noformat}
> jupyter nbconvert --to script MyScalaNotebook.ipynb
> {noformat}
> It's not a big deal as it generates the code but with .txt extension.
> The fix seems quite easy (the notebook metadata is not full jupyter 
> compliant).
> Here goes an example of metadata created by Toree: 
> {code:javascript}
> {
>   "kernelspec": {
> "name": "scala_no_sparkcontext_scala",
> "display_name": "Scala with SparkContext - Scala",
> "language": "scala"
>   },
>   "language_info": {
> "name": "scala"
>   }
> }
> {code}
> Now, for a modified version enabling good generation (with .scala extension):
> {code:javascript}
> {
>   "kernelspec": {
> "name": "scala_no_sparkcontext_scala",
> "display_name": "Scala with SparkContext - Scala",
> "language": "scala"
>   },
>   "language_info": {
> "name": "scala",
>"file_extension": ".scala"
>   }
> }
> {code}
> The modified code solves the "Download as" with the good extension.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (TOREE-317) Need system test for validating loading and using external plugins

[Chip Senkbeil (JIRA)]

Chip Senkbeil created TOREE-317:
---

 Summary: Need system test for validating loading and using 
external plugins
 Key: TOREE-317
 URL: https://issues.apache.org/jira/browse/TOREE-317
 Project: TOREE
  Issue Type: Test
Affects Versions: 0.1.0
Reporter: Chip Senkbeil
Priority: Minor


Related to https://issues.apache.org/jira/browse/TOREE-314.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (TOREE-314) External plugins using indirect traits not detected

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil reassigned TOREE-314:
---

Assignee: Chip Senkbeil

> External plugins using indirect traits not detected
> ---
>
> Key: TOREE-314
> URL: https://issues.apache.org/jira/browse/TOREE-314
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Chip Senkbeil
>    Assignee: Chip Senkbeil
>
> When loading external plugins, any class that indirectly inherits plugin 
> (such as throw LineMagic/CellMagic) is not detected. This is because we 
> recursively check all interfaces/superclasses using the class information 
> from the current jar rather than the current jar and any internal classes to 
> the kernel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (TOREE-314) External plugins using indirect traits not detected

[Chip Senkbeil (JIRA)]

Chip Senkbeil created TOREE-314:
---

 Summary: External plugins using indirect traits not detected
 Key: TOREE-314
 URL: https://issues.apache.org/jira/browse/TOREE-314
 Project: TOREE
  Issue Type: Bug
Affects Versions: 0.1.0
Reporter: Chip Senkbeil


When loading external plugins, any class that indirectly inherits plugin (such 
as throw LineMagic/CellMagic) is not detected. This is because we recursively 
check all interfaces/superclasses using the class information from the current 
jar rather than the current jar and any internal classes to the kernel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-311) commentation causes abiguos error

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280107#comment-15280107
 ] 

Chip Senkbeil commented on TOREE-311:
-

Looks like a discussion has started on a PR for this issue here: 
https://github.com/apache/incubator-toree/pull/40

> commentation causes abiguos error
> -
>
> Key: TOREE-311
> URL: https://issues.apache.org/jira/browse/TOREE-311
> Project: TOREE
>  Issue Type: Bug
> Environment: Ubuntu/Spark 1.6.0
>Reporter: Marshall Markham
> Attachments: comment_mayhem.JPG
>
>
> Adding a double forward slash comment as the last line of a cell causes an 
> error with no message or stack trace. See attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-311) commentation causes abiguos error

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278548#comment-15278548
 ] 

Chip Senkbeil commented on TOREE-311:
-

[~mmarkham], I definitely agree that it should be fixed to avoid confusion. 
We're always open for PRs and can help guide new contributors through the 
codebase if needed.

> commentation causes abiguos error
> -
>
> Key: TOREE-311
> URL: https://issues.apache.org/jira/browse/TOREE-311
> Project: TOREE
>  Issue Type: Bug
> Environment: Ubuntu/Spark 1.6.0
>Reporter: Marshall Markham
> Attachments: comment_mayhem.JPG
>
>
> Adding a double forward slash comment as the last line of a cell causes an 
> error with no message or stack trace. See attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-311) commentation causes abiguos error

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278508#comment-15278508
 ] 

Chip Senkbeil commented on TOREE-311:
-

Despite the error, is the variable you are declaring available in subsequent 
cells?

There is an known (not documented) issue with comments. We break up code cells 
and evaluate each line of code on its own. Blocks of code get grouped together 
(including method declarations, etc).

E.g.
{code}
val x = 3
val z = {
  val y = 4
  x + y
}
println(x + z)
{code}

results in the following being evaluated
{code}val x = 3{code}
{code}
val z = {
  val y = 4
  x + y
}
{code}
{code}println(x + z){code}

So, if you have a comment on its own line, it gets an individual evaluation. As 
we try to compile and execute each line, the compiler does not like empty or 
comment-only code. We have a check for empty code (and ignore it), but we don't 
filter out code that is purely comments.

Ideally, we should have some sort of filter to remove comments when compiling 
and executing.


> commentation causes abiguos error
> -
>
> Key: TOREE-311
> URL: https://issues.apache.org/jira/browse/TOREE-311
> Project: TOREE
>  Issue Type: Bug
> Environment: Ubuntu/Spark 1.6.0
>Reporter: Marshall Markham
> Attachments: comment_mayhem.JPG
>
>
> Adding a double forward slash comment as the last line of a cell causes an 
> error with no message or stack trace. See attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TOREE-310) Allow override of python executable used for pyspark

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-310.
-
   Resolution: Fixed
Fix Version/s: 0.1.0

Fixed by PR #32.

> Allow override of python executable used for pyspark
> 
>
> Key: TOREE-310
> URL: https://issues.apache.org/jira/browse/TOREE-310
> Project: TOREE
>  Issue Type: Improvement
>Reporter: Eric Chang
> Fix For: 0.1.0
>
>
> We're using virtualenvs when running pyspark, and it would be great to be 
> able to use a virtualenv as the python executable used by the Spark Driver 
> (i.e. {{--master yarn-client}}).  This value is currently hard-coded to 
> {{python}} in 
> org/apache/toree/kernel/interpreter/pyspark/PySparkProcess.scala.
> I have a branch on my repo which adds an optional kernel parameter 
> PYTHON_EXEC:
> {code}
> ...
> "SPARK_HOME": "/usr/lib/spark",
> "PYTHON_EXEC" : "/usr/local/python/virtualenvs/myvenv/bin/python",
> ...
> {code}
> If {{PYTHON_EXEC}} is unspecified, the default of {{python}} is used.
> Here's the diff of the branch, please let me know if it's ok for me to issue 
> a PR against the main repo: 
> https://github.com/ericchang/incubator-toree/compare/ericchang:master...custom-python-exec



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TOREE-309) pyspark_runner needs to be upgraded for Spark 1.5, 1.6

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-309.
-
   Resolution: Fixed
Fix Version/s: 0.1.0

Fixed by PR #31.

> pyspark_runner needs to be upgraded for Spark 1.5, 1.6
> --
>
> Key: TOREE-309
> URL: https://issues.apache.org/jira/browse/TOREE-309
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: Eric Chang
> Fix For: 0.1.0
>
>
> pyspark_runner.py includes version-specific switches for Spark 1.4 that 
> should be applied to spark 1.5/1.6:
> {code}
> if sparkVersion.startswith("1.4"):
>   gateway = JavaGateway(client, auto_convert = True)
> else:
>   gateway = JavaGateway(client)
> ...
> elif sparkVersion.startswith("1.4"):
>   java_import(gateway.jvm, "org.apache.spark.sql.*")
>   java_import(gateway.jvm, "org.apache.spark.sql.hive.*")
> {code}
> I have a branch on my fork that extends the 1.4 logic to 1.5, 1.6.  Not sure 
> if packaging is going to change significantly in 2.x so I didn't extend 
> beyond 1.6: 
> https://github.com/apache/incubator-toree/compare/master...ericchang:pyspark-runner-1.5-fix
> Let me know if it's ok for me to issue the PR against the main repo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (TOREE-272) Jupyter Notebook works; Console is broken / hangs

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil resolved TOREE-272.
-
Resolution: Fixed

Fixed hanging in the mentioned PR.

> Jupyter Notebook works; Console is broken / hangs
> -
>
> Key: TOREE-272
> URL: https://issues.apache.org/jira/browse/TOREE-272
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: Toree 0.1.5-SNAPSHOT commit 042debc2bd0c
> Jupyter 4.1.0 installed via pip, Spark 1.5.2 installed via Homebrew
>Reporter: Seth Bromberger
>Assignee: Chip Senkbeil
>Priority: Blocker
> Fix For: 0.1.0
>
>
> Jupyter notebook will successfully use Toree as a kernel, but Jupyter console 
> hangs on the first evaluation of code. Example logs / sessions are here:
> https://gist.github.com/sbromberger/8a43f79eee08f4268515
> https://gist.github.com/sbromberger/1d6fdad5f73fe0198ef4
> Without Console, integration with Atom editor via Hydrogen is broken. Plus, 
> it's nice to have a REPL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (TOREE-272) Jupyter Notebook works; Console is broken / hangs

[Chip Senkbeil (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chip Senkbeil reassigned TOREE-272:
---

Assignee: Chip Senkbeil

> Jupyter Notebook works; Console is broken / hangs
> -
>
> Key: TOREE-272
> URL: https://issues.apache.org/jira/browse/TOREE-272
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: Toree 0.1.5-SNAPSHOT commit 042debc2bd0c
> Jupyter 4.1.0 installed via pip, Spark 1.5.2 installed via Homebrew
>Reporter: Seth Bromberger
>Assignee: Chip Senkbeil
>Priority: Blocker
> Fix For: 0.1.0
>
>
> Jupyter notebook will successfully use Toree as a kernel, but Jupyter console 
> hangs on the first evaluation of code. Example logs / sessions are here:
> https://gist.github.com/sbromberger/8a43f79eee08f4268515
> https://gist.github.com/sbromberger/1d6fdad5f73fe0198ef4
> Without Console, integration with Atom editor via Hydrogen is broken. Plus, 
> it's nice to have a REPL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-272) Jupyter Notebook works; Console is broken / hangs

[Chip Senkbeil (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248466#comment-15248466
 ] 

Chip Senkbeil commented on TOREE-272:
-

See https://github.com/apache/incubator-toree/pull/27

> Jupyter Notebook works; Console is broken / hangs
> -
>
> Key: TOREE-272
> URL: https://issues.apache.org/jira/browse/TOREE-272
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: Toree 0.1.5-SNAPSHOT commit 042debc2bd0c
> Jupyter 4.1.0 installed via pip, Spark 1.5.2 installed via Homebrew
>Reporter: Seth Bromberger
>Priority: Blocker
> Fix For: 0.1.0
>
>
> Jupyter notebook will successfully use Toree as a kernel, but Jupyter console 
> hangs on the first evaluation of code. Example logs / sessions are here:
> https://gist.github.com/sbromberger/8a43f79eee08f4268515
> https://gist.github.com/sbromberger/1d6fdad5f73fe0198ef4
> Without Console, integration with Atom editor via Hydrogen is broken. Plus, 
> it's nice to have a REPL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: April Update

[Chip Senkbeil]

Specifically, the talk was called, "Apache Toree: How To Develop
Spark/Scala Apps as Interactive Notebooks"

On Wed, Apr 6, 2016 at 9:38 AM Chip Senkbeil 
wrote:

> Are you talking about the SF Data Science meetup where Asim Jalis gave a
> talk?
>
> http://www.meetup.com/SF-Data-Science/events/228958752/
>
> On Wed, Apr 6, 2016 at 9:34 AM Gino Bustelo  wrote:
>
>> @Corey What was the name of that event that was using Toree?
>>
>> On Wed, Apr 6, 2016 at 9:33 AM, Gino Bustelo  wrote:
>>
>> > Based on comments... this is what will be reported
>> >
>> > 
>> > Toree
>> >
>> > Toree provides applications with a mechanism to interactively and
>> remotely
>> > access Apache Spark. It enables interactive workloads between
>> applications
>> > and a Spark cluster. As a Jupyter Notebook extension, it provides the
>> user
>> > with a preconfigured environment for interacting with Spark using Scala,
>> > Python, R or SQL.
>> >
>> > Toree has been incubating since 2015-12-02.
>> >
>> > Three most important issues to address in the move towards graduation:
>> >
>> >   1. Resolve LGPL dependency: This is in progress, and hopefully we will
>> > have
>> >
>> > the dependency license changed to a license that is compatible with AL2.
>> >
>> >   2. Make a release: nearing completion
>> >   3. Grow a diverse community: We should put some emphasis on growing
>> the
>> >
>> > community and making it diverse (the rule is at least three independent
>> > contributors)
>> >
>> >
>> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
>> > aware of?
>> >
>> >   TOREE-262 - Progress on removal of LGPL dependency
>> >
>> > How has the community developed since the last report?
>> >
>> >   1. Active communication in mailing list and gitter with early adopters
>> >   2. Still working on transitioning users from Spark Kernel project into
>> >  Toree.
>> >   3. First external contribution made by user @*dongjoon-hyun
>> > <https://github.com/dongjoon-hyun>*
>> >   4. Apache Toree is being taught at EVENT_NAME
>> > https://github.com/asimjalis/apache-toree-quickstart
>> >
>> > How has the project developed since the last report?
>> >
>> >   1. Contrinue working with JeroMQ community to further their transition
>> > into
>> >  MPL v2 and away from LGPL.
>> >   2. Magics are now implemented using plug-in framework
>> >   3. Enable sharing of Spark and SQL context between the different
>> > supported languages
>> >   4. Added support to try Toree using Binder (http://mybinder.org/).
>> >   5. Misc bug fixes and features
>> >
>> > Date of last release:
>> >
>> >   None since incubation.
>> >
>> > When were the last committers or PMC members elected?
>> >
>> >   No new additions since incubation
>> >
>> > On Tue, Apr 5, 2016 at 5:11 PM, Gino Bustelo  wrote:
>> >
>> >> Based on Luciano's comments... the top issues to addressed will be
>> >>
>> >> - Resolve LGPL dependency: This is in progress, and hopefully we will
>> have
>> >> the dependency license changed to a license that is compatible with
>> AL2.
>> >> - Make a release: nearing completion
>> >> - Grow a diverse community: We should put some emphasis on growing the
>> >> community and making it diverse (the rule is at least three independent
>> >> contributors)
>> >>
>> >>
>> >> On Tue, Apr 5, 2016 at 3:56 PM, Luciano Resende 
>> >> wrote:
>> >>
>> >>> My comments from last month, related to "important issues before
>> >>> graduation", still applies :
>> >>>
>> >>>
>> https://www.mail-archive.com/dev@toree.incubator.apache.org/msg00262.html
>> >>>
>> >>> On Mon, Apr 4, 2016 at 11:58 AM, Gino Bustelo 
>> wrote:
>> >>>
>> >>> > Last month can be found http://wiki.apache.org/incubator/March2016.
>> >>> > Here is the draft for this month. Mostly the same except the "How
>> has
>> >>> the
>> >>> > *" questions. Please give feedback...
>> >>> >
>> >>> > 
>> >>>