Re: Removing Toree specific support for Python and R

[Corey Stubbs]

+1

On Tue, Sep 25, 2018, 13:10 Marius van Niekerk 
wrote:

> +1
>
> This would simplify usage, and remove a large class of python questions and
> confusion on our mailing list.  Several users assume that our R and Python
> featureset has parity with the primary kernels.
>
> On Tue, 25 Sep 2018 at 12:46 kbat...@gmail.com  wrote:
>
> > +1
> > The benefits that Luciano points out far outweigh the, probably
> > seldom-used, context sharing capabilities with limited features, IMHO.
> > This will eliminate an entire class of issues and questions while
> allowing
> > contributors to narrow their focus only on scala functionality and
> > improvements.
> >
> > On 2018/09/25 16:04:33, Luciano Resende  wrote:
> > > Toree currently has some support for Python and R which are a thin
> > > wrapper around what is provided in Spark. While this enables sharing
> > > the Spark session and SQL Context between Scala and Python code, it
> > > brings a huge gap on functionality when comparing with other Python
> > > kernels like IPython and IRKernel. Adding to that, there is no
> > > community member that is actively enhancing and contributing to these
> > > two areas.
> > >
> > > Based on these, I would like to suggest the following to be done for
> > > Toree 0.3.0 release timeframe:
> > > - Remove Python and R support from Toree
> > > - Document possible suggestions on Toree website (e.g. IPython,
> > > IRKernel or others more active on their respective community)
> > >
> > > Some of the benefits:
> > > - Avoid user confusion and frustration
> > > - Improve kernel startup performance
> > > - Cleaner code to maintain
> > >
> > > Please let me know your thoughts.
> > >
> > > [1]
> >
> https://github.com/apache/incubator-toree/blob/master/etc/examples/notebooks/sqlcontext_sharing.ipynb
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
> --
> regards
> Marius van Niekerk
>

Re: Speeding up Toree Startup

[Corey Stubbs]

When we originally started we chose to use Akka because we thought there
might be the need to do remote interpretation of the code. If I remember
correctly from some earlier performance evaluations, the Akka actor system
initialization did take some time.

We might be able to save some startup time and potentially reduce code
complexity by removing the actor system since there really is no need for
it. This would be a major refactor though. I would think we would want to
have a more thorough discussion about this before we consider it.

Kind Regards,
Corey Stubbs

On Thu, Jan 4, 2018 at 3:26 PM Luciano Resende <luckbr1...@gmail.com> wrote:

> On Thu, Jan 4, 2018 at 1:09 PM, Bradley Kaiser <bradley.kai...@gmail.com>
> wrote:
>
> > Hey Guys,
> >
> > I've been looking at ways to speed up Toree's startup and came up with a
> > quick proof of concept that took the startup time from around 6.5 seconds
> > to 5.5 seconds on my laptop. I'd like to check committer interest in this
> > before I go any further. This change will require rearranging the
> > KernelBootstrap class.
> >
> >
> What are you considering for the startup time ? To me, measuring it via
> yourkit as well, the Main.scala was consuming 13 secs.
>
>
>
> > My proof of concept is simple, I saw that the slowest part of startup was
> > starting the Scala interpreter, so I ripped that out, moved it to its own
> > thread, and started it very early in the bootstrap process. This way
> while
> > it runs the rest of the kernel bootstrap can continue. I did this in a
> > hacky way just to see if it would work. It would need to be implemented
> for
> > real.
> >
> > I tested the startup time by setting up a test harness that just runs the
> > kernel bootstrap and times it. I used YourKit to profile the application
> > and see what was taking the most time.
> >
> > Here are the results from my tests. Time is in seconds.
> >
> > run originalchanged
> > 1   6.876   5.2
> > 2   6.445   5.8
> > 3   6.527   5.7
> > 4   6.676   5.4
> > avg 6.631   5.525
> >
>
> Interesting, as my measurings was telling me that some big portions of the
> time was being spent on performing initialization tasks which required the
> scala interpreter to be ready. Consolidating the multiple calls with some
> other minor tweaks got about 2 secs of performance improvement on the
> kernel initialization.
>
> Another area of possible improvement might be configuring a flag to disable
> auto-loading of all different plugins, which would save us another 1 sec
> (e.g. not loading other interpreter plugins which are not used when toree
> is mainly used as a scala kernel)
>
>
> >
> > If you guys think this is a reasonable approach let me know and i will
> > make a ticket and work on implementing it for real. I think this speedup
> > will work nicely with the changes proposed in TOREE-444 also.
> >
> >
> I have attached my initial findings at
> https://issues.apache.org/jira/browse/TOREE-444
>
>
> Others, any possible additions here ?
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Removing support for Scala 2.10

[Corey Stubbs]

+1 as well

On Mon, Nov 6, 2017 at 11:28 AM Ryan Blue  wrote:

> +1
>
> On Mon, Nov 6, 2017 at 6:04 AM, Marius van Niekerk <
> marius.v.niek...@gmail.com> wrote:
>
> > I don't think that the current support for 2.10 in master even works
> > reliably.  Remove it as it will make the maintenance of this simpler.
> >
> > On Sat, 4 Nov 2017 at 18:56 Chip Senkbeil 
> wrote:
> >
> > > Go for it.
> > >
> > > On Sat, Nov 4, 2017, 3:08 PM Luciano Resende 
> > wrote:
> > >
> > > > The current master only works with Apache Spark 2.x which is based on
> > > Scala
> > > > 2.11 and moving towards Scala 2.12.
> > > >
> > > > Anyone opposed on removing Scala 2.10 from master ?
> > > >
> > > > --
> > > > Luciano Resende
> > > > http://twitter.com/lresende1975
> > > > http://lresende.blogspot.com/
> > > >
> > >
> > --
> > regards
> > Marius van Niekerk
> >
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: [VOTE] Apache Toree 0.2.0 RC1

[Corey Stubbs]

+1

On Sun, Oct 22, 2017 at 5:54 PM Marius van Niekerk <
marius.v.niek...@gmail.com> wrote:

> +1
>
> Ran some tests using mybinder.org
>
>
>
>
> On Sat, 21 Oct 2017 at 23:08 Luciano Resende  wrote:
>
> > Here is my +1
> >
> > Would others please help review the release candidate.
> >
> > On Mon, Oct 2, 2017 at 11:09 PM, Luciano Resende 
> > wrote:
> >
> > > Please vote to approve the release of Apache Toree 0.2.0-incubating
> > > (RC1).
> > >
> > > Tag: v0.2.0-incubating-rc1 (01cd97e9bad04878a8014016c154a50e2a00f21d)
> > >
> > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc1
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > > found at:
> > >
> > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.
> > > 0-incubating-rc1/
> > >
> > > Staging artifacts can be found at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachetoree-1007
> > >
> > > ## Testing Instructions
> > >
> > > The fastest way to get up and running is to using Jupyter.
> > >
> > > 1. Install Jupyter if you haven't already (
> > http://jupyter.org/install.html
> > > )
> > >
> > > 2. Install Apache Toree via `pip install
> https://dist.apache.org/repos/
> > > dist/dev/incubator/toree/0.2.0-incubating-rc1/toree-pip/
> > > toree-0.2.0.tar.gz`
> > > <
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0-incubating-rc1/toree-pip/toree-0.2.0.tar.gz
> > >
> > > followed by `jupyter toree install`
> > >
> > > - You need to set a valid Apache Spark 2.x home, which can be done via
> > > `jupyter toree install --spark_home=/usr/local/spark`
> > >
> > > - You may need to run with `sudo` for installation permission
> > >
> > > - For all installation options, run `jupyter toree install --help-all`
> > >
> > > 4. Run a Jupyter notebook server via `jupyter notebook`
> > >
> > > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > > you can install via `pip install notebook`
> > >
> > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
> under
> > > new
> > > dev
> > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> > the
> > > notebook
> > >
> > > ## Voting Instructions
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating
> > > [ ] -1 Do not release this package because ...
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
> --
> regards
> Marius van Niekerk
>

Re: CI Builds

[Corey Stubbs]

I know there have been flaky tests in the past due to testing akka actors
and dealing with the concurrency for those classes. My best guess is that
current issues relate to similar timing issues.

It might be worth having some record of all the test runs so we can see
which ones are consistently. Anybody know of a way to do this with travis?
It basically would entail publishing the test report to some static host.

On Fri, Mar 17, 2017 at 8:18 PM Luciano Resende 
wrote:

> The current CI builds seems a little flaky, and will usually fail unrelated
> to the changes being proposed. Has anyone investigated these issues and has
> an understanding of what causes them? Is it related to Travis.CI? The
> actual tests?  Timing?
>
> Anyway, we should make the CI build more stable.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [VOTE] Apache Toree 0.1.0 RC6

[Corey Stubbs]

+1 Ran and installed the pip dist with Apache Spark 1.6.3.

On Thu, Feb 16, 2017 at 11:24 AM Marius van Niekerk <
marius.v.niek...@gmail.com> wrote:

> +1
>
> Going to +1 this since I assisted with getting rid of some of the bundled
> pieces that were blocking for RC5.
>
> On Thu, Feb 16, 2017, 12:14 Chip Senkbeil  wrote:
>
> I'll go ahead and give a +1 since I've done a bit of testing using
> Scala/SparkR to get this back up to speed.
>
> Also resolved the remaining issues we had from the incubator general vote.
> We're still bundling additional license information in the binary release,
> but the source release no longer has the extra license info tacked onto the
> LICENSE file. I've also fixed the source release such that it is no longer
> dependent on being in a Git repository to be built nor does it require you
> to have sbt installed.
>
> On Thu, Feb 16, 2017 at 11:09 AM Chip Senkbeil 
> wrote:
>
> > Please vote to approve the release of the following candidate as Apache
> > Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE
> files
> > since this is our first release.
> >
> > ## Information and Artifacts
> >
> > The tag to be voted on is v0.1.0-rc6
> > (51fa49cb5898e0c5b7824f986382436b969cabc7), located here:
> >
> >
> >
>
> https://github.com/apache/incubator-toree/commit/51fa49cb5898e0c5b7824f986382436b969cabc7
> >
> > All distribution packages, including signatures, digests, etc. can be
> found
> > at:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/
> >
> > Staging artifacts can be found at:
> >
> > https://repository.apache.org/content/repositories/orgapachetoree-1006
> >
> > ## Testing Instructions
> >
> > The fastest way to get up and running is to using Jupyter.
> >
> > 1. Install Jupyter if you haven't already (
> http://jupyter.org/install.html
> > )
> >
> > 2. Grab the Apache Toree archive from
> >
> >
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/toree-pip/apache-toree-0.1.0.tar.gz
> >
> > 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> > followed by `jupyter toree install`
> >
> > - You need to set a valid Apache Spark 1.6.x home, which can be done via
> > `jupyter
> > toree install --spark_home=/usr/local/spark`
> >
> > - You may need to run with `sudo` for installation permission
> >
> > - For all installation options, run `jupyter toree install --help-all`
> >
> > 4. Run a Jupyter notebook server via `jupyter notebook`
> >
> > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > you can install via `pip install notebook`
> >
> > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> > new
> >
> > - If you want other interpreter profiles than Scala, you can change the
> > interpreters via `jupyter toree install --interpreters=PySpark,SQL`
> >
> > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> the
> > notebook
> >
> > ## Voting Instructions
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> least
> > 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Toree 0.1.0
> > [ ] -1 Do not release this package because ...
> >
>
> --
> regards
> Marius van Niekerk
>

Re: Development model

[Corey Stubbs]

Sounds good.

Kind Regards,
Corey Stubbs

On Thu, Feb 2, 2017 at 9:50 AM Luciano Resende <luckbr1...@gmail.com> wrote:

> I am not suggesting doing "everything" via SBT, just the basic compile,
> test, build and package so that contributors used to other sbt based
> projects fill comfortable getting started with the project. For all the
> other more optional/complex tasks, I am all for using a set of fully
> documented tasks.
>
> On Thu, Feb 2, 2017 at 7:23 AM, Corey Stubbs <cas5...@gmail.com> wrote:
>
> > My typical development flow is to write code, run pip-release , and then
> > install the pip release locally on my machine (pip install
> > dist/toree-pip/pip-release; jupyter toree install), and test changes with
> > the install.
> >
> > +1 on working on documenting the make targets.
> >
> > In terms of getting everything done via SBT, I don't know if that will
> > ultimately be possible. Targets like, compile, test, and build can (and a
> > majority should be) done with SBT. However, since we are doing some
> python
> > things to ease the pain of installing toree as a kernel into the Jupyter
> > environment (via pip install).
> >
> > Regardless, I agree the Makefile has become out of hand and can use some
> > cleanup and loving.
> >
> > I can start the work on documenting the targets that are there and clean
> up
> > old ones. Once I get a PR open I will post it here and in the related
> > issue: https://issues.apache.org/jira/browse/TOREE-367
> >
> > On Tue, Jan 31, 2017 at 1:47 PM Luciano Resende <luckbr1...@gmail.com>
> > wrote:
> >
> > > On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
> > > marius.v.niek...@gmail.com> wrote:
> > >
> > > > I think just documenting development workflow and process properly
> will
> > > > help a great deal.
> > > > Building Toree is a non-trivial exercise sadly.
> > > >
> > > >
> > > Completely agree that is a non-trivial exercise, but should not be.
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Toree client programming model

[Corey Stubbs]

So the client programming model is meant to reflect the Jupyter Protocol,
which has taken into account issues like streaming. However, the client
could still be cleaned up. Maybe something along the lines of:

val exRes: DeferredExecution = client.execute(code)
.onError(executeReplyError =>{
  ...
}).onSuccess(Option(executeResult) => {
  ...
}).onStream(streamResult => {
  ...
})

Key thing being the optional execute result as a parameter to onSuccess.
This would be a better interface to handle when there are/are not results
from an execution.

Kind Regards,
Corey Stubbs

On Tue, Jan 31, 2017 at 10:29 AM Luciano Resende <luckbr1...@gmail.com>
wrote:

The Toree client seems to be a client utility to enable other applications
to interact with Toree/Spark. These applications are not based or aware of
the ZeroMQ protocol, but nevertheless, we are imposing them on a very
similar programming model while evaluating code using the client.

See an example below, where the application programmer has to handle
different options, where more than one option can be called per eval
result, and the order cannot be known prior to the execution time :

val exRes: DeferredExecution = client.execute(code)
.onResult(executeResult => {
  ...
}).onError(executeReplyError =>{
  ...
}).onSuccess(executeReplyOk => {
  ...
}).onStream(streamResult => {
  ...
})


I was wondering what are the community thoughts on enhancing the client to
abstract its programming model, and having all this abstracted and handled
on the client implementation, and the eval would return one promise (e.g.
onResult which one option would be status success or error) or at most two
promises (e.g. onSuccess and onError).

The only issue I haven't fully understood yet is how streaming is handled
and what are the implication of these changes on this scenario.

Thoughts?

--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Development model

[Corey Stubbs]

My typical development flow is to write code, run pip-release , and then
install the pip release locally on my machine (pip install
dist/toree-pip/pip-release; jupyter toree install), and test changes with
the install.

+1 on working on documenting the make targets.

In terms of getting everything done via SBT, I don't know if that will
ultimately be possible. Targets like, compile, test, and build can (and a
majority should be) done with SBT. However, since we are doing some python
things to ease the pain of installing toree as a kernel into the Jupyter
environment (via pip install).

Regardless, I agree the Makefile has become out of hand and can use some
cleanup and loving.

I can start the work on documenting the targets that are there and clean up
old ones. Once I get a PR open I will post it here and in the related
issue: https://issues.apache.org/jira/browse/TOREE-367

On Tue, Jan 31, 2017 at 1:47 PM Luciano Resende 
wrote:

> On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
> marius.v.niek...@gmail.com> wrote:
>
> > I think just documenting development workflow and process properly will
> > help a great deal.
> > Building Toree is a non-trivial exercise sadly.
> >
> >
> Completely agree that is a non-trivial exercise, but should not be.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Publishing Toree snapshots ?

[Corey Stubbs]

We do not publish the uber jar, but there is a target to generate it. You
can run "make build" and this will generate the assembly jar in
target/scala-$(SCALA_VERSION) .

Kind Regards,
Corey Stubbs


On Fri, Jan 13, 2017 at 1:20 PM Luciano Resende <luckbr1...@gmail.com>
wrote:

> How can one publish the generated ubber jar from dist as SNAPSHOT to maven
> ? Is there any specific make task for that  ? Or have anyone done that
> before ?
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [VOTE] Apache Toree 0.1.0 RC4

[Corey Stubbs]

+1

On Wed, Jan 11, 2017 at 9:42 AM Gino Bustelo  wrote:

> +1
>
> I created a simple target in our Makefile to test a release. Pass the URL
> to the pip package and it will install it on jupyter inside a docker
> container. I tested using that and ran a notebook. It all worked.
>
> On Wed, Jan 11, 2017 at 8:21 AM Marius van Niekerk <
> marius.v.niek...@gmail.com> wrote:
>
> > +1
> >
> > Ran some sample spark notebooks in yarn-client mode against the provided
> > binaries and a locally built copy.
> >
> > -Marius van Niekerk
> >
> >
> > On Wed, 11 Jan 2017 at 00:38 Luciano Resende 
> wrote:
> >
> > > +1
> > >
> > > Went over some legal files, verified some parts of the uber jar, built
> > > locally, etc
> > >
> > >
> > > On Tue, Jan 10, 2017 at 2:15 PM, Chip Senkbeil <
> chip.senkb...@gmail.com>
> > > wrote:
> > >
> > > > Please vote to approve the release of the following candidate as
> Apache
> > > > Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE
> > > files
> > > > since this is our first release.
> > > >
> > > > ## Information and Artifacts
> > > >
> > > > The tag to be voted on is v0.1.0-rc4
> > > > (1d526954ecaba1d5dc0f40ec555dd598b2b11df7), located here:
> > > >
> > > > https://github.com/apache/incubator-toree/commit/
> > > > 1d526954ecaba1d5dc0f40ec555dd598b2b11df7
> > > >
> > > > All distribution packages, including signatures, digests, etc. can be
> > > found
> > > > at:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc4/
> > > >
> > > > Staging artifacts can be found at:
> > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapachetoree-1002/
> > > >
> > > > ## Testing Instructions
> > > >
> > > > The fastest way to get up and running is to using Jupyter.
> > > >
> > > > 1. Install Jupyter if you haven't already (
> > > http://jupyter.org/install.html
> > > > )
> > > >
> > > > 2. Grab the Apache Toree archive from
> > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.
> > > > 0/rc4/toree-pip/apache-toree-0.1.0.tar.gz
> > > >
> > > > 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> > > > followed by `jupyter toree install`
> > > >
> > > > - You need to set a valid Spark home, which can be done via
> > `jupyter
> > > > toree install --spark_home=/usr/local/spark`
> > > >
> > > > - You may need to run with `sudo` for installation permission
> > > >
> > > > - For all installation options, run `jupyter toree install
> > > --help-all`
> > > >
> > > > 4. Run a Jupyter notebook server via `jupyter notebook`
> > > >
> > > > - If the notebook portion of Jupyter is not installed but Jupyter
> > is,
> > > > you can install via `pip install notebook`
> > > >
> > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
> > under
> > > > new
> > > >
> > > > - If you want other interpreter profiles than Scala, you can
> change
> > > the
> > > > interpreters via `jupyter toree install --interpreters=PySpark,SQL`
> > > >
> > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()`
> in
> > > the
> > > > notebook
> > > >
> > > > ## Voting Instructions
> > > >
> > > > The vote is open for at least 72 hours and passes if a majority of at
> > > least
> > > > 3 +1 PMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Toree 0.1.0
> > > > [ ] -1 Do not release this package because ...
> > > >
> > >
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> > --
> > regards
> > Marius van Niekerk
> >
>

Re: Should we start another release vote?

[Corey Stubbs]

+1

On Thu, Jan 5, 2017 at 12:31 PM Marius van Niekerk <
marius.v.niek...@gmail.com> wrote:

> +1 looks good to me. Just some SBT changes
>
> On Thu, Jan 5, 2017, 08:08 Luciano Resende  wrote:
>
> > + 1
> >
> > On Wed, Jan 4, 2017 at 6:12 AM Chip Senkbeil 
> > wrote:
> >
> > > This is long overdue, but since the artifacts weren't in staging at the
> > >
> > > time of the vote for RC3, should we have another release vote?
> > >
> > > --
> > Sent from my Mobile device
> >
> --
> regards
> Marius van Niekerk
>

Re: [VOTE] Apache Toree 0.1.0 RC2

[Corey Stubbs]

+1 Release this package as Apache Toree 0.1.0

On Tue, Oct 11, 2016 at 2:16 PM Gino Bustelo  wrote:

> Please vote to approve the release of the following candidate as
> Apache Toree version 0.1.0
>
>
> The commit to be voted on is 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> <
> https://github.com/apache/incubator-toree/tree/119bf3e2d1d16986f55802cf2323e8629ea25ef8
> >
>
>
> https://github.com/apache/incubator-toree/tree/119bf3e2d1d16986f55802cf2323e8629ea25ef8
> <
> https://github.com/apache/bahir/tree/368c436ae2ad34b3ca64d11801aee69e478555f7
> >
>
> All distribution packages, including signatures, digests, etc. can be
> found at:
>
> *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc2/
> *
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.1.0
> [ ] -1 Do not release this package because ...
>

Re: Error For Toree Site Repo svn ci

[Corey Stubbs]

Luciano, misread your last comment. I updated to point back to HTTPS and
here is my svn info and the error I receive:

svn info:
Path: .
Working Copy Root Path: /private/tmp/toree-site
URL: https://svn.apache.org/repos/asf/incubator/toree/site
Repository Root: https://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1748413
Node Kind: directory
Schedule: normal
Last Changed Author: lresende
Last Changed Rev: 1733392
Last Changed Date: 2016-03-02 18:10:36 -0600 (Wed, 02 Mar 2016)

Error Text:
svn: E195023: Changing file '/private/tmp/toree-site/index.html' is
forbidden by the server
svn: E175013: Access to
'/repos/asf/!svn/txr/1748457-12dpf/incubator/toree/site/index.html'
forbidden

On Tue, Jun 14, 2016 at 8:51 AM Corey Stubbs <cas5...@gmail.com> wrote:

> Luciano,
> I was using HTTPS. Switched over to
> http://svn.apache.org/repos/asf/incubator/toree/site, but am now getting
> this error:
>
> svn: E175013: Commit failed (details follow):
> svn: E175013: Access to '/repos/asf/!svn/me' forbidden
>
> svn info returns the following:
> Path: .
> Working Copy Root Path: /private/tmp/toree-site
> URL: http://svn.apache.org/repos/asf/incubator/toree/site
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1748413
> Node Kind: directory
> Schedule: normal
> Last Changed Author: lresende
> Last Changed Rev: 1733392
> Last Changed Date: 2016-03-02 18:10:36 -0600 (Wed, 02 Mar 2016)
>
> Kind Regards,
> Corey Stubbs
>
> On Mon, Jun 13, 2016 at 7:06 PM Luciano Resende <luckbr1...@gmail.com>
> wrote:
>
>> Moving to Toree dev (bcc general@a.o)
>>
>> What svn info returns to you ? Just want to make sure you are using https
>> which is writable compared to http which is read only.
>>
>>
>> On Mon, Jun 13, 2016 at 2:12 PM, Corey Stubbs <cas5...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying to update the site for Apache Toree, but am unable to push
>> to
>> > svn. When I run *svn ci* I see the following error:
>> >
>> > svn: E195023: Commit failed (details follow):
>> > svn: E195023: Changing directory
>> > '/private/tmp/toree-site/site/assets/image' is forbidden by the server
>> > svn: E175013: Access to
>> > '/repos/asf/!svn/txr/1748309-12dl1/incubator/toree/site/assets/image'
>> > forbidden
>> > svn: E195023: Your commit message was left in a temporary file:
>> > svn: E195023:'/private/tmp/toree-site/svn-commit.tmp'
>> >
>> >
>> > The image folder mentioned in the error is new as of my commit. I am
>> new to
>> > svn so I don't know if I need to update something on the server or not.
>> >
>> >
>> > Kind Regards,
>> > Corey Stubbs
>> >
>>
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>

Re: Help from Mentors

[Corey Stubbs]

I found this article

explaining podling releases. It appears we should our own directory, under
http://www.apache.org/dist/incubator/, to distribute our releases. This
 seems to be the
complete release process for binary package for apache projects. I found
all of these links (and more) on the dev package under the releases section
.



On Tue, Apr 5, 2016 at 9:24 AM Gino Bustelo  wrote:

> We are getting close to completing work to our build scripts (PR #13
> ) to make the project
> follow the Apache release criteria that we've been able to find through
> search. Mainly auditing license headers, POM content, jar generation with
> NOTICE/LICENSE files, etc.
>
> At this point we need someone with experience that can verify all the steps
> we've done. Also, it is not clear to me what the process is to getting all
> the assets published for a release vote. Several question are:
>
> 1. Can publishing of assets be automated using Travis? Including maven
> publish, and binary package distribution.
> 2. Where do we publish binary packages? Not talking about JARs, rather a
> package with libraries and scripts to start/run Toree.
>
> Any help is appreciated...
>
> Thanks,
> Gino
>

Re: April Update

[Corey Stubbs]

I believe "Magics are not implemented using plug-in framework" should read
"Magics are now implemented using plug-in framework"

On Mon, Apr 4, 2016 at 1:59 PM Gino Bustelo  wrote:

> Last month can be found http://wiki.apache.org/incubator/March2016.
> Here is the draft for this month. Mostly the same except the "How has the
> *" questions. Please give feedback...
>
> 
> Toree
>
> Toree provides applications with a mechanism to interactively and remotely
> access Apache Spark. It enables interactive workloads between applications
> and a Spark cluster. As a Jupyter Notebook extension, it provides the user
> with a preconfigured environment for interacting with Spark using Scala,
> Python, R or SQL.
>
> Toree has been incubating since 2015-12-02.
>
> Three most important issues to address in the move towards graduation:
>
>   1. Removal of LGPL dependency
>   2. Ease of installation
>   3. Documentation of APIs
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
>
>   TOREE-262 - Progress on removal of LGPL dependency
>
> How has the community developed since the last report?
>
>   1. Active communication in mailing list and gitter with early adopters
>   2. Still working on transitioning users from Spark Kernel project into
>  Toree.
>   3. First external contribution made by user @*dongjoon-hyun
> *
>   4. Apache Toree is being taught at EVENT_NAME
> https://github.com/asimjalis/apache-toree-quickstart
>
> How has the project developed since the last report?
>
>   1. Contrinue working with JeroMQ community to further their transition
> into
>  MPL v2 and away from LGPL.
>   2. Magics are not implemented using plug-in framework
>   3. Enable sharing of Spark and SQL context between the different
> supported languages
>   4. Added support to try Toree using Binder (http://mybinder.org/).
>   5. Misc bug fixes and features
>
> Date of last release:
>
>   None since incubation.
>
> When were the last committers or PMC members elected?
>
>   No new additions since incubation
>

Re: [jira] [Commented] (TOREE-280) PYTHON_PATH Needs To Be Configured At Install

[Corey Stubbs]

Yavuz,  I agree adding more options is not in the best interest of the
users. +1 to the idea of looking at what is in the Spark distribution.
Will update the jira description to reflect that idea.
On Mar 18, 2016 5:15 PM, "Yavuz Nuzumlali (JIRA)"  wrote:

>
> [
> https://issues.apache.org/jira/browse/TOREE-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202266#comment-15202266
> ]
>
> Yavuz Nuzumlali commented on TOREE-280:
> ---
>
> What about the following option:
>
> - Since we know SPARK_HOME (either given by user or the default value), we
> could look inside the ??$SPARK_HOME/python/lib?? folder for a file name
> matching py4j, and add that as the value.
>
> I think holding a mapping would add a level of complexity which seems
> redundant.
>
> Also, other alternatives does not seem to be convenient for users, adding
> SPARK_HOME and the interpreter should be enough for toree to run.
>
> > PYTHON_PATH Needs To Be Configured At Install
> > -
> >
> > Key: TOREE-280
> > URL: https://issues.apache.org/jira/browse/TOREE-280
> > Project: TOREE
> >  Issue Type: Improvement
> >Reporter: Corey A Stubbs
> >Priority: Minor
> >
> > When installing the interpreters, the PYTHON_PATH env var will need to
> change based on the spark distribution. There are a couple paths to do this.
> > 1. We can allow the PY4J version to be passed
> > 2. Figure out the SPARK_VERSION and keep a mapping of the PY4J versions
> and substitute the value
> > 3. The user must specify the value for the PYTHON_PATH
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Re: Issue Tracker not available

[Corey Stubbs]

Luciano,
My jira id is Lull3rSkat3r. Please add me to jira.

On Tue, Jan 19, 2016 at 9:01 AM Luciano Resende <luckbr1...@gmail.com>
wrote:

> On Tuesday, January 19, 2016, Chip Senkbeil <chip.senkb...@gmail.com>
> wrote:
>
> > Oh, I assumed you had all of the usernames for pmc members since they
> were
> > supposed to email you their IDs, right? Would it make sense to list them
> > here? Or for each of us to comment on the JIRA itself?
> >
> >
> The jira id is not the same as apache userid
>
>
> > I've been getting emails when others got their accounts set up and the
> only
> > one I haven't seen is Miao Wang.
> >
> >
> He said he submit, but maybe he is having the same issue with corporate
> emails being flagged as span.
>
>
> > On Mon, Jan 18, 2016 at 5:50 PM Luciano Resende <luckbr1...@gmail.com
> > <javascript:;>>
> > wrote:
> >
> > > On Mon, Jan 18, 2016 at 12:31 PM, Corey Stubbs <cas5...@gmail.com
> > <javascript:;>> wrote:
> > >
> > > > Everything seems to have imported correctly. And +1 on all PMC
> members
> > > > should have access to the project JIRA.
> > > >
> > > >
> > > I would need the jira username for the pmc members in the jira.
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://people.apache.org/~lresende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>
> --
> Sent from my Mobile device
>

Re: Issue Tracker not available

[Corey Stubbs]

Everything seems to have imported correctly. And +1 on all PMC members
should have access to the project JIRA.

On Mon, Jan 18, 2016 at 9:58 AM Chip Senkbeil 
wrote:

> What feedback were you wanting on the infrastructure jira? It looks like
> the issues imported fine, although I'm wondering how we link ourselves to
> the existing comments under our names.
>
> Were we planning to use the dev mailing list as the destination for jira
> notifications? Or is there some other mailing list that makes more sense?
>
> I'm assuming that all of the PMC members for Toree would be project admins
> for the jira project, right? Did you want us to comment individually, or
> were you going to submit a list like was asked in the jira?
>
> On Mon, Jan 18, 2016 at 12:59 AM Luciano Resende 
> wrote:
>
> > Could you please review
> >
> > https://issues.apache.org/jira/browse/TOREE
> >
> > And provide feedback on
> >
> > https://issues.apache.org/jira/browse/INFRA-11082
> >
> > Note that the jira has some questions on the last comment from Gavin.
> >
> > Thanks
> >
> >
> > On Fri, Jan 15, 2016 at 7:00 AM, Gino Bustelo  wrote:
> >
> > > @lresende Awesome, Thanks!
> > >
> > > On Thu, Jan 14, 2016 at 3:22 PM, Luciano Resende  >
> > > wrote:
> > >
> > > > Created task for creating Toree jira and importing issues from Github
> > > >
> > > > https://issues.apache.org/jira/browse/INFRA-11082
> > > >
> > > > On Wed, Jan 13, 2016 at 12:16 PM, Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks for the tip, Hitesh!
> > > > >
> > > > > On Wed, Jan 13, 2016 at 12:55 PM Hitesh Shah 
> > > wrote:
> > > > >
> > > > > > I believe the INFRA folks may have tools to do the import. A jira
> > can
> > > > be
> > > > > > filed against INFRA to get some help in this regard.
> > > > > >
> > > > > > The Apex folks went through a complex import earlier (
> > > > > > https://issues.apache.org/jira/browse/INFRA-10144 ) but that was
> > > from
> > > > > > their own atlassian instance.
> > > > > >
> > > > > > — Hitesh
> > > > > >
> > > > > > On Jan 13, 2016, at 6:58 AM, Chip Senkbeil <
> > chip.senkb...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 to that. Any automation would be a great help. Going back
> and
> > > > > closing
> > > > > > > ones we don't need is a lot easier than selectively picking
> them
> > at
> > > > the
> > > > > > > beginning.
> > > > > > >
> > > > > > > On Wed, Jan 13, 2016 at 8:55 AM Gino Bustelo <
> g...@bustelos.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Do we want all the Issues from Github ported or no?
> > > > > > >>
> > > > > > >> @lresende hinted of a tool Apache has to port Issues into
> JIRA.
> > I
> > > > > > mentioned
> > > > > > >> to him that if it was easy, than then porting the issues would
> > be
> > > > good
> > > > > > and
> > > > > > >> we would then go through them and clean them up.
> > > > > > >>
> > > > > > >> Thoughts?
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Jan 13, 2016 at 8:30 AM, Chip Senkbeil <
> > > > > chip.senkb...@gmail.com
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Is there a timeline for when the issue tracker will be
> > available
> > > > for
> > > > > > >> Toree?
> > > > > > >>> Is there something we haven't done yet?
> > > > > > >>>
> > > > > > >>> Currently, https://issues.apache.org/jira/browse/TOREE gives
> > me
> > > a
> > > > > > >> "Project
> > > > > > >>> does not exist" message.
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Luciano Resende
> > > > http://people.apache.org/~lresende
> > > > http://twitter.com/lresende1975
> > > > http://lresende.blogspot.com/
> > > >
> > >
> >
> >
> >
> > --
> > Luciano Resende
> > http://people.apache.org/~lresende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>