Re: [VOTE] Apache Toree 0.4.0 RC1

[Marius van Niekerk]

And my +1

Tested a few things in a local docker setup.

On Fri, Jul 17, 2020, 20:41 Kevin Bates  wrote:

> +1, Thanks Luciano!
>
> On Fri, Jul 17, 2020 at 10:32 AM Luciano Resende 
> wrote:
>
> > Off course, my +1
> >
> > On Thu, Jul 16, 2020 at 12:25 PM Luciano Resende 
> > wrote:
> > >
> > > Please vote to approve the release of Apache Toree 0.4.0-incubating
> > (RC1).
> > >
> > > Tag: v0.4.0-incubating-rc1 (b2374e28ce66ae28079703d725b7c0db4d4a)
> > >
> > > https://github.com/apache/incubator-toree/tree/v0.4.0-incubating-rc1
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > found at:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/
> > >
> > > Staging artifacts can be found at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachetoree-1016
> > >
> > > ## Testing Instructions
> > >
> > > The fastest way to get up and running is to using Jupyter.
> > >
> > > 1. Install Jupyter if you haven't already (
> > http://jupyter.org/install.html)
> > >
> > > 2. Install Apache Toree via `pip install
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz`
> 
> > <
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.4.0-incubating-rc1/toree-pip/toree-0.4.0.tar.gz
> >
> > > followed by `jupyter toree install`
> > >
> > > - You need to set a valid Apache Spark 2.x home, which can be done via
> > > `jupyter toree install --spark_home=/usr/local/spark`
> > >
> > > - You may need to run with `sudo` for installation permission
> > >
> > > - For all installation options, run `jupyter toree install --help-all`
> > >
> > > 4. Run a Jupyter notebook server via `jupyter notebook`
> > >
> > > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > > you can install via `pip install notebook`
> > >
> > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
> > under new
> > > dev
> > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> > the
> > > notebook
> > >
> > > ## Voting Instructions
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.4.0-incubating
> > > [ ] -1 Do not release this package because ...
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>

Re: [VOTE] Apache Toree 0.3.0 RC1

[Marius van Niekerk]

+1

Tested a sample notebook in a binderhub instance running toree.

On Wed, 24 Oct 2018 at 03:27 Manu Zhang  wrote:

> +1 (non-binding)
>
>
> I downloaded the toree-0.3.0.tar.gz, installed with pip and checked the
> following functionalities with jupyterlab-0.28.2.
>
> 1. Created a SparkSession,  read from a HDFS parquet file into Spark
> DataFrame and showed some rows.
> 2. Disabled console output with showOutput magic
> 3. SQL magic with variables in sql
>
>
> Thanks,
> Manu Zhang
> On Oct 18, 2018, 11:16 AM +0800, Luciano Resende ,
> wrote:
> > On Tue, Oct 16, 2018 at 9:30 AM Luciano Resende 
> wrote:
> > >
> > > Please vote to approve the release of Apache Toree 0.3.0-incubating
> (RC1).
> > >
> > > Tag: v0.3.0-incubating-rc1 (3ff5df33cc459ea214e542a4798fd9ba6e0a2ef8)
> > >
> > > https://github.com/apache/incubator-toree/tree/v0.3.0-incubating-rc1
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> found at:
> > >
> > >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/
> > >
> > > Staging artifacts can be found at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachetoree-1014
> > >
> > > ## Testing Instructions
> > >
> > > The fastest way to get up and running is to using Jupyter.
> > >
> > > 1. Install Jupyter if you haven't already (
> http://jupyter.org/install.html)
> > >
> > > 2. Install Apache Toree via `pip install
> > >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz`
> <https://dist.apache.org/repos/dist/dev/incubator/toree/0.3.0-incubating-rc1/toree-pip/toree-0.3.0.tar.gz>
> > > followed by `jupyter toree install`
> > >
> > > - You need to set a valid Apache Spark 2.x home, which can be done via
> > > `jupyter toree install --spark_home=/usr/local/spark`
> > >
> > > - You may need to run with `sudo` for installation permission
> > >
> > > - For all installation options, run `jupyter toree install --help-all`
> > >
> > > 4. Run a Jupyter notebook server via `jupyter notebook`
> > >
> > > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > > you can install via `pip install notebook`
> > >
> > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
> under new
> > > dev
> > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()`
> in the
> > > notebook
> > >
> > > ## Voting Instructions
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.3.0-incubating
> > > [ ] -1 Do not release this package because ...
> > >
> >
> >
> > Off course, my +1
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: [VOTE] Apache Toree 0.2.0 RC6

[Marius van Niekerk]

+1

Ran through a few sample spark RDD, DataSet and ML queries.
Tested on a freshly built jupyter/docker-stack/all-spark-notebook against
0.2.0-rc6


On Mon, 30 Jul 2018 at 13:39 Luciano Resende  wrote:

> Please vote to approve the release of Apache Toree 0.2.0-incubating
>  (RC6).
>
> Tag: v0.2.0-incubating-rc6 (eae2da61828a0c88e2ab64a7ffb9ff7647909a2c)
>
> https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc6
>
> All distribution packages, including signatures, digests, etc. can be
> found at:
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0
> -incubating-rc6/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachetoree-1013
>
> ## Testing Instructions
>
> The fastest way to get up and running is to using Jupyter.
>
> 1. Install Jupyter if you haven't already (http://jupyter.org/install.html
> )
>
> 2. Install Apache Toree via `pip install https://dist.apache.org/repos/
> dist/dev/incubator/toree/0.2.0-incubating-rc6/toree-pip/toree
> -0.2.0.tar.gz`
> followed by `jupyter toree install`
>
> - You need to set a valid Apache Spark 2.x home, which can be done via
> `jupyter toree install --spark_home=/usr/local/spark`
>
> - You may need to run with `sudo` for installation permission
>
> - For all installation options, run `jupyter toree install --help-all`
>
> 4. Run a Jupyter notebook server via `jupyter notebook`
>
> - If the notebook portion of Jupyter is not installed but Jupyter is,
> you can install via `pip install notebook`
>
> 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> new
> dev
> 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in the
> notebook
>
> ## Voting Instructions
>
> The vote is open for at least 72 hours and passes if a majority of at
> least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.2.0-incubating
> [ ] -1 Do not release this package because ...
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: jar import issues

[Marius van Niekerk]

Toree 0.1 is compiled against scala 2.10

On Fri, May 18, 2018, 18:39 Igor Fialko <igorfia...@gmail.com> wrote:

> Hello community,
>
> name is Igor and I'm working in my spare time on a trading system. So far I
> used jupyter notebook with python kernel for visualisation/playing around
> with data. I wanted to go a step forward and reuse my existing scala code,
> so I came up with toree as seemingly the only one still maintained jupyter
> kernel.
>
> Therefore I've assembled a jar via sbt and tried to add it in the notebook.
> It has failed with an exact same error as here.
>
>
> https://stackoverflow.com/questions/46403426/adding-dependency-to-apache-toree-fails-on-import
>
> 2018-05-18 23:43:51 WARN  MagicManager:111 - Magic AddJar failed to execute
> with error:
> object Predef does not have a member classOf
> scala.reflect.internal.FatalError: object Predef does not have a member
> classOf
>
>
> 2018-05-18 23:43:51 ERROR OneForOneStrategy:159 - object Predef does not
> have a member classOf
> scala.reflect.internal.FatalError: object Predef does not have a member
> classOf
>
> So I've tried to import the jar via toree_opts like
>
> sudo jupyter toree install --spark_home=/path/bin/spark-2.3.0
> --toree_opts='--jar-dir /path/target/scala-2.12'
>
> which didn't make any impact unfortunately (from source code I could see
> that it should print a message that jar could not be loaded, neither did an
> import work).
>
> I could assume that it's maybe a scala version issue? My project is
> compiled against 2.12.3, I couldn't find scala version that toree is using.
>
> Did somebody face this issue? How could I debug what is wrong?
>
> Thanks
> Best regards,
> Igor
>
-- 
regards
Marius van Niekerk

Re: [VOTE] Apache Toree 0.2.0 RC5

[Marius van Niekerk]

Tested this on binderhub and seems to work fine +1

On Wed, 25 Apr 2018 at 15:32 Luciano Resende <luckbr1...@gmail.com> wrote:

> On Sat, Apr 21, 2018 at 3:58 PM, Luciano Resende <luckbr1...@gmail.com>
> wrote:
>
> > Please vote to approve the release of Apache Toree 0.2.0-incubating
> > (RC5).
> >
> > Tag: v0.2.0-incubating-rc5 (1b2d4aac2113bf59bbb4ec1cff75adbe127f221a)
> >
> > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc5
> >
> > All distribution packages, including signatures, digests, etc. can be
> > found at:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0
> > -incubating-rc5/
> >
> > Staging artifacts can be found at:
> >
> > https://repository.apache.org/content/repositories/orgapachetoree-1012
> >
> > ## Testing Instructions
> >
> > The fastest way to get up and running is to using Jupyter.
> >
> > 1. Install Jupyter if you haven't already (
> http://jupyter.org/install.html
> > )
> >
> > 2. Install Apache Toree via `pip install https://dist.apache.org/repos/
> > dist/dev/incubator/toree/0.2.0-incubating-rc5/toree-pip/toree
> > -0.2.0.tar.gz`
> > followed by `jupyter toree install`
> >
> > - You need to set a valid Apache Spark 2.x home, which can be done via
> > `jupyter toree install --spark_home=/usr/local/spark`
> >
> > - You may need to run with `sudo` for installation permission
> >
> > - For all installation options, run `jupyter toree install --help-all`
> >
> > 4. Run a Jupyter notebook server via `jupyter notebook`
> >
> > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > you can install via `pip install notebook`
> >
> > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> > new
> > dev
> > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> the
> > notebook
> >
> > ## Voting Instructions
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Toree 0.2.0-incubating
> > [ ] -1 Do not release this package because ...
> >
> >
>
> Off course, here is my +1.
>
> Other committers and mentors, please vote here.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: [VOTE] Apache Toree 0.2.0 RC4

[Marius van Niekerk]

Tested using binderhub, ran a few notebook tests.

On Sat, Apr 7, 2018, 07:26 Marius van Niekerk <marius.v.niek...@gmail.com>
wrote:

> +1
>
> On Wed, Apr 4, 2018, 21:45 Luciano Resende <luckbr1...@gmail.com> wrote:
>
>> Thanks Ryan,
>>
>> All others, we need one more vote before we can move this to IPMC
>> approval.
>>
>> On Wed, Apr 4, 2018 at 11:16 AM, Ryan Blue <rb...@netflix.com.invalid>
>> wrote:
>>
>> > +1 (binding)
>> >
>> > Checked licenses in src tarball, built and tested using `make test`,
>> > validated checksums and signature.
>> >
>> > Thanks for working on this, Luciano!
>> >
>> > On Sat, Mar 31, 2018 at 9:46 AM, Luciano Resende <luckbr1...@gmail.com>
>> > wrote:
>> >
>> > > My +1.
>> > >
>> > > Also, notice that the only change between RC3 and RC4 is to remove the
>> > > binary test jars from the source code, which should make the review a
>> > > little easier for those that already reviewed the previous release
>> > > candidate.
>> > >
>> > > On Fri, Mar 30, 2018 at 10:28 PM, Luciano Resende <
>> luckbr1...@gmail.com>
>> > > wrote:
>> > >
>> > > > Please vote to approve the release of Apache Toree 0.2.0-incubating
>> > > > (RC4).
>> > > >
>> > > > Tag: v0.2.0-incubating-rc4
>> (8849271e28a4ea4e77c57ef8fb3da77b73469799)
>> > > >
>> > > >
>> https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc4
>> > > >
>> > > > All distribution packages, including signatures, digests, etc. can
>> be
>> > > > found at:
>> > > >
>> > > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0
>> > > > -incubating-rc4/
>> > > >
>> > > > Staging artifacts can be found at:
>> > > >
>> > > >
>> https://repository.apache.org/content/repositories/orgapachetoree-1011
>> > > >
>> > > > ## Testing Instructions
>> > > >
>> > > > The fastest way to get up and running is to using Jupyter.
>> > > >
>> > > > 1. Install Jupyter if you haven't already (
>> http://jupyter.org/install.
>> > > html
>> > > > )
>> > > >
>> > > > 2. Install Apache Toree via `pip install
>> > https://dist.apache.org/repos/
>> > > > dist/dev/incubator/toree/0.2.0-incubating-rc4/toree-pip/toree
>> > > > -0.2.0.tar.gz`
>> > > > followed by `jupyter toree install`
>> > > >
>> > > > - You need to set a valid Apache Spark 2.x home, which can be done
>> via
>> > > > `jupyter toree install --spark_home=/usr/local/spark`
>> > > >
>> > > > - You may need to run with `sudo` for installation permission
>> > > >
>> > > > - For all installation options, run `jupyter toree install
>> --help-all`
>> > > >
>> > > > 4. Run a Jupyter notebook server via `jupyter notebook`
>> > > >
>> > > > - If the notebook portion of Jupyter is not installed but Jupyter
>> is,
>> > > > you can install via `pip install notebook`
>> > > >
>> > > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
>> > under
>> > > > new
>> > > > dev
>> > > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to
>> 100).sum()` in
>> > > the
>> > > > notebook
>> > > >
>> > > > ## Voting Instructions
>> > > >
>> > > > The vote is open for at least 72 hours and passes if a majority of
>> at
>> > > > least
>> > > > 3 +1 PMC votes are cast.
>> > > >
>> > > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating
>> > > > [ ] -1 Do not release this package because ...
>> > > >
>> > > > --
>> > > > Luciano Resende
>> > > > http://twitter.com/lresende1975
>> > > > http://lresende.blogspot.com/
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Luciano Resende
>> > > http://twitter.com/lresende1975
>> > > http://lresende.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Ryan Blue
>> > Software Engineer
>> > Netflix
>> >
>>
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
> --
> regards
> Marius van Niekerk
>
-- 
regards
Marius van Niekerk

Re: [VOTE] Apache Toree 0.2.0 RC3

[Marius van Niekerk]

+1

Ran a bunch of tests with https://github.com/mariusvniekerk/binder-toree using
mybinder.org

On Fri, 19 Jan 2018 at 16:54 Chip Senkbeil <chip.senkb...@gmail.com> wrote:

> I just ran through the usual basic stuff and it looked fine. +1
>
> On Thu, Jan 18, 2018 at 11:03 AM Luciano Resende <luckbr1...@gmail.com>
> wrote:
>
> > Off course, my +1
> >
> > Any others?
> >
> > On Sun, Jan 14, 2018 at 6:30 PM, Luciano Resende <luckbr1...@gmail.com>
> > wrote:
> >
> > > Please vote to approve the release of Apache Toree 0.2.0-incubating
> > >  (RC3).
> > >
> > > Tag: v0.2.0-incubating-rc3 (32bbefa121aafd8713afab81516917234d72d690)
> > >
> > > https://github.com/apache/incubator-toree/tree/v0.2.0-incubating-rc3
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > > found at:
> > >
> > > https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.
> > > 0-incubating-rc3/
> > >
> > > Staging artifacts can be found at:
> > >
> > > https://repository.apache.org/content/repositories/orgapachetoree-1010
> > >
> > > ## Testing Instructions
> > >
> > > The fastest way to get up and running is to using Jupyter.
> > >
> > > 1. Install Jupyter if you haven't already (
> > http://jupyter.org/install.html
> > > )
> > >
> > > 2. Install Apache Toree via `pip install
> https://dist.apache.org/repos/
> > > dist/dev/incubator/toree/0.2.0-incubating-rc3/toree-
> > > pip/toree-0.2.0.tar.gz`
> > > followed by `jupyter toree install`
> > >
> > > - You need to set a valid Apache Spark 2.x home, which can be done via
> > > `jupyter toree install --spark_home=/usr/local/spark`
> > >
> > > - You may need to run with `sudo` for installation permission
> > >
> > > - For all installation options, run `jupyter toree install --help-all`
> > >
> > > 4. Run a Jupyter notebook server via `jupyter notebook`
> > >
> > > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > > you can install via `pip install notebook`
> > >
> > > 5. Create a notebook using "Apache Toree - Scala" from the dropdown
> under
> > > new
> > > dev
> > > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> > the
> > > notebook
> > >
> > > ## Voting Instructions
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.2.0-incubating
> > > [ ] -1 Do not release this package because ...
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>


-- 
regards
Marius van Niekerk

Re: Pyspark Support for magic commands

[Marius van Niekerk]

Matplotlib inline in particular is quite tricky, as it it highly coupled
with ipykernel.

It should be feasible using the agg backend and rendering to a png.

On Mon, Oct 2, 2017, 15:08 Yunli Tang <ta...@us.ibm.com> wrote:

> Hi all,
>
> Our team is using apache toree - pyspark. I noticed that the current
> version of the pyspark kernel does not have magic commands, in particular
> "%matplotlib inline" Will this feature be included in the future? If not,
> is there anyway to show the matplotlib graphs in-line in the jupyter
> notebook?
>
> Lili
>
>
> --
regards
Marius van Niekerk

Re: Toree Release 0.2

[Marius van Niekerk]

I think we should cut a 0.2 release sooner rather than later.  Having Spark
2+ compatibility is a big plus.

On Sun, 2 Jul 2017 at 23:21 Luciano Resende <luckbr1...@gmail.com> wrote:

> There are a few of very interesting open PRs currently under review, and I
> would like to start testing/merging them and prepare for a 0.2 release
> maybe by middle to end of July ?
>
> Thoughts ?
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: Running out of space on Travis.CI build

[Marius van Niekerk]

I read a little and from what it seems this is due to us being assigned to
some overcomitted host.  Apache Toree builds do take a LOT of disk space to
work.

I've reached out to Travis CI support -- we'll see what happens?

--Marius

On Mon, 3 Jul 2017 at 13:25 Luciano Resende <luckbr1...@gmail.com> wrote:

> I am seeing some Travis.CI issues while building PRs related to disk space.
> Have we experienced this in the past ? Any workarounds ?
>
>
> build link:
>
> https://travis-ci.org/apache/incubator-toree/builds/249702416?utm_source=github_status_medium=notification
>
> Removing intermediate container d49f5d682b97
>
> Step 5/6 : RUN cd /tmp && wget -q
>
> http://d3kbcqa49mib13.cloudfront.net/spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6.tgz
> &&
> <http://d3kbcqa49mib13.cloudfront.net/spark-$%7BAPACHE_SPARK_VERSION%7D-bin-hadoop2.6.tgz&;>
> echo "e17d9da4b3ac463ea3ce42289f2a71cefb479d154b1ffd00310c7d7ab207aa2c
> *spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6.tgz" | sha256sum -c - && tar
> xzf spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6.tgz -C /usr/local && rm
> spark-${APACHE_SPARK_VERSION}-bin-hadoop2.6.tgz
>
> ---> Running in 1f2dff3deb22
>
> spark-2.0.0-bin-hadoop2.6.tgz: OK
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/scala-compiler-2.11.8.jar: Wrote only
> 512 of 10240 bytes
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/hk2-api-2.4.0-b34.jar: Cannot write: No
> space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.0.jar: Cannot
> write: No space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/guava-14.0.1.jar: Cannot write: No
> space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/spark-mllib-local_2.11-2.0.0.jar:
> Cannot write: No space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/chill_2.11-0.8.0.jar: Cannot write: No
> space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/scala-library-2.11.8.jar: Cannot write:
> No space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/jars/osgi-resource-locator-1.0.1.jar: Cannot
> write: No space left on device
>
> tar: spark-2.0.0-bin-hadoop2.6/python: Cannot mkdir: No space left on
> device
>
> tar: spark-2.0.0-bin-hadoop2.6/python: Cannot mkdir: No space left on
> device
>
> tar: spark-2.0.0-bin-hadoop2.6/python/run-tests.py: Cannot open: No such
> file or directory
>
> tar: spark-2.0.0-bin-hadoop2.6/python: Cannot mkdir: No space left on
> device
>
> tar: spark-2.0.0-bin-hadoop2.6/python/test_support: Cannot mkdir: No such
> file or directory
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: Uploading PIP for 0.1.0 release

[Marius van Niekerk]

Airflow doesn't, and they are still incubating

-Marius

On Wed, Apr 19, 2017, 18:27 Hitesh Shah <hit...@apache.org> wrote:

> I believe the version should have "incubating" within it.
>
> -- Hitesh
>
> On Mon, Apr 17, 2017 at 1:30 PM, Luciano Resende <luckbr1...@gmail.com>
> wrote:
>
> > Great, thank you.
> >
> > On Mon, Apr 17, 2017 at 12:59 PM, Chip Senkbeil <chip.senkb...@gmail.com
> >
> > wrote:
> >
> > > I just changed everything to use toree again. >_>
> > >
> > > https://pypi.python.org/pypi?:action=display=toree=0.1.0
> > >
> > > I can delete this and undo all of it so we can use apache-toree if
> that's
> > > what you'd rather do.
> > >
> > > On Mon, Apr 17, 2017 at 2:24 PM Luciano Resende <luckbr1...@gmail.com>
> > > wrote:
> > >
> > >> Just to clarify, we are going to upload:
> > >> https://dist.apache.org/repos/dist/release/incubator/toree/
> > >> 0.1.0-incubating/toree-pip/
> > >>
> > >>
> > >> I don't have strong opinions for the name, "apache-toree" is even
> better
> > >> than just toree... but we will need to update docs and other stuff,
> > right ?
> > >>
> > >> On Mon, Apr 17, 2017 at 11:57 AM, Chip Senkbeil <
> > chip.senkb...@gmail.com>
> > >> wrote:
> > >>
> > >>> It looks like all of our makefile work was updated awhile ago to use
> > >>> apache-toree. This must have been from an earlier conversation. I can
> > >>> update our release artifacts for pip to use toree, but we'll have to
> go
> > >>> through the makefile and update it to produce toree in the future if
> > that's
> > >>> the route we want to go.
> > >>>
> > >>> On Mon, Apr 17, 2017 at 1:27 PM Luciano Resende <
> luckbr1...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> I looked into some other Apache projects, I believe we could follow
> > the
> > >>>> same pattern and use "toree". but describe it as "Apache Toree
> > >>>> (incubating)". in the description field.
> > >>>>
> > >>>> On Mon, Apr 17, 2017 at 9:50 AM, Chip Senkbeil <
> > chip.senkb...@gmail.com
> > >>>> > wrote:
> > >>>>
> > >>>>> What is the exact name that you want for the release?
> > >>>>>
> > >>>>> Previously, we were using just "toree", but do you want
> > "apache-toree"
> > >>>>> or "apache-toree-incubating"?
> > >>>>>
> > >>>>> On Mon, Apr 17, 2017 at 10:47 AM Luciano Resende <
> > luckbr1...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> On Mon, Apr 17, 2017 at 8:40 AM, Chip Senkbeil <
> > >>>>>> chip.senkb...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> I've gotten mixed messages regarding where the pip artifacts
> should
> > >>>>>>> go. Originally, we were going to publish them on
> > >>>>>>> https://pypi.python.org/pypi (which we were previously doing for
> > >>>>>>> dev snapshots), but there has been some issues with hosting
> > externally from
> > >>>>>>> Apache? Is that still the case? Can we host official releases on
> > Pypi?
> > >>>>>>>
> > >>>>>>
> > >>>>>> Official releases are ok to go there.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>> If not, we can just indicate that they install via `pip install
> > >>>>>>>
> https://dist.apache.org/repos/dist/release/incubator/toree/0.1.0-
> > >>>>>>> incubating/toree-pip/apache-toree-0.1.0.tar.gz` or whatever the
> > >>>>>>> command is that Gino was mentioning.
> > >>>>>>>
> > >>>>>>> If we want and are allowed to publish on Pypi, I'll do that soon.
> > >>>>>>> Just let me know.
> > >>>>>>>
> > >>>>>>>
> > >>>>>> Yes, please.
> > >>>>>>
> > >>>>>> --
> > >>>>>> Luciano Resende
> > >>>>>> http://twitter.com/lresende1975
> > >>>>>> http://lresende.blogspot.com/
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Luciano Resende
> > >>>> http://twitter.com/lresende1975
> > >>>> http://lresende.blogspot.com/
> > >>>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Luciano Resende
> > >> http://twitter.com/lresende1975
> > >> http://lresende.blogspot.com/
> > >>
> > >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>
-- 
regards
Marius van Niekerk

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936967#comment-15936967
 ] 

Marius Van Niekerk commented on TOREE-395:
--

I'm perfectly fine with it being removed completely.  The degree of pollution 
that spark by default puts in the jupyter log without changing your log4j is 
already intense.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Apache Toree 0.1.0 RC6

[Marius van Niekerk]

+1

Going to +1 this since I assisted with getting rid of some of the bundled
pieces that were blocking for RC5.

On Thu, Feb 16, 2017, 12:14 Chip Senkbeil <chip.senkb...@gmail.com> wrote:

I'll go ahead and give a +1 since I've done a bit of testing using
Scala/SparkR to get this back up to speed.

Also resolved the remaining issues we had from the incubator general vote.
We're still bundling additional license information in the binary release,
but the source release no longer has the extra license info tacked onto the
LICENSE file. I've also fixed the source release such that it is no longer
dependent on being in a Git repository to be built nor does it require you
to have sbt installed.

On Thu, Feb 16, 2017 at 11:09 AM Chip Senkbeil <chip.senkb...@gmail.com>
wrote:

> Please vote to approve the release of the following candidate as Apache
> Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE files
> since this is our first release.
>
> ## Information and Artifacts
>
> The tag to be voted on is v0.1.0-rc6
> (51fa49cb5898e0c5b7824f986382436b969cabc7), located here:
>
>
>
https://github.com/apache/incubator-toree/commit/51fa49cb5898e0c5b7824f986382436b969cabc7
>
> All distribution packages, including signatures, digests, etc. can be
found
> at:
>
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachetoree-1006
>
> ## Testing Instructions
>
> The fastest way to get up and running is to using Jupyter.
>
> 1. Install Jupyter if you haven't already (http://jupyter.org/install.html
> )
>
> 2. Grab the Apache Toree archive from
>
>
https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc6/toree-pip/apache-toree-0.1.0.tar.gz
>
> 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> followed by `jupyter toree install`
>
> - You need to set a valid Apache Spark 1.6.x home, which can be done via
> `jupyter
> toree install --spark_home=/usr/local/spark`
>
> - You may need to run with `sudo` for installation permission
>
> - For all installation options, run `jupyter toree install --help-all`
>
> 4. Run a Jupyter notebook server via `jupyter notebook`
>
> - If the notebook portion of Jupyter is not installed but Jupyter is,
> you can install via `pip install notebook`
>
> 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> new
>
> - If you want other interpreter profiles than Scala, you can change the
> interpreters via `jupyter toree install --interpreters=PySpark,SQL`
>
> 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
the
> notebook
>
> ## Voting Instructions
>
> The vote is open for at least 72 hours and passes if a majority of at
least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.1.0
> [ ] -1 Do not release this package because ...
>

-- 
regards
Marius van Niekerk

Re: Toree for 1.6.x is broken

[Marius van Niekerk]

Cool

On Thu, Feb 16, 2017, 12:00 Chip Senkbeil <chip.senkb...@gmail.com> wrote:

> Looks like there was a missing SparkR::: for connExists. Maybe it became
> public in 2.0, but it was hidden in 1.6. Testing  now before starting vote
> for RC6.
>
> On Thu, Feb 16, 2017 at 10:54 AM Marius van Niekerk <
> marius.v.niek...@gmail.com> wrote:
>
> > The sparkR error might be something i did wrong when backporting the
> spark
> > 2.0 R things
> >
> > On Thu, 16 Feb 2017 at 10:18 Chip Senkbeil <chip.senkb...@gmail.com>
> > wrote:
> >
> > > Turns out I needed to set my hostname on my machine to localhost. Doing
> > so
> > > fixed that error.
> > >
> > > As for the SparkR error, I'll take a look.
> > >
> > > On Thu, Feb 16, 2017 at 9:07 AM Gino Bustelo <g...@bustelos.com>
> wrote:
> > >
> > > > I tried your pip package at
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/
> > > > using
> > > > the `make test-release` target and it worked fine. I ran the magics
> > test
> > > > notebook and everything worked except there is an issue when calling
> > the
> > > > %SparkR magic. It ends up in an infinite loop printing
> > > >
> > > > Warning message:
> > > >
> > > > In rm(".sparkRcon", envir = .sparkREnv) : object '.sparkRcon' not
> found
> > > >
> > > > Error in sparkR.connect() : could not find function "connExists"
> > > >
> > > > Execution halted
> > > >
> > > > 17/02/15 23:15:51 [ERROR] o.a.t.k.i.s.SparkRProcessHandler - null
> > process
> > > > exited: 1
> > > >
> > > > Loading required package: methods
> > > >
> > > >
> > > > Attaching package: ‘SparkR’
> > > >
> > > >
> > > > The following objects are masked from ‘package:stats’:
> > > >
> > > >
> > > > cov, filter, lag, na.omit, predict, sd, var
> > > >
> > > >
> > > > The following objects are masked from ‘package:base’:
> > > >
> > > >
> > > > colnames, colnames<-, endsWith, intersect, rank, rbind, sample,
> > > >
> > > > startsWith, subset, summary, table, transform
> > > >
> > > >
> > > > On Thu, Feb 16, 2017 at 9:03 AM Corey Stubbs <cas5...@gmail.com>
> > wrote:
> > > >
> > > > > I was able to run from the pip package just fine.
> > > > >
> > > > > On Wed, Feb 15, 2017 at 4:50 PM Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > By master, I mean the 0.1.x branch. Was trying to get the next
> vote
> > > > > > started.
> > > > > >
> > > > > > On Wed, Feb 15, 2017, 4:44 PM Chip Senkbeil <
> > chip.senkb...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Just built master, got all of the artifacts ready, blah blah
> > blah.
> > > > > Tested
> > > > > > > by installing the artifact from
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc5/toree-pip/
> > > > > > and
> > > > > > > now it's failing with not being able to bind to an ephemeral
> port
> > > for
> > > > > the
> > > > > > > sparkDriver. Can someone help me take a look at this? I just
> > > > installed
> > > > > > like
> > > > > > > usual via `pip install apache-toree-0.1.0.tar.gz` and pointed
> to
> > > > spark
> > > > > > > distributions I downloaded (1.6.1 and 1.6.3) when running
> > `jupyter
> > > > > toree
> > > > > > > install --spark_home=...`. When launching a kernel, it fails
> > > with...
> > > > > > >
> > > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> > > could
> > > > > not
> > > > > > > bind on port 0. Attempting port 1.
> > > > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDr

Re: Toree for 1.6.x is broken

[Marius van Niekerk]

; > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> could
> > > not
> > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> could
> > > not
> > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> could
> > > not
> > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> could
> > > not
> > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> could
> > > not
> > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [WARN] o.a.s.u.Utils - Service 'sparkDriver'
> could
> > > not
> > > > > bind on port 0. Attempting port 1.
> > > > > 17/02/15 16:41:29 [ERROR] o.a.s.SparkContext - Error initializing
> > > > > SparkContext.
> > > > > java.net.BindException: Can't assign requested address: Service
> > > > > 'sparkDriver' failed after 16 retries!
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:433)
> > > > > at sun.nio.ch.Net.bind(Net.java:425)
> > > > > at
> > > > > sun.nio.ch
> > > > .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> > > > > at sun.nio.ch
> .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
> > > > > at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
> > > > > at
> > > io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> > > > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> > > > > at
> > > > >
> > > >
> > >
> >
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > >
> > >
> >
>
-- 
regards
Marius van Niekerk

Re: Classloader exceptions when testing

[Marius van Niekerk]

That's great news.

I recall some other parts of Akka where migrating the code to spark 2.0 /
scala 2.11 required specifying a bunch of classloaders in the source.

On Sat, 11 Feb 2017 at 17:45 Jakob Odersky <ja...@odersky.com> wrote:

> Hi everyone,
>
> I may have found a fix for the classloader issues I'm having when
> running unit tests
> (https://groups.google.com/forum/#!topic/akka-user/U0mLX3mCmAk).
> Basically the issue appears to happen when a test suite that uses the
> Akka TestKit base class does not define an explicit classloader.
>
> I.e. the following will result in a cryptic error:
>
> class ExecuteRequestHandlerSpec extends TestKit(
>   ActorSystem("ExecuteRequestHandlerSpec")) { //tests }
>
> whereas explicitly defining a classloader for the actor system works fine:
>
> class ExecuteRequestHandlerSpec extends TestKit(
>   ActorSystem("ExecuteRequestHandlerSpec",
> org.apache.toree.Main.getClass.getClassLoader)) { //tests }
>
> I have seen that latter approach used in various tests, however the
> classloaders are not specified consistently in all tests. My proposed
> approach to fixing the tests is to make classloaders explicit
> everywhere (thinking about creating a utility mixin, instead of using
> TestKit directly).
>
> I'm not very well versed in all the class loader magic that happens
> within Spark, Akka and sbt, and would greatly appreciate any input as
> to why the exceptions are raised in the first place and whether my
> proposal is actually a clean fix or just some quick hack.
>
> cheers,
> --Jakob
>
-- 
regards
Marius van Niekerk

Re: Linking JIRA with GitHub

[Marius van Niekerk]

So spark has a whole bunch of scripts that you can run to do the merges and
close things. For something a little lighter we can see if we can adapt the
ones from airflow or arrow since they looked quite nice to me last time I
checked.

On Sat, Feb 11, 2017, 15:30 Jakob Odersky <ja...@odersky.com> wrote:

> Hi,
> does anyone how we can link jira with github, so that pull requests
> with a title of the form [TOREE-X] will close the issue X when merged.
> Basically I'm looking for something similar to the way Spark handles
> issues and pull requests.
>
> cheers,
> --Jakob
>
-- 
regards
Marius van Niekerk

[jira] [Commented] (TOREE-363) Syntax Highlighting Breaks

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857194#comment-15857194
 ] 

Marius Van Niekerk commented on TOREE-363:
--

So i believe we are passing the wrong mimetype to the frontend.

In spylon-kernel (another alternative kernel for scala spark) I pass the 
following and it seems to work

https://github.com/maxpoint/spylon-kernel/blob/master/spylon_kernel/scala_kernel.py#L21-L29

> Syntax Highlighting Breaks
> --
>
> Key: TOREE-363
> URL: https://issues.apache.org/jira/browse/TOREE-363
> Project: TOREE
>  Issue Type: Bug
>Reporter: Aleksei Aleksinov
>Priority: Minor
> Attachments: Screen Shot 2017-01-26 at 18.40.54.png
>
>
> Syntax highlighting breaks after a while of usage. But Scala code execution 
> and Spark jobs work well.
> Affects Toree 0.2.0-dev1
> Jupyter 4.2.1
> Scala 2.12.1 or 2.11.8
> Spark 2.1.0
> Python 3.6



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856905#comment-15856905
 ] 

Marius Van Niekerk commented on TOREE-374:
--

As for actual fixes for the cell repl classes,  I dont really think that is 
possible.  Its certainly out of scope for Apache Toree.  Basically i tend to 
run Toree on some beefier edge nodes, so consequently i allocate at least 8g of 
memory per driver process.  

> Variables declared on the Notebook are not garbage collected
> 
>
> Key: TOREE-374
> URL: https://issues.apache.org/jira/browse/TOREE-374
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: David Taieb
>
> I'm not sure if it's a bug or a limitation of the underlying scala REPL.
> As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) 
> auto-visualization feature within Scala gateway, I have implemented a weak 
> hashmap that tracks objects declared on the Scala REPL. However, I have found 
> that objects are not correctly gc'ed when the object is declared in a cell 
> with a val or var keyword and then the cell is ran again. One would expect 
> that the original object has no more references and should be gc'ed but it's 
> not. 
> However, when the object is declare with var keyword and then set to null in 
> another cell, then it is correctly gc'ed.
> I'm concerned that users who run the same cell multiple times would 
> unwittingly have memory leaks which can eventually lead to OOM errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-374) Variables declared on the Notebook are not garbage collected

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856898#comment-15856898
 ] 

Marius Van Niekerk commented on TOREE-374:
--

So zeppelin has a partial workaround for the valueOfTerm style thing in this

https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L1134



> Variables declared on the Notebook are not garbage collected
> 
>
> Key: TOREE-374
> URL: https://issues.apache.org/jira/browse/TOREE-374
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
>Reporter: David Taieb
>
> I'm not sure if it's a bug or a limitation of the underlying scala REPL.
> As part of supporting PixieDust (https://github.com/ibm-cds-labs/pixiedust) 
> auto-visualization feature within Scala gateway, I have implemented a weak 
> hashmap that tracks objects declared on the Scala REPL. However, I have found 
> that objects are not correctly gc'ed when the object is declared in a cell 
> with a val or var keyword and then the cell is ran again. One would expect 
> that the original object has no more references and should be gc'ed but it's 
> not. 
> However, when the object is declare with var keyword and then set to null in 
> another cell, then it is correctly gc'ed.
> I'm concerned that users who run the same cell multiple times would 
> unwittingly have memory leaks which can eventually lead to OOM errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-375) Incorrect fully qualified name for spark context

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856688#comment-15856688
 ] 

Marius Van Niekerk commented on TOREE-375:
--

Yeah,  zeppelin works around this with a pretty crazy hack, but having 
valueOfTerm work in class repl would be great. 

> Incorrect fully qualified name for spark context
> 
>
> Key: TOREE-375
> URL: https://issues.apache.org/jira/browse/TOREE-375
> Project: TOREE
>  Issue Type: Bug
> Environment: Jupyter Notebook with Toree latest master 
> (1a9c11f5f1381c15b691a716acd0e1f0432a9a35) and Spark 2.0.2, Scala 2.11
>Reporter: Felix Schüler
>
> When running below snippet in a cell I get a compile error for the MLContext 
> Constructor. Somehow the fully qualified name of the SparkContext gets messed 
> up. 
> The same does not happen when I start a Spark shell with the --jars command 
> and create the MLContext there.
> Snippet (the systemml jar is build with the latest master of SystemML):
> {code}
> %addjar 
> file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar
>  -f
> import org.apache.sysml.api.mlcontext._
> import org.apache.sysml.api.mlcontext.ScriptFactory._
> val ml = new MLContext(sc)
> Starting download from 
> file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar
> Finished download of systemml-0.13.0-incubating-SNAPSHOT.jar
> Name: Compile Error
> Message: :25: error: overloaded method constructor MLContext with 
> alternatives:
>   (x$1: 
> org.apache.spark.api.java.JavaSparkContext)org.apache.sysml.api.mlcontext.MLContext
>  
>   (x$1: 
> org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)org.apache.sysml.api.mlcontext.MLContext
>  cannot be applied to 
> (org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)
>val ml = new MLContext(sc)
> ^
> StackTrace: 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Podling Report

[Marius van Niekerk]

Seems that my wiki user no longer works?

Not sure what happened

On Wed, 1 Feb 2017 at 13:00 Marius van Niekerk <marius.v.niek...@gmail.com>
wrote:

> I can take a stab at it.   No real major changes from last time, other
> than removal of the binary pieces
>
> On Wed, Feb 1, 2017, 11:46 Chip Senkbeil <chip.senkb...@gmail.com> wrote:
>
> Looks like the report is due today. Marius, are you available to take care
> of it this month? I don't have time today.
>
> --
> regards
> Marius van Niekerk
>
-- 
regards
Marius van Niekerk

Calling a vote again?

[Marius van Niekerk]

I think we've cleared out the required static jar pieces.  Should we call a
vote again?

-Marius
-- 
regards
Marius van Niekerk

Re: Development model

[Marius van Niekerk]

I think just documenting development workflow and process properly will
help a great deal.
Building Toree is a non-trivial exercise sadly.


On Tue, 31 Jan 2017 at 11:14 Luciano Resende <luckbr1...@gmail.com> wrote:

> On Mon, Jan 30, 2017 at 5:47 PM, Jakob Odersky <ja...@odersky.com> wrote:
>
> > Hi everyone,
> > I was wondering how everyone usually develops toree, specifically how
> > changes to toree are tested with a jupyter notebook? I couldn't find
> > any documentation on the website so I thought I'd ask here.
> >
> > I tried running the various makefile targets, including `make dev` and
> > `make jupyter`, however those targets use a docker image that already
> > contains a published upstream version of toree, which thereby
> > overrides any changes made to the local version. Running "make
> > release", then pip-installing the archive and finally running "jupyter
> > install toree" somehow also uses an upstream version of toree.
> >
> > Is there an easy way to build and install a local source tree?
> >
> > thanks,
> > --Jakob
> >
>
>
> It seems that the project has various, and mostly undocumented, utilities
> done via make. To me this is good, but don't usually follow the patterns of
> projects using a specific build system (e.g. maven, sbt, etc).
>
> IMHO I believe we should be able to conform with the standards of a basic
> sbt project, and enable compile, unit test, package jars and distribution
> and publish locally using vanilla sbt commands. And then use and document
> the make facilities for more complex tasks such as tests that require
> docker, environment setup, build a jupyter env, etc  Which will then make
> it easy for newcomers that are used to sbt to get started while they learn
> about the make magics.
>
> Thoughts ?
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: [VOTE] Apache Toree 0.1.0 RC4

[Marius van Niekerk]

+1

Ran some sample spark notebooks in yarn-client mode against the provided
binaries and a locally built copy.

-Marius van Niekerk


On Wed, 11 Jan 2017 at 00:38 Luciano Resende <luckbr1...@gmail.com> wrote:

> +1
>
> Went over some legal files, verified some parts of the uber jar, built
> locally, etc
>
>
> On Tue, Jan 10, 2017 at 2:15 PM, Chip Senkbeil <chip.senkb...@gmail.com>
> wrote:
>
> > Please vote to approve the release of the following candidate as Apache
> > Toree version 0.1.0. Pay special attention to the LICENSE and NOTICE
> files
> > since this is our first release.
> >
> > ## Information and Artifacts
> >
> > The tag to be voted on is v0.1.0-rc4
> > (1d526954ecaba1d5dc0f40ec555dd598b2b11df7), located here:
> >
> > https://github.com/apache/incubator-toree/commit/
> > 1d526954ecaba1d5dc0f40ec555dd598b2b11df7
> >
> > All distribution packages, including signatures, digests, etc. can be
> found
> > at:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc4/
> >
> > Staging artifacts can be found at:
> >
> > https://repository.apache.org/content/repositories/orgapachetoree-1002/
> >
> > ## Testing Instructions
> >
> > The fastest way to get up and running is to using Jupyter.
> >
> > 1. Install Jupyter if you haven't already (
> http://jupyter.org/install.html
> > )
> >
> > 2. Grab the Apache Toree archive from
> > https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.
> > 0/rc4/toree-pip/apache-toree-0.1.0.tar.gz
> >
> > 3. Install Apache Toree via `pip install apache-toree-0.1.0.tar.gz`
> > followed by `jupyter toree install`
> >
> > - You need to set a valid Spark home, which can be done via `jupyter
> > toree install --spark_home=/usr/local/spark`
> >
> > - You may need to run with `sudo` for installation permission
> >
> > - For all installation options, run `jupyter toree install
> --help-all`
> >
> > 4. Run a Jupyter notebook server via `jupyter notebook`
> >
> > - If the notebook portion of Jupyter is not installed but Jupyter is,
> > you can install via `pip install notebook`
> >
> > 5. Create a notebook using "Apache Toree - Scala" from the dropdown under
> > new
> >
> > - If you want other interpreter profiles than Scala, you can change
> the
> > interpreters via `jupyter toree install --interpreters=PySpark,SQL`
> >
> > 6. Run Scala/Spark commands such as `sc.parallelize(1 to 100).sum()` in
> the
> > notebook
> >
> > ## Voting Instructions
> >
> > The vote is open for at least 72 hours and passes if a majority of at
> least
> > 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Toree 0.1.0
> > [ ] -1 Do not release this package because ...
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
-- 
regards
Marius van Niekerk

Re: Should we start another release vote?

[Marius van Niekerk]

+1 looks good to me. Just some SBT changes

On Thu, Jan 5, 2017, 08:08 Luciano Resende <luckbr1...@gmail.com> wrote:

> + 1
>
> On Wed, Jan 4, 2017 at 6:12 AM Chip Senkbeil <chip.senkb...@gmail.com>
> wrote:
>
> > This is long overdue, but since the artifacts weren't in staging at the
> >
> > time of the vote for RC3, should we have another release vote?
> >
> > --
> Sent from my Mobile device
>
-- 
regards
Marius van Niekerk

Re: [jira] [Created] (TOREE-354) Scala Error with Apache Spark when run in Jupyter

[Marius van Niekerk]

List.foreach(List.scala:381)
> at
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
> at scala.App$class.main(App.scala:76)
> at org.apache.toree.Main$.main(Main.scala:24)
> at org.apache.toree.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
> at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> [W 03:15:26.738 NotebookApp] Timeout waiting for kernel_info reply from
> 94a63354-d294-4de7-a12c-2e05905e0c45
> {noformat}
>
> When running Scala shell, this is my output logs
>
> {noformat}
> root@ubuntu-2gb-sgp1-01:~# spark-shell
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel).
> 16/11/20 03:17:11 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/11/20 03:17:12 WARN Utils: Your hostname, ubuntu-2gb-sgp1-01 resolves
> to a loopback address: 127.0.1.1; using 10.15.0.5 instead (on interface
> eth0)
> 16/11/20 03:17:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
> 16/11/20 03:17:13 WARN SparkContext: Use an existing SparkContext, some
> configuration may not take effect.
> Spark context Web UI available at http://10.15.0.5:4040
> Spark context available as 'sc' (master = local[*], app id =
> local-1479611833426).
> Spark session available as 'spark'.
> Welcome to
>     __
>  / __/__  ___ _/ /__
> _\ \/ _ \/ _ `/ __/  '_/
>/___/ .__/\_,_/_/ /_/\_\   version 2.0.2
>   /_/
>
> Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_111)
> Type in expressions to have them evaluated.
> Type :help for more information.
>
> scala>
> {noformat}
>
> This problem was highlighted before in jira
> https://issues.apache.org/jira/browse/TOREE-336 . However, I'm still
> unable to get it working for some reason.
>
> I followed the instructions listed on their official site.
> https://toree.apache.org/documentation/user/quick-start
>
> This is my path variables
> {noformat}
> scala> root@ubuntu-2gb-sgp1-01:~# echo $PATH
>
> /root/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/spark:/usr/local/spark/bin
> {noformat}
>
> Thanks and much appreciated.
>
>
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
-- 
regards
Marius van Niekerk

Re: Issue when running toree, with jupyterhub, and ambari

[Marius van Niekerk]

So those are typically run when you are trying to discover nodes when
running under Yarn.

It shouldn't have anything toree specific with it. You're probably just
using a version of python by default that doesn't cause errors.

On Thu, Nov 17, 2016, 16:02 <ian.malo...@tdameritrade.com> wrote:

> I¹d prefer not to change those scripts, that¹s the issue. I¹m wondering
> why toree is running them, but not my pyspark notebook.
>
>
> Ian Maloney
> Platform Architect
> Advanced Analytics
> Internal: 828716
> Office: (734) 623-8716
> Mobile: (313) 910-9272
>
>
>
>
>
>
>
>
> On 11/17/16, 3:15 PM, "Marius van Niekerk" <marius.v.niek...@gmail.com>
> wrote:
>
> >So for the topology files are read by /usr/bin/env Python .  You can
> >change
> >it to point at the system Python or make those scripts py2 and 3
> >compatible
> >
> >On Thu, Nov 17, 2016, 13:41 <ian.malo...@tdameritrade.com> wrote:
> >
> >> Hi,
> >>
> >> I¹m experiencing a strange issue when running a toree kernel with
> >> jupyterhub. The python version used for spark in the kernel.json is
> >>2.7, I
> >> verified that in the notebook itself, but in the jupyterhub logs, I see
> >> errors from two python files, created by ambari:
> >>
> >> /usr/bin/hdp-select
> >> /etc/hadoop/conf/topology_script.py
> >>
> >> The errors come from any code which works in python 2, but not 3, since
> >> ambari needs python 2 to run. Unfortunately, jupyterhub needs python 3.
> >>I¹m
> >> not sure why Toree is using python 3 with these files, instead of the
> >> python 2, specified in the kernel.json. I tested with a pyspark notebook
> >> and did not have the same issue, so it seems to be something related to
> >> toree¹s integration with jupyterhub.
> >>
> >> I¹ve updated the files to handle both python 2 & 3, but they seem to be
> >> recreated when ambari and the cluster restart. I¹m looking for a more
> >> stable long term solution.
> >>
> >> Any Ideas?
> >>
> >> Ian
> >>
> >--
> >regards
> >Marius van Niekerk
>
> --
regards
Marius van Niekerk

Re: Issue when running toree, with jupyterhub, and ambari

[Marius van Niekerk]

So for the topology files are read by /usr/bin/env Python .  You can change
it to point at the system Python or make those scripts py2 and 3 compatible

On Thu, Nov 17, 2016, 13:41 <ian.malo...@tdameritrade.com> wrote:

> Hi,
>
> I’m experiencing a strange issue when running a toree kernel with
> jupyterhub. The python version used for spark in the kernel.json is 2.7, I
> verified that in the notebook itself, but in the jupyterhub logs, I see
> errors from two python files, created by ambari:
>
> /usr/bin/hdp-select
> /etc/hadoop/conf/topology_script.py
>
> The errors come from any code which works in python 2, but not 3, since
> ambari needs python 2 to run. Unfortunately, jupyterhub needs python 3. I’m
> not sure why Toree is using python 3 with these files, instead of the
> python 2, specified in the kernel.json. I tested with a pyspark notebook
> and did not have the same issue, so it seems to be something related to
> toree’s integration with jupyterhub.
>
> I’ve updated the files to handle both python 2 & 3, but they seem to be
> recreated when ambari and the cluster restart. I’m looking for a more
> stable long term solution.
>
> Any Ideas?
>
> Ian
>
-- 
regards
Marius van Niekerk

Re: toree does not work out of the box

[Marius van Niekerk]

>
> unknown resolver sbt-chain
>
>
> unknown resolver sbt-chain
>
>
> unknown resolver sbt-chain
>
>
> unknown resolver sbt-chain
>
>
> unknown resolver null
>
>
> unknown resolver sbt-chain
>
>
>
> :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>
> :: retrieving :: org.apache.spark#spark-submit-parent
>
> confs: [default]
>
> 0 artifacts copied, 82 already retrieved (0kB/17ms)
>
> Exception in thread "main" java.lang.NoSuchMethodError:
>
> scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;
>
> at akka.actor.ActorCell$.(ActorCell.scala:336)
>
> at akka.actor.ActorCell$.(ActorCell.scala)
>
> at akka.actor.RootActorPath.$div(ActorPath.scala:185)
>
> at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:465)
>
> at akka.actor.LocalActorRefProvider.(ActorRefProvider.scala:453)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at
>
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
>
> at scala.util.Try$.apply(Try.scala:192)
>
> at
>
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
>
> at
>
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>
> at
>
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>
> at scala.util.Success.flatMap(Try.scala:231)
>
> at
>
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
>
> at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585)
>
> at akka.actor.ActorSystemImpl.(ActorSystem.scala:578)
>
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
>
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)
>
> at
>
> org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71)
>
> at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:35)
>
> at
>
> org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60)
>
> at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:35)
>
> at
> org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:72)
>
> at org.apache.toree.Main$delayedInit$body.apply(Main.scala:40)
>
> at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
>
> at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
>
> at scala.App$$anonfun$main$1.apply(App.scala:76)
>
> at scala.App$$anonfun$main$1.apply(App.scala:76)
>
> at scala.collection.immutable.List.foreach(List.scala:381)
>
> at
>
> scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
>
> at scala.App$class.main(App.scala:76)
>
> at org.apache.toree.Main$.main(Main.scala:24)
>
> at org.apache.toree.Main.main(Main.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at
>
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> [W 10:57:42.178 NotebookApp] Timeout waiting for kernel_info reply from
> ab31e5cc-068b-4884-ab31-bb0dc4cf8286
>
> So at this point, I don't know how to use Toree.  I tried to build it from
> source but there are no instruction on how to install from source.
>
> Can you guys take a look at this?
>
> Thanks,
> Ryan
>
-- 
regards
Marius van Niekerk

[jira] [Created] (TOREE-352) Handle the `kernel_info_request` jupyter message

[Marius Van Niekerk (JIRA)]

Marius Van Niekerk created TOREE-352:


 Summary: Handle the `kernel_info_request` jupyter message
 Key: TOREE-352
 URL: https://issues.apache.org/jira/browse/TOREE-352
 Project: TOREE
  Issue Type: Improvement
Reporter: Marius Van Niekerk


Details on the message are found here.

https://jupyter-client.readthedocs.io/en/latest/messaging.html#kernel-info



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Podling Report Reminder - November 2016

[Marius van Niekerk]

What is general@incubator ? A Mailing list?

On Tue, 1 Nov 2016 at 20:04 Gino Bustelo <lbust...@gmail.com> wrote:

> Marius... thanks for reminder. If you want to give it go, be my guest.
>
> First, create a wiki user in wiki.apache.org/incubator and then ask to
> get write access on general@incubator.
>
> Take last month report as a template (search for Toree) and create a
> thread hear so that we can all contribute.
>
> Then edit this months wiki page.
>
> Gino B.
>
> > On Nov 1, 2016, at 6:03 PM, Marius van Niekerk <
> mariusvniek...@apache.org> wrote:
> >
> >
> >
> >> On 2016-10-29 11:45 (-0400), Luciano Resende <luckbr1...@gmail.com>
> wrote:
> >> Any volunteers to help with this month report ?
> >>
> >>> On Sat, Oct 29, 2016 at 6:22 AM, <johndam...@apache.org> wrote:
> >>>
> >>> Dear podling,
> >>>
> >>> This email was sent by an automated system on behalf of the Apache
> >>> Incubator PMC. It is an initial reminder to give you plenty of time to
> >>> prepare your quarterly board report.
> >>>
> >>> The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
> >>> The report for your podling will form a part of the Incubator PMC
> >>> report. The Incubator PMC requires your report to be submitted 2 weeks
> >>> before the board meeting, to allow sufficient time for review and
> >>> submission (Wed, November 02).
> >>>
> >>> Please submit your report with sufficient time to allow the Incubator
> >>> PMC, and subsequently board members to review and digest. Again, the
> >>> very latest you should submit your report is 2 weeks prior to the board
> >>> meeting.
> >>>
> >>> Thanks,
> >>>
> >>> The Apache Incubator PMC
> >>>
> >>> Submitting your Report
> >>>
> >>> --
> >>>
> >>> Your report should contain the following:
> >>>
> >>> *   Your project name
> >>> *   A brief description of your project, which assumes no knowledge of
> >>>the project or necessarily of its field
> >>> *   A list of the three most important issues to address in the move
> >>>towards graduation.
> >>> *   Any issues that the Incubator PMC or ASF Board might wish/need to
> be
> >>>aware of
> >>> *   How has the community developed since the last report
> >>> *   How has the project developed since the last report.
> >>>
> >>> This should be appended to the Incubator Wiki page at:
> >>>
> >>> http://wiki.apache.org/incubator/November2016
> >>>
> >>> Note: This is manually populated. You may need to wait a little before
> >>> this page is created from a template.
> >>>
> >>> Mentors
> >>> ---
> >>>
> >>> Mentors should review reports for their project(s) and sign them off on
> >>> the Incubator wiki page. Signing off reports shows that you are
> >>> following the project - projects that are not signed may raise alarms
> >>> for the Incubator PMC.
> >>>
> >>> Incubator PMC
> >>>
> >>
> >>
> >>
> >> --
> >> Luciano Resende
> >> http://twitter.com/lresende1975
> >> http://lresende.blogspot.com/
> >>
> >
> > What is involved with this?
>

Re: [VOTE] Apache Toree 0.1.0 RC3

[Marius van Niekerk]

On 2016-10-28 08:49 (-0400), Marius van Niekerk <marius.v.niek...@gmail.com> 
wrote: 
> +1
> On Thu, Oct 27, 2016 at 10:35 Gino Bustelo <g...@bustelos.com> wrote:
> 
> > Please vote to approve the release of the following candidate as
> > Apache Toree version 0.1.0. Pay special attention to the LICENSE and
> > NOTICE files since this is our first release.
> >
> >
> > The tag to be voted on is v0.1.0-rc3
> > (5bef39490daeda69d98a1e4cf4096103e80be44d)
> >
> > *
> > https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> > <
> > https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> > >*
> >
> > All distribution packages, including signatures, digests, etc. can be
> > found at:
> >
> > *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/
> > <https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/>*
> >
> > The vote is open for at least 72 hours and passes if a majority of at least
> > 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Toree 0.1.0
> > [ ] -1 Do not release this package because ...
> >
> 

Tested with 

make system-test

Deployed to my data science team internally.

[jira] [Commented] (TOREE-345) Spark 2.0.0 Fails when trying to use spark-avro

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620532#comment-15620532
 ] 

Marius Van Niekerk commented on TOREE-345:
--

Hi Hollin, I can help guide you on our gitter channel 
https://gitter.im/ibm-et/spark-kernel.

We probably need to build some unit tests that test spark-avro and other common 
dependencies in particular.

> Spark 2.0.0 Fails when trying to use spark-avro
> ---
>
> Key: TOREE-345
> URL: https://issues.apache.org/jira/browse/TOREE-345
> Project: TOREE
>  Issue Type: Bug
> Environment: Spark 2.0.0, Scala 2.11, Hadoop 2.7, Toree Git SHA: 
> 7c1bfb6df7130477c558e69bbb518b0af364e06a
>Reporter: Hollin Wilkins
>
> When trying to use the spark-avro project to load Avro files from Jupyter, we 
> get errors.
> First:
> {code}
> %AddDeps com.databricks spark-avro_2.11 3.0.1 --transitive --trace
> {code}
> Then try to load an Avro file and show it:
> {code}
> spark.sqlContext.read.format("com.databricks.spark.avro").load("/tmp/test.avro").show()
> {code}
> And we get an error. I will attach the trace as a file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Apache Toree 0.1.0 RC3

[Marius van Niekerk]

+1
On Thu, Oct 27, 2016 at 10:35 Gino Bustelo  wrote:

> Please vote to approve the release of the following candidate as
> Apache Toree version 0.1.0. Pay special attention to the LICENSE and
> NOTICE files since this is our first release.
>
>
> The tag to be voted on is v0.1.0-rc3
> (5bef39490daeda69d98a1e4cf4096103e80be44d)
>
> *
> https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> <
> https://github.com/apache/incubator-toree/commit/5bef39490daeda69d98a1e4cf4096103e80be44d
> >*
>
> All distribution packages, including signatures, digests, etc. can be
> found at:
>
> *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/
> *
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Toree 0.1.0
> [ ] -1 Do not release this package because ...
>

Re: [VOTE] Apache Toree 0.1.0 RC2

[Marius van Niekerk]

I believe we need to make another vote thread.
On Wed, Oct 26, 2016 at 16:18 Gino Bustelo  wrote:

> I address the NOTICE and the KEYS issues above. Put out another RC at
> https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc3/. Do I
> create another VOTE thread here of move it to general?
>
>
> On Thu, Oct 20, 2016 at 8:30 AM Gino Bustelo  wrote:
>
> > Thanks Hitesh... I will address your comments above and put out an RC3
> for
> > vote here.
> >
> > On Tue, Oct 18, 2016 at 11:58 PM Hitesh Shah  wrote:
> >
> > KEYS file should be at the top of the source tree and also inside the
> > https://dist.apache.org/repos/dist/release/incubator/toree/ dir.
> > markdown files support comments so ideally the apache license header
> > should be added.
> > When in doubt, keep the NOTICE file as small as possible. I would
> probably
> > err on the side of keeping it out of the NOTICE file for now and retain
> the
> > copyright line in the LICENSE file. I would recommending adding a couple
> of
> > lines in the incubator vote for folks to review the license/notice files
> > carefully as this is the first release.
> > For the one license per type - not a stop-ship but something to look at
> in
> > the future based on comments from the IPMC vote.
> > test-jars: Any reason they cannot be downloaded as part of the tests and
> > cached? In any case, as long as there is an ALv2 lineage for them, this
> is
> > not a stop-ship for now.
> >
> > thanks
> > — Hitesh
> >
> > > On Oct 17, 2016, at 8:41 AM, Gino Bustelo  wrote:
> > >
> > > I'm working on changes in
> > https://github.com/apache/incubator-toree/pull/75.
> > > I've address most of the issues above.
> > >
> > >> - no KEYS file so no way to verify the gpg signature ( unless one
> > > downloads an unverified key from a public server )
> > > Where do I put this KEYS file?
> > >
> > >> - Something to check on - are the copyrights needed in the NOTICE file
> > or
> > > the LICENSE file? I am not too sure if there are needed in the NOTICE
> > file.
> > > Who can answer this? Is this a stop ship problem?
> > >
> > >> - Most projects tend to have one license file per license type and
> not a
> > > license file per dependency - with the copyrights called out in the
> main
> > > LICENSE file I believe.
> > > We did the initial work to prep for release a while back, so I can't
> > > remember what project we modeled against. Is this a stop ship problem?
> > >
> > >> - bunch of markdown files without a license header
> > > Do we have to add headers to these? Look at markdown files in Spark (
> > >
> >
> https://raw.githubusercontent.com/apache/spark/cff560755244dd4ccb998e0c56e81d2620cd4cff/docs/quick-start.md
> > ),
> > > they have no headers.
> > >
> > >> - there are a bunch of test jars checked into the source. Is there
> ALv2
> > > provenance for all of them (including the sparkr tarball )?
> > > Test jars are part of testing Magic downloading. They are ours.
> > >
> > >
> > > On Thu, Oct 13, 2016 at 6:00 PM Hitesh Shah  wrote:
> > >
> > >> md5/sha sigs look fine. pgp sig also looked good though see below.
> > >>
> > >> Comments:
> > >>   - no KEYS file so no way to verify the gpg signature ( unless one
> > >> downloads an unverified key from a public server )
> > >>   - bin tarball should untar into an apache-toree-incubating or
> > >> apache-toree directory, source tarball untars into ./ - should be
> fixed
> > to
> > >> use a top-level dir IMO.
> > >>   - Something to check on - are the copyrights needed in the NOTICE
> file
> > >> or the LICENSE file? I am not too sure if there are needed in the
> NOTICE
> > >> file.
> > >>   - licenses/LICENSE-jline.txt seems to have some html but not the
> > actual
> > >> license content. Did not look at all the files so folks should
> re-check
> > >> those.
> > >>   - Most projects tend to have one license file per license type and
> not
> > >> a license file per dependency - with the copyrights called out in the
> > main
> > >> LICENSE file I believe.
> > >>   - source tarball seems to have too many licenses. Unless
> > >> jline/scala,asm.ammonite, etc  are bundled into the source tarball,
> > they do
> > >> not need to be called out in the LICENSE and/or NOTICE file.
> > >>   - bunch of markdown files without a license header
> > >>   - there are a bunch of test jars checked into the source. Is there
> > ALv2
> > >> provenance for all of them (including the sparkr tarball )?
> > >>
> > >> Vote thread has a bahir related typo.
> > >>
> > >> thanks
> > >> — Hitesh
> > >>
> > >>
> > >>> On Oct 11, 2016, at 12:16 PM, Gino Bustelo 
> wrote:
> > >>>
> > >>> Please vote to approve the release of the following candidate as
> > >>> Apache Toree version 0.1.0
> > >>>
> > >>>
> > >>> The commit to be voted on is 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> > >>> <
> > >>
> >
> 

[jira] [Commented] (TOREE-349) ClassCastException when reading Avro from another thread (Toree master / Spark 2.0.0)

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583801#comment-15583801
 ] 

Marius Van Niekerk commented on TOREE-349:
--

So do you need to do the same thing for spark-shell?

> ClassCastException when reading Avro from another thread (Toree master / 
> Spark 2.0.0)
> -
>
> Key: TOREE-349
> URL: https://issues.apache.org/jira/browse/TOREE-349
> Project: TOREE
>  Issue Type: Bug
>Reporter: Andrew Kerr
> Attachments: avro-csv-threading.scala.ipynb, run.sh
>
>
> When using Toree (master branch commit 
> e8ecd0623c65ad104045b1797fb27f69b8dfc23f)
> with `--packages=com.databricks:spark-avro_2.11:3.0.1` in `SPARK_OPTS`
> and attempting to load an avro file into a dataframe *in a separate thread*
> then an exception is thrown
> `java.lang.ClassCastException: 
> com.databricks.spark.avro.DefaultSource$SerializableConfiguration cannot be 
> cast to com.databricks.spark.avro.DefaultSource$SerializableConfiguration`
> here
> https://github.com/databricks/spark-avro/blob/v3.0.1/src/main/scala/com/databricks/spark/avro/DefaultSource.scala#L156
> Will attach a Jupyter notebook that illustrates the problem and includes full
> stack trace, with a script showing environment.
> The class that throws the exception `DefaultSource` broadcasts Hadoop config
> and returns an anonymous function that accesses that config. The exception
> occurs when that function is executed and it attempts to access the config.
> This looks like a class loader mismatch problem to me ("Class Identity 
> Crisis").
> With a bit of hacking of `spark-avro` I've seen the class loader for 
> `DefaultSource` when the config is broadcast to be 
> `scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@31ac5411`
> and when the config is read to be
> `org.apache.spark.util.MutableURLClassLoader@3d3fcdb0`
> If a fat jar including `spark-avro` is built and included with `--jars=...`
> then the same problem occurs.
> Interestingly the Spark's included support for CSV uses the same pattern as
> Avro, broadcasting a config, but works as expected as shown in the notebook.
> https://github.com/apache/spark/blob/v2.0.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala#L108
> Avro also works as expected when an application fat jar is built and passed 
> to 
> `spark-submit` without involving Toree.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-349) ClassCastException when reading Avro from another thread (Toree master / Spark 2.0.0)

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575398#comment-15575398
 ] 

Marius Van Niekerk commented on TOREE-349:
--

So for your separate thread do you set the contextClassLoader explicitly ?

You might have to do that to ensure that you use the proper classloader. (the 
org.apache.spark one)

> ClassCastException when reading Avro from another thread (Toree master / 
> Spark 2.0.0)
> -
>
> Key: TOREE-349
> URL: https://issues.apache.org/jira/browse/TOREE-349
> Project: TOREE
>  Issue Type: Bug
>Reporter: Andrew Kerr
> Attachments: avro-csv-threading.scala.ipynb, run.sh
>
>
> When using Toree (master branch commit 
> e8ecd0623c65ad104045b1797fb27f69b8dfc23f)
> with `--packages=com.databricks:spark-avro_2.11:3.0.1` in `SPARK_OPTS`
> and attempting to load an avro file into a dataframe *in a separate thread*
> then an exception is thrown
> `java.lang.ClassCastException: 
> com.databricks.spark.avro.DefaultSource$SerializableConfiguration cannot be 
> cast to com.databricks.spark.avro.DefaultSource$SerializableConfiguration`
> here
> https://github.com/databricks/spark-avro/blob/v3.0.1/src/main/scala/com/databricks/spark/avro/DefaultSource.scala#L156
> Will attach a Jupyter notebook that illustrates the problem and includes full
> stack trace, with a script showing environment.
> The class that throws the exception `DefaultSource` broadcasts Hadoop config
> and returns an anonymous function that accesses that config. The exception
> occurs when that function is executed and it attempts to access the config.
> This looks like a class loader mismatch problem to me ("Class Identity 
> Crisis").
> With a bit of hacking of `spark-avro` I've seen the class loader for 
> `DefaultSource` when the config is broadcast to be 
> `scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@31ac5411`
> and when the config is read to be
> `org.apache.spark.util.MutableURLClassLoader@3d3fcdb0`
> If a fat jar including `spark-avro` is built and included with `--jars=...`
> then the same problem occurs.
> Interestingly the Spark's included support for CSV uses the same pattern as
> Avro, broadcasting a config, but works as expected as shown in the notebook.
> https://github.com/apache/spark/blob/v2.0.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala#L108
> Avro also works as expected when an application fat jar is built and passed 
> to 
> `spark-submit` without involving Toree.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Apache Toree 0.1.0 RC2

[Marius van Niekerk]

+1

On Tue, 11 Oct 2016 at 16:22 Gino Bustelo  wrote:

> Tagged as v0.1.0-rc2 and points to the commit. I was trying to avoid overly
> tagging and reserving them for actual releases. Instead... I was placing a
> commit to mark the RC.
>
> On Tue, Oct 11, 2016 at 3:14 PM Luciano Resende 
> wrote:
>
> > On Tue, Oct 11, 2016 at 12:16 PM, Gino Bustelo 
> wrote:
> >
> > > Please vote to approve the release of the following candidate as
> > > Apache Toree version 0.1.0
> > >
> > >
> > > The commit to be voted on is 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> > >  > > 119bf3e2d1d16986f55802cf2323e8629ea25ef8>
> > >
> > > https://github.com/apache/incubator-toree/tree/
> > > 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> > >  > > 9e478555f7>
> > >
> > > All distribution packages, including signatures, digests, etc. can be
> > > found at:
> > >
> > > *https://dist.apache.org/repos/dist/dev/incubator/toree/0.1.0/rc2/
> > > *
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Toree 0.1.0
> > > [ ] -1 Do not release this package because ...
> > >
> >
> >
> > Note there should be a tag associated with the vote. I am assuming that
> the
> > tag will be based on 119bf3e2d1d16986f55802cf2323e8629ea25ef8
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>

[jira] [Created] (TOREE-342) Switch tab completer to the default used for scala 2.11 console

[Marius Van Niekerk (JIRA)]

Marius Van Niekerk created TOREE-342:


 Summary: Switch tab completer to the default used for scala 2.11 
console
 Key: TOREE-342
 URL: https://issues.apache.org/jira/browse/TOREE-342
 Project: TOREE
  Issue Type: Improvement
 Environment: Scala 2.11
Reporter: Marius Van Niekerk
Priority: Minor


Scala 2.11 adds a new tab completion engine (PresentationCompilerCompleter) 
which should allow much richer tab completion for Toree



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-343) Added support for jupyter message `is_complete_request`

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526458#comment-15526458
 ] 

Marius Van Niekerk commented on TOREE-343:
--

Pull request started in https://github.com/apache/incubator-toree/pull/62

> Added support for jupyter message `is_complete_request`
> ---
>
> Key: TOREE-343
> URL: https://issues.apache.org/jira/browse/TOREE-343
> Project: TOREE
>  Issue Type: Improvement
>        Reporter: Marius Van Niekerk
>
> According to the spec here we need to implement support for 
> is_complete_request in order to get some jupyter clients working properly 
> (notably jupyter console)
> https://github.com/jupyter/jupyter_client/blob/4.4.0/docs/messaging.rst#code-completeness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (TOREE-343) Added support for jupyter message `is_complete_request`

[Marius Van Niekerk (JIRA)]

Marius Van Niekerk created TOREE-343:


 Summary: Added support for jupyter message `is_complete_request`
 Key: TOREE-343
 URL: https://issues.apache.org/jira/browse/TOREE-343
 Project: TOREE
  Issue Type: Improvement
Reporter: Marius Van Niekerk


According to the spec here we need to implement support for is_complete_request 
in order to get some jupyter clients working properly (notably jupyter console)

https://github.com/jupyter/jupyter_client/blob/4.4.0/docs/messaging.rst#code-completeness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-303) jupyter console requires extra blank lines following first command; throws warnings

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526401#comment-15526401
 ] 

Marius Van Niekerk commented on TOREE-303:
--

This is addressed for the Scala kernel by 
https://github.com/apache/incubator-toree/pull/62

> jupyter console requires extra blank lines following first command; throws 
> warnings
> ---
>
> Key: TOREE-303
> URL: https://issues.apache.org/jira/browse/TOREE-303
> Project: TOREE
>  Issue Type: Bug
>Affects Versions: 0.1.0
> Environment: Spark 1.6.1 with Scala 2.11.8
>Reporter: Seth Bromberger
>Priority: Minor
>
> Steps to reproduce: 
> 1) Start toree (`jupyter console --kernel toree_scala`).
> 2) ignore multitude of warnings / info that scroll across screen (how to 
> suppress these?)
> 3) type a command (e.g., `var a = 6`); hit enter
> 4) ignore more warnings / info
> 5) type another command (e.g., `var b = 7`); hit enter
> 6) Observe continuation prompt (`...:`) instead of an evaluation
> 7) hit enter again.
> 8) get result along with warning
> Gist of log: 
> https://gist.github.com/sbromberger/df4c3052255f67e6f32aff4cbf1b9564



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (TOREE-341) Toree scala kernel doesn't handle exceptions correctly

[Marius Van Niekerk (JIRA)]

Marius Van Niekerk created TOREE-341:


 Summary: Toree scala kernel doesn't handle exceptions correctly
 Key: TOREE-341
 URL: https://issues.apache.org/jira/browse/TOREE-341
 Project: TOREE
  Issue Type: Bug
Reporter: Marius Van Niekerk


Currently in master (2016-09-20) toree will swallow exceptions instead of doing 
the right thing and returning them to the notebook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (TOREE-112) Add Scala 2.11.7 support

[Marius Van Niekerk (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485214#comment-15485214
 ] 

Marius Van Niekerk commented on TOREE-112:
--

I think we can close this one now

> Add Scala 2.11.7 support
> 
>
> Key: TOREE-112
> URL: https://issues.apache.org/jira/browse/TOREE-112
> Project: TOREE
>  Issue Type: Bug
>Reporter: elbamos
>Assignee: Gino Bustelo
>
> First off, great job.  Compiles and works very well.
> Do you have suggestions for getting it to compile with 2.11 though?  The 
> akka.zeromq dependency has to get changed, after that though I start getting 
> errors like:  
> [info] Compiling 44 Scala sources to 
> /home/aelberg/spark-kernel/protocol/target/scala-2.11/classes...
> [error] 
> /home/aelberg/spark-kernel/kernel-api/src/main/scala/com/ibm/spark/interpreter/ScalaInterpreter.scala:26:
>  object SparkJLineCompletion is not a member of package org.apache.spark.repl
> [error] import org.apache.spark.repl.{SparkJLineCompletion, SparkIMain}
> [error]^
> [error] 
> /home/aelberg/spark-kernel/kernel-api/src/main/scala/com/ibm/spark/interpreter/ScalaInterpreter.scala:153:
>  value forMSIL is not a member of scala.tools.nsc.Global
> [error] require(!sparkIMain.global.forMSIL) // Only support JavaPlatform
> [error]^
> [error] 
> /home/aelberg/spark-kernel/kernel-api/src/main/scala/com/ibm/spark/interpreter/ScalaInterpreter.scala:366:
>  value isReportingErrors is not a member of 
> scala.tools.nsc.interpreter.SparkIMain
> [error] if (sparkIMain.isReportingErrors)
> [error]^
> [error] 
> /home/aelberg/spark-kernel/kernel-api/src/main/scala/com/ibm/spark/interpreter/ScalaInterpreter.scala:340:
>  type mismatch;
> [error]  found   : Option[Any]
> [error]  required: Option[AnyRef]
> Which I find rather mysterious.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)