incubator-toree-website git commit: Added configuration options for Toree

[lbustelo]

Repository: incubator-toree-website
Updated Branches:
  refs/heads/OverhaulSite 9c556c53e -> 26fb60dea


Added configuration options for Toree


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-toree-website/commit/26fb60de
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-toree-website/tree/26fb60de
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-toree-website/diff/26fb60de

Branch: refs/heads/OverhaulSite
Commit: 26fb60dea7a0e0de7f42c62427ad8b1767a6352a
Parents: 9c556c5
Author: Gino Bustelo 
Authored: Mon Jun 13 17:11:11 2016 -0500
Committer: Gino Bustelo 
Committed: Mon Jun 13 17:11:11 2016 -0500

--
 documentation/user/installation.md | 44 -
 1 file changed, 43 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree-website/blob/26fb60de/documentation/user/installation.md
--
diff --git a/documentation/user/installation.md 
b/documentation/user/installation.md
index 5a9096c..e2df8ad 100644
--- a/documentation/user/installation.md
+++ b/documentation/user/installation.md
@@ -80,7 +80,9 @@ on full configurables, see '--help-all'.
 
 # Configuring Spark
 
-There are two options for setting configuration options for Spark. 
+Toree is started using the `spark-submit` script. All configuration options 
from Spark are consistent with configuring
+a [Spark 
Submit](http://spark.apache.org/docs/latest/submitting-applications.html) job. 
There are two ways of 
+setting configuration options for Spark. 
 
 The first is at install time with the `--spark_opts` command line option.
 
@@ -97,6 +99,46 @@ SPARK_OPTS='--master=local[4]' jupyter notebook
 __Note:__ There is an order of precedence to the configuration options. 
`SPARK_OPTS` will overwrite any values configured in `--spark_opts`.
 
 
+# Configuring Toree
+
+There are some configuration options that are specific to Toree. 
+
+```
+Option   Description  
+--   ---   
  
+--default-interpreterdefault interpreter for the kernel   
+--default-repositories   comma seperated list of additional   
+   repositories to resolve
+--default-repository-credentials comma seperated list of credential   
+   files to use   
+-h, --help   display help information  

+--interpreter-plugin   
ip used to bind sockets  
+--jar-dirdirectory where user added jars are  
+   stored (MUST EXIST)
+--magic-url  path to a magic jar  
+--max-interpreter-threads   total number of worker threads to use
+   to execute code
+--nosparkcontext kernel should not create a spark context  
   
+-v, --versiondisplay version information 
+```
+
+There are two way of setting these configuration options. 
+
+The first is at install time with the `--toree_opts` command line option.
+
+```
+jupyter toree instal --toree_opts='--nosparkcontext'
+```
+
+The second option is configured at run time through the `TOREE_OPTS` 
environment variable.
+
+```
+TOREE_OPTS='--nosparkcontext' jupyter notebook 
+```
+
+__Note:__ There is an order of precedence to the configuration options. 
`TOREE_OPTS` will overwrite any values configured in `--toree_opts`.
+
+
 ## Installing Multiple Kernels
 
 Apache Toree provides support for multiple languages. To enable this you need 
to install the configurations for these

incubator-toree-website git commit: Added 'How it works' content

[lbustelo]

Repository: incubator-toree-website
Updated Branches:
  refs/heads/OverhaulSite 9b329ef1f -> e7ff553dd


Added 'How it works' content


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree-website/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-toree-website/commit/e7ff553d
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-toree-website/tree/e7ff553d
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-toree-website/diff/e7ff553d

Branch: refs/heads/OverhaulSite
Commit: e7ff553ddc9cabf263db3d18eb4ce241e9937233
Parents: 9b329ef
Author: Gino Bustelo 
Authored: Mon Jun 13 15:21:11 2016 -0500
Committer: Gino Bustelo 
Committed: Mon Jun 13 15:23:31 2016 -0500

--
 assets/images/batch_mode.png  | Bin 0 -> 61060 bytes
 assets/images/interactive_mode.png| Bin 0 -> 65268 bytes
 assets/images/toree_spark_gateway.png | Bin 0 -> 74504 bytes
 assets/images/toree_with_notebook.png | Bin 0 -> 52906 bytes
 documentation/user/how-it-works.md|  60 +++--
 5 files changed, 57 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree-website/blob/e7ff553d/assets/images/batch_mode.png
--
diff --git a/assets/images/batch_mode.png b/assets/images/batch_mode.png
new file mode 100644
index 000..18082f3
Binary files /dev/null and b/assets/images/batch_mode.png differ

http://git-wip-us.apache.org/repos/asf/incubator-toree-website/blob/e7ff553d/assets/images/interactive_mode.png
--
diff --git a/assets/images/interactive_mode.png 
b/assets/images/interactive_mode.png
new file mode 100644
index 000..55abbc2
Binary files /dev/null and b/assets/images/interactive_mode.png differ

http://git-wip-us.apache.org/repos/asf/incubator-toree-website/blob/e7ff553d/assets/images/toree_spark_gateway.png
--
diff --git a/assets/images/toree_spark_gateway.png 
b/assets/images/toree_spark_gateway.png
new file mode 100644
index 000..a18daa0
Binary files /dev/null and b/assets/images/toree_spark_gateway.png differ

http://git-wip-us.apache.org/repos/asf/incubator-toree-website/blob/e7ff553d/assets/images/toree_with_notebook.png
--
diff --git a/assets/images/toree_with_notebook.png 
b/assets/images/toree_with_notebook.png
new file mode 100644
index 000..873142c
Binary files /dev/null and b/assets/images/toree_with_notebook.png differ

http://git-wip-us.apache.org/repos/asf/incubator-toree-website/blob/e7ff553d/documentation/user/how-it-works.md
--
diff --git a/documentation/user/how-it-works.md 
b/documentation/user/how-it-works.md
index 86c6472..3aa999a 100644
--- a/documentation/user/how-it-works.md
+++ b/documentation/user/how-it-works.md
@@ -9,7 +9,61 @@ tagline: Apache Project !
 
 {% include JB/setup %}
 
-- Architecture in relation to Jupyter and Spark
-- Links to Jupyter kernel spec
-- Links to keynotes and presentations
+# How it works
+
+Toree provides an interactive programming interface to a Spark Cluster. It's 
API takes in `code` in a variety of 
+languages and executes it. The `code` can perform Spark tasks using the 
provided Spark Context. 
+
+To further understand how Toree works, it is worth exploring the role that it 
plays in several usage scenarios. 
+
+### As a Kernel to Jupyter Notebooks
+
+Toree's primary role is as a [Jupyter](http://jupyter.org/) Kernel. It was 
originally created to add full Spark API 
+support to a Jupyter Notebook using the Scala language. It since has grown to 
also support Python an R. The diagram 
+below shows Toree in relation to a running Jupyter Notebook.
+
+![Toree with Jupyter Notebook](/assets/images/toree_with_notebook.png)
+
+When the user creates a new Notebook and selects Toree, the Notebook server 
launches a new Toree process that is
+configured to connect to a Spark cluster. Once in the Notebook, the user can 
interact with Spark by writing code that
+uses the managed Spark Context instance.
+
+The Notebook server and Toree communicate using the [Jupyter Kernel 
Protocol](https://ipython.org/ipython-doc/3/development/messaging.html). 
+This is a [0MQ](http://zeromq.org/) based protocol that is language agnostic 
and allows for bidirectional communication
+between the client and the kernel (i.e. Toree). This protocol is the __ONLY__ 
network interface for communicating with a 
+Toree process. 
+
+When using Toree within a Jupyter Notebook, these technical details can be 
ignored, but they are very relevant when 
+building custom clients. Several options are 

incubator-toree git commit: [TOREE 249] Enable pip install This commit enables the building, packaging and release of a pip package. The package can be installed via `pip install toree`. This then ena

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 846292233 -> 05faf9f5b


[TOREE 249] Enable pip install
This commit enables the building, packaging and release of a pip package.
The package can be installed via `pip install toree`. This then enables the
kernel installation via `jupyter toree install`.


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/05faf9f5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/05faf9f5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/05faf9f5

Branch: refs/heads/master
Commit: 05faf9f5bd5083b21fa9ede80a09092e0a841829
Parents: 8462922
Author: Corey A. Stubbs 
Authored: Wed Feb 3 10:36:42 2016 -0600
Committer: Corey A. Stubbs 
Committed: Thu Feb 4 10:52:49 2016 -0600

--
 .gitignore|   5 +-
 Makefile  |  72 +
 README.md |  22 --
 Vagrantfile   |   6 +-
 etc/bin/run.sh|  36 +
 etc/bin/toree-kernel  |  36 -
 etc/pip_install/MANIFEST.in   |   3 +
 etc/pip_install/setup.py  |  69 +
 etc/pip_install/toree/__init__.py |  18 +
 etc/pip_install/toree/__main__.py |  21 +
 etc/pip_install/toree/_version.py |  18 +
 etc/pip_install/toree/toreeapp.py | 138 +
 12 files changed, 383 insertions(+), 61 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/05faf9f5/.gitignore
--
diff --git a/.gitignore b/.gitignore
index 12db42c..c5eead5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,7 @@
-
+build/
+**/derby.log
+**/metastore_db
+**/*__pycache__
 **.swp
 target/
 .idea/

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/05faf9f5/Makefile
--
diff --git a/Makefile b/Makefile
index 1e90a4c..fbba6f7 100644
--- a/Makefile
+++ b/Makefile
@@ -15,16 +15,33 @@
 # limitations under the License
 #
 
-.PHONY: help clean clean-dist build dev test test-travis
+.PHONY: help clean clean-dist build dev test test-travis release pip-release 
bin-release
 
-VERSION?=0.1.0
+VERSION?=0.1.0.dev2
+COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
+ifeq (, $(findstring dev, $(VERSION)))
+IS_SNAPSHOT?=false
+else
 IS_SNAPSHOT?=true
-APACHE_SPARK_VERSION?=1.5.1
-
-ifeq ($(IS_SNAPSHOT),true)
 SNAPSHOT:=-SNAPSHOT
 endif
 
+APACHE_SPARK_VERSION?=1.5.1
+IMAGE?=jupyter/pyspark-notebook:2988869079e6
+DOCKER_WORKDIR?=/srv/toree
+DOCKER_ARGS?=
+define DOCKER
+docker run -it --rm \
+   --workdir $(DOCKER_WORKDIR) \
+   -e PYTHONPATH='/srv/toree' \
+   -v `pwd`:/srv/toree $(DOCKER_ARGS)
+endef
+
+define GEN_PIP_PACKAGE_INFO
+printf "__version__ = '$(VERSION)'\n" >> dist/toree/_version.py
+printf "__commit__ = '$(COMMIT)'\n" >> dist/toree/_version.py
+endef
+
 USE_VAGRANT?=
 RUN_PREFIX=$(if $(USE_VAGRANT),vagrant ssh -c "cd $(VM_WORKDIR) && )
 RUN_SUFFIX=$(if $(USE_VAGRANT),")
@@ -33,18 +50,19 @@ RUN=$(RUN_PREFIX)$(1)$(RUN_SUFFIX)
 
 ENV_OPTS:=APACHE_SPARK_VERSION=$(APACHE_SPARK_VERSION) VERSION=$(VERSION) 
IS_SNAPSHOT=$(IS_SNAPSHOT)
 
-FULL_VERSION:=$(VERSION)$(SNAPSHOT)
-ASSEMBLY_JAR:=toree-kernel-assembly-$(FULL_VERSION).jar
+ASSEMBLY_JAR:=toree-kernel-assembly-$(VERSION)$(SNAPSHOT).jar
 
 help:
@echo '  clean - clean build files'
@echo 'dev - starts ipython'
-   @echo '   dist - build a packaged distribution'
+   @echo '   dist - build a directory with contents to package'
@echo '  build - builds assembly'
@echo '   test - run all units'
+   @echo 'release - creates packaged distribution'
+   @echo 'jupyter - starts a Jupyter Notebook with Toree installed'
 
 build-info:
-   @echo '$(ENV_OPTS) $(FULL_VERSION)'
+   @echo '$(ENV_OPTS) $(VERSION)'
 
 clean-dist:
-rm -r dist
@@ -69,17 +87,39 @@ test: VM_WORKDIR=/src/toree-kernel
 test:
$(call RUN,$(ENV_OPTS) sbt compile test)
 
-dist: COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
-dist: VERSION_FILE=dist/toree-kernel/VERSION
+dist: VERSION_FILE=dist/toree/VERSION
 dist: kernel/target/scala-2.10/$(ASSEMBLY_JAR) ${shell find ./etc/bin/*}
-   @mkdir -p dist/toree-kernel/bin dist/toree-kernel/lib
-   @cp -r etc/bin/* dist/toree-kernel/bin/.
-   @cp kernel/target/scala-2.10/$(ASSEMBLY_JAR) dist/toree-kernel/lib/.
-   @echo "VERSION: $(FULL_VERSION)" > $(VERSION_FILE)
+   @mkdir -p dist/toree/bin dist/toree/lib
+   @cp -r etc/bin/* dist/toree/bin/.
+   @cp kernel/target/scala-2.10/$(ASSEMBLY_JAR) dist/toree/lib/.
+  

[1/2] incubator-toree git commit: Added functions to send rich display content to client

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 1d95136f9 -> 085c2974c


Added functions to send rich display content to client


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/8bbd7eed
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/8bbd7eed
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/8bbd7eed

Branch: refs/heads/master
Commit: 8bbd7eedbff1c6a2cf36c195ffbcce8bb2ad293b
Parents: 1d95136
Author: Gino Bustelo 
Authored: Tue Feb 23 09:14:32 2016 -0600
Committer: Gino Bustelo 
Committed: Tue Feb 23 09:14:32 2016 -0600

--
 .gitignore  |  2 +
 .../toree/kernel/api/DisplayMethodsLike.scala   | 50 +
 .../apache/toree/kernel/api/KernelLike.scala|  8 +++
 .../boot/layer/HandlerInitialization.scala  |  2 +
 .../toree/kernel/api/DisplayMethods.scala   | 59 
 .../org/apache/toree/kernel/api/Kernel.scala| 24 
 .../protocol/v5/handler/CommCloseHandler.scala  |  3 +
 .../protocol/v5/handler/CommMsgHandler.scala|  3 +
 .../protocol/v5/handler/CommOpenHandler.scala   |  3 +
 9 files changed, 154 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/8bbd7eed/.gitignore
--
diff --git a/.gitignore b/.gitignore
index c730183..b9c205d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,3 +21,5 @@ dist/
 .example-image
 .ensime
 .ensime_cache/
+
+Untitled*.ipynb

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/8bbd7eed/kernel-api/src/main/scala/org/apache/toree/kernel/api/DisplayMethodsLike.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/kernel/api/DisplayMethodsLike.scala
 
b/kernel-api/src/main/scala/org/apache/toree/kernel/api/DisplayMethodsLike.scala
new file mode 100644
index 000..25a0f97
--- /dev/null
+++ 
b/kernel-api/src/main/scala/org/apache/toree/kernel/api/DisplayMethodsLike.scala
@@ -0,0 +1,50 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
+ */
+
+package org.apache.toree.kernel.api
+
+/**
+ * Represents the methods available to display data of different mime-types 
from the kernel to the
+ * client.
+ */
+trait DisplayMethodsLike {
+
+  /**
+   * Send a display message of the specified mime-type and content
+   * @param mimeType The mime-type of the content (i.e. text/html)
+   * @param data The content to send for display
+   */
+  def content(mimeType: String, data: String): Unit
+
+  /**
+   * Send html content to the client
+   * @param data The content to send for display
+   */
+  def html(data: String): Unit = content("text/html", data)
+
+  /**
+   * Send javascript content to the client
+   * @param data The content to send for display
+   */
+  def javascript(data: String): Unit = content("application/javascript", data)
+
+  /**
+   * Sends a clear-output message to client
+   * @param wait if true, client waits for next display_data to clear
+   */
+  def clear(wait: Boolean = false): Unit
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/8bbd7eed/kernel-api/src/main/scala/org/apache/toree/kernel/api/KernelLike.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/kernel/api/KernelLike.scala 
b/kernel-api/src/main/scala/org/apache/toree/kernel/api/KernelLike.scala
index 9c110c5..af645f2 100644
--- a/kernel-api/src/main/scala/org/apache/toree/kernel/api/KernelLike.scala
+++ b/kernel-api/src/main/scala/org/apache/toree/kernel/api/KernelLike.scala
@@ -60,6 +60,14 @@ trait KernelLike {
   def stream: StreamMethodsLike
 
   /**
+   * Returns a collection of methods that can be used to display data from the
+   * kernel to the client.
+   *
+   * @return The collection of display methods
+   */
+  def display: DisplayMethodsLike
+
+  /**
* 

[39/51] [abbrv] incubator-toree git commit: Changes to license header on all files

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/SparkRTypes.scala
--
diff --git 
a/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/SparkRTypes.scala
 
b/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/SparkRTypes.scala
index 2fd2663..370704f 100644
--- 
a/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/SparkRTypes.scala
+++ 
b/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/SparkRTypes.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
  *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
 package org.apache.toree.kernel.interpreter.sparkr
 

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/package.scala
--
diff --git 
a/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/package.scala
 
b/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/package.scala
index c84de63..e5d3709 100644
--- 
a/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/package.scala
+++ 
b/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/package.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
  *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
 package org.apache.toree.kernel.interpreter
 

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/sparkr-interpreter/src/main/scala/org/apache/toree/magic/builtin/SparkR.scala
--
diff --git 
a/sparkr-interpreter/src/main/scala/org/apache/toree/magic/builtin/SparkR.scala 
b/sparkr-interpreter/src/main/scala/org/apache/toree/magic/builtin/SparkR.scala
index 915be74..9f12973 100644
--- 
a/sparkr-interpreter/src/main/scala/org/apache/toree/magic/builtin/SparkR.scala
+++ 
b/sparkr-interpreter/src/main/scala/org/apache/toree/magic/builtin/SparkR.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * 

[46/51] [abbrv] incubator-toree git commit: Changes to license header on all files

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/kernel-api/src/main/scala/org/apache/toree/global/StreamState.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/global/StreamState.scala 
b/kernel-api/src/main/scala/org/apache/toree/global/StreamState.scala
index 57cdd63..03614ed 100644
--- a/kernel-api/src/main/scala/org/apache/toree/global/StreamState.scala
+++ b/kernel-api/src/main/scala/org/apache/toree/global/StreamState.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2014 IBM Corp.
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
 
 package org.apache.toree.global

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/kernel-api/src/main/scala/org/apache/toree/interpreter/ExecuteFailure.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/interpreter/ExecuteFailure.scala 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/ExecuteFailure.scala
index 0148057..3e13508 100644
--- 
a/kernel-api/src/main/scala/org/apache/toree/interpreter/ExecuteFailure.scala
+++ 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/ExecuteFailure.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
  *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
 
 package org.apache.toree.interpreter

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/kernel-api/src/main/scala/org/apache/toree/interpreter/Interpreter.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/interpreter/Interpreter.scala 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/Interpreter.scala
index 2227fea..9a0ffcb 100644
--- a/kernel-api/src/main/scala/org/apache/toree/interpreter/Interpreter.scala
+++ b/kernel-api/src/main/scala/org/apache/toree/interpreter/Interpreter.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional 

[04/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/InspectRequestSpec.scala
--
diff --git 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/InspectRequestSpec.scala
 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/InspectRequestSpec.scala
new file mode 100644
index 000..2ff561d
--- /dev/null
+++ 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/InspectRequestSpec.scala
@@ -0,0 +1,73 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.content
+
+import org.scalatest.{FunSpec, Matchers}
+import play.api.data.validation.ValidationError
+import play.api.libs.json._
+
+class InspectRequestSpec extends FunSpec with Matchers {
+  val inspectRequestJson: JsValue = Json.parse("""
+  {
+"code": "",
+"cursor_pos": 999,
+"detail_level": 1
+  }
+  """)
+
+  val inspectRequest: InspectRequest = InspectRequest(
+"", 999, 1
+  )
+
+  describe("InspectRequest") {
+describe("#toTypeString") {
+  it("should return correct type") {
+InspectRequest.toTypeString should be ("inspect_request")
+  }
+}
+
+describe("implicit conversions") {
+  it("should implicitly convert from valid json to a InspectRequest 
instance") {
+// This is the least safe way to convert as an error is thrown if it 
fails
+inspectRequestJson.as[InspectRequest] should be (inspectRequest)
+  }
+
+  it("should also work with asOpt") {
+// This is safer, but we lose the error information as it returns
+// None if the conversion fails
+val newInspectRequest = inspectRequestJson.asOpt[InspectRequest]
+
+newInspectRequest.get should be (inspectRequest)
+  }
+
+  it("should also work with validate") {
+// This is the safest as it collects all error information (not just 
first error) and reports it
+val InspectRequestResults = inspectRequestJson.validate[InspectRequest]
+
+InspectRequestResults.fold(
+  (invalid: Seq[(JsPath, Seq[ValidationError])]) => println("Failed!"),
+  (valid: InspectRequest) => valid
+) should be (inspectRequest)
+  }
+
+  it("should implicitly convert from a InspectRequest instance to valid 
json") {
+Json.toJson(inspectRequest) should be (inspectRequestJson)
+  }
+}
+  }
+}
+

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelInfoReplySpec.scala
--
diff --git 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelInfoReplySpec.scala
 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelInfoReplySpec.scala
new file mode 100644
index 000..7d704e2
--- /dev/null
+++ 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelInfoReplySpec.scala
@@ -0,0 +1,76 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.content
+
+import org.scalatest.{FunSpec, Matchers}
+import play.api.data.validation.ValidationError
+import play.api.libs.json._
+
+class KernelInfoReplySpec extends FunSpec with Matchers {
+  val kernelInfoReplyJson: JsValue = Json.parse("""
+  {
+"protocol_version": "x.y.z",
+"implementation": "",
+"implementation_version": "z.y.x",
+"language_info": { "name": "" },
+"language_version": "a.b.c",
+"banner": ""
+  }
+  """)
+
+  val kernelInfoReply: KernelInfoReply = KernelInfoReply(
+"x.y.z", "", "z.y.x", Map("name" -> ""), "a.b.c", 
""
+  )
+
+  describe("KernelInfoReply") {
+

[34/51] [abbrv] incubator-toree git commit: Renamed com.ibm.spark to org.apache.toree

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
--
diff --git 
a/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
 
b/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
index 939b896..248cf56 100644
--- 
a/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
+++ 
b/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
@@ -14,24 +14,24 @@
  * limitations under the License.
  */
 
-package com.ibm.spark.boot.layer
+package org.apache.toree.boot.layer
 
 import java.util
 import java.util.concurrent.ConcurrentHashMap
 
 import akka.actor.ActorRef
-import com.ibm.spark.comm.{CommManager, KernelCommManager, CommRegistrar, 
CommStorage}
-import com.ibm.spark.dependencies.{DependencyDownloader, 
IvyDependencyDownloader}
-import com.ibm.spark.global
-import com.ibm.spark.interpreter._
-import com.ibm.spark.kernel.api.{KernelLike, Kernel}
-import com.ibm.spark.kernel.protocol.v5.KMBuilder
-import com.ibm.spark.kernel.protocol.v5.kernel.ActorLoader
-import com.ibm.spark.kernel.protocol.v5.stream.KernelOutputStream
-import com.ibm.spark.magic.MagicLoader
-import com.ibm.spark.magic.builtin.BuiltinLoader
-import com.ibm.spark.magic.dependencies.DependencyMap
-import com.ibm.spark.utils.{MultiClassLoader, TaskManager, KeyValuePairUtils, 
LogLike}
+import org.apache.toree.comm.{CommManager, KernelCommManager, CommRegistrar, 
CommStorage}
+import org.apache.toree.dependencies.{DependencyDownloader, 
IvyDependencyDownloader}
+import org.apache.toree.global
+import org.apache.toree.interpreter._
+import org.apache.toree.kernel.api.{KernelLike, Kernel}
+import org.apache.toree.kernel.protocol.v5.KMBuilder
+import org.apache.toree.kernel.protocol.v5.kernel.ActorLoader
+import org.apache.toree.kernel.protocol.v5.stream.KernelOutputStream
+import org.apache.toree.magic.MagicLoader
+import org.apache.toree.magic.builtin.BuiltinLoader
+import org.apache.toree.magic.dependencies.DependencyMap
+import org.apache.toree.utils.{MultiClassLoader, TaskManager, 
KeyValuePairUtils, LogLike}
 import com.typesafe.config.Config
 import org.apache.spark.sql.SQLContext
 import org.apache.spark.{SparkContext, SparkConf}
@@ -148,7 +148,7 @@ trait StandardComponentInitialization extends 
ComponentInitialization {
 /*
 interpreter.doQuietly {
   interpreter.bind(
-"kernel", "com.ibm.spark.kernel.api.Kernel",
+"kernel", "org.apache.toree.kernel.api.Kernel",
 kernel, List( """@transient implicit""")
   )
 }

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/kernel/src/main/scala/org/apache/toree/boot/layer/HandlerInitialization.scala
--
diff --git 
a/kernel/src/main/scala/org/apache/toree/boot/layer/HandlerInitialization.scala 
b/kernel/src/main/scala/org/apache/toree/boot/layer/HandlerInitialization.scala
index 1d16ac4..d86f826 100644
--- 
a/kernel/src/main/scala/org/apache/toree/boot/layer/HandlerInitialization.scala
+++ 
b/kernel/src/main/scala/org/apache/toree/boot/layer/HandlerInitialization.scala
@@ -14,23 +14,23 @@
  *  limitations under the License.
  */
 
-package com.ibm.spark.boot.layer
+package org.apache.toree.boot.layer
 
 import akka.actor.{ActorRef, ActorSystem, Props}
-import com.ibm.spark.comm.{CommRegistrar, CommStorage}
-import com.ibm.spark.interpreter.Interpreter
-import com.ibm.spark.kernel.api.Kernel
-import com.ibm.spark.kernel.protocol.v5.MessageType.MessageType
-import com.ibm.spark.kernel.protocol.v5.SocketType.SocketType
-import com.ibm.spark.kernel.protocol.v5.handler._
-import com.ibm.spark.kernel.protocol.v5.interpreter.InterpreterActor
-import 
com.ibm.spark.kernel.protocol.v5.interpreter.tasks.InterpreterTaskFactory
-import com.ibm.spark.kernel.protocol.v5.kernel.ActorLoader
-import com.ibm.spark.kernel.protocol.v5.magic.{MagicParser, PostProcessor}
-import com.ibm.spark.kernel.protocol.v5.relay.ExecuteRequestRelay
-import com.ibm.spark.kernel.protocol.v5.{MessageType, SocketType, 
SystemActorType}
-import com.ibm.spark.magic.MagicLoader
-import com.ibm.spark.utils.LogLike
+import org.apache.toree.comm.{CommRegistrar, CommStorage}
+import org.apache.toree.interpreter.Interpreter
+import org.apache.toree.kernel.api.Kernel
+import org.apache.toree.kernel.protocol.v5.MessageType.MessageType
+import org.apache.toree.kernel.protocol.v5.SocketType.SocketType
+import org.apache.toree.kernel.protocol.v5.handler._
+import org.apache.toree.kernel.protocol.v5.interpreter.InterpreterActor
+import 
org.apache.toree.kernel.protocol.v5.interpreter.tasks.InterpreterTaskFactory
+import org.apache.toree.kernel.protocol.v5.kernel.ActorLoader
+import org.apache.toree.kernel.protocol.v5.magic.{MagicParser, PostProcessor}
+import 

[25/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/communication/src/test/scala/org/apache/toree/communication/socket/ZeroMQSocketRunnableSpec.scala
--
diff --git 
a/communication/src/test/scala/org/apache/toree/communication/socket/ZeroMQSocketRunnableSpec.scala
 
b/communication/src/test/scala/org/apache/toree/communication/socket/ZeroMQSocketRunnableSpec.scala
new file mode 100644
index 000..90c509c
--- /dev/null
+++ 
b/communication/src/test/scala/org/apache/toree/communication/socket/ZeroMQSocketRunnableSpec.scala
@@ -0,0 +1,259 @@
+/*
+ * Copyright 2015 IBM Corp.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package com.ibm.spark.communication.socket
+
+import org.scalatest.concurrent.Eventually
+import org.scalatest.mock.MockitoSugar
+import org.scalatest.time.{Milliseconds, Span}
+import org.scalatest.{BeforeAndAfter, FunSpec, Matchers}
+import org.zeromq.ZMQ
+import org.zeromq.ZMQ.{Socket, Context}
+
+import scala.util.Try
+
+class ZeroMQSocketRunnableSpec extends FunSpec with Matchers
+  with MockitoSugar with Eventually with BeforeAndAfter {
+
+  implicit override val patienceConfig = PatienceConfig(
+timeout = scaled(Span(2000, Milliseconds)),
+interval = scaled(Span(5, Milliseconds))
+  )
+
+  private val TestAddress = "inproc://test-address"
+  private var mockSocketType: SocketType = _
+  private var zmqContext: ZMQ.Context = _
+  private var pubSocket: ZMQ.Socket = _
+
+  private class TestRunnable(
+private val socket: ZMQ.Socket,
+private val context: Context,
+private val socketType: SocketType,
+private val inboundMessageCallback: Option[(Seq[String]) => Unit],
+private val socketOptions: SocketOption*
+  ) extends ZeroMQSocketRunnable(
+context,
+socketType,
+inboundMessageCallback,
+socketOptions: _*
+  ) {
+override protected def newZmqSocket(zmqContext: Context, socketType: Int): 
Socket = socket
+  }
+
+  before {
+mockSocketType = mock[SocketType]
+zmqContext = ZMQ.context(1)
+pubSocket = zmqContext.socket(PubSocket.`type`)
+  }
+
+  after {
+Try(zmqContext.close())
+  }
+
+  describe("ZeroMQSocketRunnable") {
+describe("constructor") {
+  it("should throw an exception if there is no bind or connect") {
+intercept[IllegalArgumentException] {
+  new ZeroMQSocketRunnable(zmqContext, mockSocketType, None)
+}
+pubSocket.close()
+  }
+
+  it("should throw an exception if there is more than one connect") {
+intercept[IllegalArgumentException] {
+  new ZeroMQSocketRunnable(
+zmqContext,
+mockSocketType,
+None,
+Connect(TestAddress),
+Connect(TestAddress)
+  )
+}
+pubSocket.close()
+  }
+
+  it("should throw an exception if there is more than one bind") {
+intercept[IllegalArgumentException] {
+  new ZeroMQSocketRunnable(
+zmqContext,
+mockSocketType,
+None,
+Bind(TestAddress),
+Bind(TestAddress)
+  )
+}
+pubSocket.close()
+  }
+
+  it("should throw an exception if there is a connect and bind") {
+intercept[IllegalArgumentException] {
+  new ZeroMQSocketRunnable(
+zmqContext,
+mockSocketType,
+None,
+Connect(""),
+Bind("")
+  )
+}
+pubSocket.close()
+  }
+}
+
+describe("#run"){
+  it("should set the linger option when provided") {
+val expected = 999
+
+val runnable: TestRunnable = new TestRunnable(
+  pubSocket,
+  zmqContext,
+  PubSocket,
+  None,
+  Connect(TestAddress),
+  Linger(expected)
+)
+val thread = new Thread(runnable)
+
+thread.start()
+
+eventually {
+  val actual = pubSocket.getLinger
+  actual should be (expected)
+}
+
+runnable.close()
+  }
+
+  it("should set the identity option when provided") {
+val expected = "my identity".getBytes(ZMQ.CHARSET)
+
+val runnable: TestRunnable = new TestRunnable(
+  pubSocket,
+  zmqContext,
+  PubSocket,
+  None,
+  Connect(TestAddress),
+  Identity(expected)
+)
+val 

[36/51] [abbrv] incubator-toree git commit: Renamed com.ibm.spark to org.apache.toree

[lbustelo]

Renamed com.ibm.spark to org.apache.toree


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/9612a625
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/9612a625
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/9612a625

Branch: refs/heads/master
Commit: 9612a625ac78c9def7e623238fc2f43842134f17
Parents: 68f7ddd
Author: Gino Bustelo 
Authored: Fri Jan 15 11:13:06 2016 -0600
Committer: Gino Bustelo 
Committed: Fri Jan 15 11:34:05 2016 -0600

--
 .../apache/toree/comm/ClientCommManager.scala   |  8 ++---
 .../apache/toree/comm/ClientCommWriter.scala| 12 +++
 .../kernel/protocol/v5/client/ActorLoader.scala |  2 +-
 .../protocol/v5/client/SparkKernelClient.scala  | 22 ++---
 .../kernel/protocol/v5/client/Utilities.scala   | 10 +++---
 .../v5/client/boot/ClientBootstrap.scala| 12 +++
 .../boot/layers/HandlerInitialization.scala | 12 +++
 .../boot/layers/SystemInitialization.scala  | 14 
 .../v5/client/exception/ShellException.scala|  2 +-
 .../v5/client/execution/DeferredExecution.scala |  6 ++--
 .../execution/DeferredExecutionManager.scala|  6 ++--
 .../execution/DeferredExecutionTuple.scala  |  4 +--
 .../client/execution/ExecuteRequestTuple.scala  |  4 +--
 .../v5/client/handler/ExecuteHandler.scala  | 10 +++---
 .../v5/client/socket/HeartbeatClient.scala  | 10 +++---
 .../protocol/v5/client/socket/IOPubClient.scala | 16 -
 .../protocol/v5/client/socket/ShellClient.scala | 16 -
 .../v5/client/socket/SocketConfig.scala |  2 +-
 .../v5/client/socket/SocketConnection.scala |  2 +-
 .../v5/client/socket/SocketFactory.scala|  4 +--
 .../protocol/v5/client/socket/StdinClient.scala | 16 -
 client/src/test/resources/testng.yaml   |  2 +-
 .../scala/examples/DocumentationExamples.scala  |  8 ++---
 .../scala/examples/ScalaSparkClientUsage.scala  | 10 +++---
 .../ClientToHeartbeatSpecForIntegration.scala   |  8 ++---
 .../ClientToIOPubSpecForIntegration.scala   |  6 ++--
 .../ClientToShellSpecForIntegration.scala   |  6 ++--
 .../toree/comm/ClientCommManagerSpec.scala  | 10 +++---
 .../toree/comm/ClientCommWriterSpec.scala   |  8 ++---
 .../v5/client/SparkKernelClientSpec.scala   | 10 +++---
 .../execution/DeferredExecutionTest.scala   |  8 ++---
 .../v5/client/socket/HeartbeatClientSpec.scala  |  6 ++--
 .../v5/client/socket/IOPubClientSpec.scala  | 18 +--
 .../v5/client/socket/ShellClientSpec.scala  | 12 +++
 .../v5/client/socket/StdinClientSpec.scala  | 16 -
 .../scala/system/ClientCommSpecForSystem.scala  | 10 +++---
 .../scala/test/utils/SparkClientDeployer.scala  | 14 
 .../toree/communication/SocketManager.scala |  4 +--
 .../apache/toree/communication/ZMQMessage.scala |  2 +-
 .../actors/DealerSocketActor.scala  |  6 ++--
 .../communication/actors/PubSocketActor.scala   | 10 +++---
 .../communication/actors/RepSocketActor.scala   |  6 ++--
 .../communication/actors/ReqSocketActor.scala   |  6 ++--
 .../actors/RouterSocketActor.scala  |  6 ++--
 .../communication/actors/SubSocketActor.scala   |  6 ++--
 .../toree/communication/security/Hmac.scala |  4 +--
 .../security/SignatureCheckerActor.scala|  6 ++--
 .../security/SignatureManagerActor.scala|  8 ++---
 .../security/SignatureProducerActor.scala   |  8 ++---
 .../toree/communication/security/package.scala  |  2 +-
 .../communication/socket/JeroMQSocket.scala |  2 +-
 .../socket/PubSocketRunnable.scala  |  2 +-
 .../socket/ReqSocketRunnable.scala  |  2 +-
 .../toree/communication/socket/SocketLike.scala |  2 +-
 .../communication/socket/SocketOption.scala |  2 +-
 .../communication/socket/SocketRunnable.scala   |  2 +-
 .../toree/communication/socket/SocketType.scala |  2 +-
 .../socket/ZeroMQSocketRunnable.scala   |  4 +--
 .../communication/utils/OrderedSupport.scala|  4 +--
 .../JeroMQSocketIntegrationSpec.scala   |  2 +-
 ...ignatureCheckerActorSpecForIntegration.scala |  4 +--
 ...ignatureManagerActorSpecForIntegration.scala |  4 +--
 ...gnatureProducerActorSpecForIntegration.scala |  4 +--
 .../toree/communication/security/HmacSpec.scala |  2 +-
 .../communication/socket/JeroMQSocketSpec.scala |  2 +-
 .../socket/ZeroMQSocketRunnableSpec.scala   |  2 +-
 .../utils/OrderedSupportSpec.scala  |  2 +-
 etc/bin/spark-kernel|  2 +-
 etc/bin/toree-kernel|  2 +-
 .../dependencies/DependencyDownloader.scala |  2 +-
 .../dependencies/IvyDependencyDownloader.scala  |  4 +--
 .../org/apache/toree/global/StreamState.scala   |  2 +-
 .../toree/interpreter/ExecuteFailure.scala  

[01/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/TestBranch [created] 846292233


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/sql-interpreter/src/main/scala/org/apache/toree/magic/builtin/Sql.scala
--
diff --git 
a/sql-interpreter/src/main/scala/org/apache/toree/magic/builtin/Sql.scala 
b/sql-interpreter/src/main/scala/org/apache/toree/magic/builtin/Sql.scala
new file mode 100644
index 000..a8f439c
--- /dev/null
+++ b/sql-interpreter/src/main/scala/org/apache/toree/magic/builtin/Sql.scala
@@ -0,0 +1,53 @@
+/*
+ * Copyright 2015 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.ibm.spark.magic.builtin
+
+import com.ibm.spark.interpreter.{ExecuteError, ExecuteAborted}
+import com.ibm.spark.kernel.interpreter.sql.{SqlInterpreter, SqlException}
+import com.ibm.spark.kernel.protocol.v5.MIMEType
+import com.ibm.spark.magic.{CellMagicOutput, CellMagic}
+import com.ibm.spark.magic.dependencies.IncludeKernel
+
+/**
+ * Represents the magic interface to use the SQL interpreter.
+ */
+class Sql extends CellMagic with IncludeKernel {
+  override def execute(code: String): CellMagicOutput = {
+val sparkR = kernel.interpreter("SQL")
+
+if (sparkR.isEmpty || sparkR.get == null)
+  throw new SqlException("SQL is not available!")
+
+sparkR.get match {
+  case sparkRInterpreter: SqlInterpreter =>
+val (_, output) = sparkRInterpreter.interpret(code)
+output match {
+  case Left(executeOutput) =>
+CellMagicOutput(MIMEType.PlainText -> executeOutput)
+  case Right(executeFailure) => executeFailure match {
+case executeAborted: ExecuteAborted =>
+  throw new SqlException("SQL code was aborted!")
+case executeError: ExecuteError =>
+  throw new SqlException(executeError.value)
+  }
+}
+  case otherInterpreter =>
+val className = otherInterpreter.getClass.getName
+throw new SqlException(s"Invalid SQL interpreter: $className")
+}
+  }
+}
+

[08/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
new file mode 100644
index 000..4ba6888
--- /dev/null
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
@@ -0,0 +1,47 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.content
+
+import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import play.api.libs.json.Json
+
+import scala.language.implicitConversions
+
+case class ErrorContent(
+  ename: String,
+  evalue: String,
+  traceback: List[String]
+) extends KernelMessageContent{
+  override def content : String =
+Json.toJson(this)(ErrorContent.errorContentWrites).toString
+}
+
+object ErrorContent extends TypeString {
+  implicit val errorContentReads = Json.reads[ErrorContent]
+  implicit val errorContentWrites = Json.writes[ErrorContent]
+
+  implicit def ErrorContentToString(errorContent: ErrorContent): String ={
+Json.toJson(errorContent).toString
+  }
+
+  /**
+   * Returns the type string associated with this object.
+   *
+   * @return The type as a string
+   */
+  override def toTypeString: String = "error"
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
new file mode 100644
index 000..a1ec262
--- /dev/null
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
@@ -0,0 +1,40 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.content
+
+import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import play.api.libs.json._
+
+case class ExecuteInput(
+  code: String,
+  execution_count: Int
+) extends KernelMessageContent {
+  override def content : String =
+Json.toJson(this)(ExecuteInput.executeInputWrites).toString
+}
+
+object ExecuteInput extends TypeString {
+  implicit val executeInputReads = Json.reads[ExecuteInput]
+  implicit val executeInputWrites = Json.writes[ExecuteInput]
+
+  /**
+   * Returns the type string associated with this object.
+   *
+   * @return The type as a string
+   */
+  override def toTypeString: String = "execute_input"
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
new file mode 100644
index 000..3462e73
--- /dev/null
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
@@ -0,0 +1,59 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" 

[32/51] [abbrv] incubator-toree git commit: Renamed com.ibm.spark to org.apache.toree

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReply.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReply.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReply.scala
index 6bf986b..41b2d47 100644
--- 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReply.scala
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReply.scala
@@ -14,9 +14,9 @@
  *  limitations under the License.
  */
 
-package com.ibm.spark.kernel.protocol.v5.content
+package org.apache.toree.kernel.protocol.v5.content
 
-import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import org.apache.toree.kernel.protocol.v5.KernelMessageContent
 import play.api.libs.json.Json
 
 case class ShutdownReply(

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequest.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequest.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequest.scala
index 6681169..e965b5c 100644
--- 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequest.scala
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequest.scala
@@ -14,9 +14,9 @@
  * limitations under the License.
  */
 
-package com.ibm.spark.kernel.protocol.v5.content
+package org.apache.toree.kernel.protocol.v5.content
 
-import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import org.apache.toree.kernel.protocol.v5.KernelMessageContent
 import play.api.libs.json.Json
 
 case class ShutdownRequest(

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/StreamContent.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/StreamContent.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/StreamContent.scala
index 12d5366..e818926 100644
--- 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/StreamContent.scala
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/StreamContent.scala
@@ -14,9 +14,9 @@
  * limitations under the License.
  */
 
-package com.ibm.spark.kernel.protocol.v5.content
+package org.apache.toree.kernel.protocol.v5.content
 
-import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import org.apache.toree.kernel.protocol.v5.KernelMessageContent
 import play.api.libs.json._
 
 case class StreamContent(

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/TypeString.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/TypeString.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/TypeString.scala
index e163b28..cf6d43d 100644
--- 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/TypeString.scala
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/TypeString.scala
@@ -14,7 +14,7 @@
  *  limitations under the License.
  */
 
-package com.ibm.spark.kernel.protocol.v5.content
+package org.apache.toree.kernel.protocol.v5.content
 
 /**
  * Indicates that the implementation contains a method to return the type

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/package.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/package.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/package.scala
index ad10884..304ad86 100644
--- 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/package.scala
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/package.scala
@@ -14,7 +14,7 @@
  * limitations under the License.
  */
 
-package com.ibm.spark.kernel.protocol.v5
+package org.apache.toree.kernel.protocol.v5
 
 package object content {
   // Provide an ExecuteReplyOk type and object representing a

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9612a625/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/package.scala
--
diff --git 

[48/51] [abbrv] incubator-toree git commit: Changes to license header on all files

[lbustelo]

Changes to license header on all files


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/c3b736a4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/c3b736a4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/c3b736a4

Branch: refs/heads/TestBranch
Commit: c3b736a413a38fc3746dbfc8ba7db12f65e74ee0
Parents: 39cdd69
Author: Gino Bustelo 
Authored: Fri Jan 15 16:05:53 2016 -0600
Committer: Gino Bustelo 
Committed: Fri Jan 15 16:40:37 2016 -0600

--
 Dockerfile  | 13 +
 Makefile| 13 +
 NOTICE  |  6 ++--
 Vagrantfile | 13 +
 client/build.sbt| 23 
 .../apache/toree/comm/ClientCommManager.scala   | 13 +
 .../apache/toree/comm/ClientCommWriter.scala| 13 +
 .../kernel/protocol/v5/client/ActorLoader.scala | 23 
 .../protocol/v5/client/SparkKernelClient.scala  | 23 
 .../kernel/protocol/v5/client/Utilities.scala   | 23 
 .../v5/client/boot/ClientBootstrap.scala| 23 
 .../boot/layers/HandlerInitialization.scala | 13 +
 .../boot/layers/SystemInitialization.scala  | 13 +
 .../v5/client/exception/ShellException.scala| 23 
 .../v5/client/execution/DeferredExecution.scala | 23 
 .../execution/DeferredExecutionManager.scala| 23 
 .../execution/DeferredExecutionTuple.scala  | 23 
 .../client/execution/ExecuteRequestTuple.scala  | 23 
 .../v5/client/handler/ExecuteHandler.scala  | 23 
 .../v5/client/socket/HeartbeatClient.scala  | 23 
 .../protocol/v5/client/socket/IOPubClient.scala | 23 
 .../protocol/v5/client/socket/ShellClient.scala | 23 
 .../v5/client/socket/SocketConfig.scala | 23 
 .../v5/client/socket/SocketConnection.scala | 23 
 .../v5/client/socket/SocketFactory.scala| 23 
 .../protocol/v5/client/socket/StdinClient.scala | 13 +
 client/src/test/resources/testng.yaml   | 13 +
 .../scala/examples/DocumentationExamples.scala  | 23 
 .../scala/examples/ScalaSparkClientUsage.scala  | 23 
 .../ClientToHeartbeatSpecForIntegration.scala   | 25 +
 .../ClientToIOPubSpecForIntegration.scala   | 25 +
 .../ClientToShellSpecForIntegration.scala   | 25 +
 .../toree/comm/ClientCommManagerSpec.scala  | 13 +
 .../toree/comm/ClientCommWriterSpec.scala   | 15 ++
 .../v5/client/SparkKernelClientSpec.scala   | 23 
 .../execution/DeferredExecutionTest.scala   | 23 
 .../v5/client/socket/HeartbeatClientSpec.scala  | 23 
 .../v5/client/socket/IOPubClientSpec.scala  | 23 
 .../v5/client/socket/ShellClientSpec.scala  | 23 
 .../v5/client/socket/StdinClientSpec.scala  | 13 +
 .../scala/system/ClientCommSpecForSystem.scala  | 13 +
 .../scala/test/utils/SparkClientDeployer.scala  | 13 +
 communication/build.sbt | 23 
 .../toree/communication/SocketManager.scala | 13 +
 .../apache/toree/communication/ZMQMessage.scala | 13 +
 .../actors/DealerSocketActor.scala  | 13 +
 .../communication/actors/PubSocketActor.scala   | 13 +
 .../communication/actors/RepSocketActor.scala   | 13 +
 .../communication/actors/ReqSocketActor.scala   | 13 +
 .../actors/RouterSocketActor.scala  | 13 +
 .../communication/actors/SubSocketActor.scala   | 13 +
 .../toree/communication/security/Hmac.scala | 23 
 .../security/SignatureCheckerActor.scala| 23 
 .../security/SignatureManagerActor.scala| 23 
 .../security/SignatureProducerActor.scala   | 23 
 .../toree/communication/security/package.scala  | 23 
 .../communication/socket/JeroMQSocket.scala | 13 +
 .../socket/PubSocketRunnable.scala  | 13 +
 .../socket/ReqSocketRunnable.scala  | 13 +
 .../toree/communication/socket/SocketLike.scala | 13 +
 .../communication/socket/SocketOption.scala | 13 +
 .../communication/socket/SocketRunnable.scala   | 13 +
 .../toree/communication/socket/SocketType.scala | 13 +
 .../socket/ZeroMQSocketRunnable.scala   | 13 

[15/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel/src/main/scala/org/apache/toree/magic/builtin/LSMagic.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/magic/builtin/LSMagic.scala 
b/kernel/src/main/scala/org/apache/toree/magic/builtin/LSMagic.scala
new file mode 100644
index 000..db99cc1
--- /dev/null
+++ b/kernel/src/main/scala/org/apache/toree/magic/builtin/LSMagic.scala
@@ -0,0 +1,65 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.magic.builtin
+
+import java.io.PrintStream
+
+import com.ibm.spark.magic._
+import com.ibm.spark.magic.dependencies.IncludeOutputStream
+
+class LSMagic extends LineMagic with IncludeOutputStream {
+
+  private lazy val printStream = new PrintStream(outputStream)
+
+  /**
+   * Lists all available magics.
+   * @param code The single line of code
+   * @return The output of the magic
+   */
+  override def execute(code: String): Unit = {
+val classes = new BuiltinLoader().loadClasses().toList
+val lineMagics = magicNames("%", classOf[LineMagic], classes)
+  .mkString(" ").toLowerCase
+val cellMagics = magicNames("%%", classOf[CellMagic], classes)
+  .mkString(" ").toLowerCase
+val message =
+  s"""|Available line magics:
+   |$lineMagics
+   |
+   |Available cell magics:
+   |$cellMagics
+   |
+   |Type % for usage info.
+ """.stripMargin
+
+printStream.println(message)
+  }
+
+  /**
+   * Provides a list of class names from the given list that implement
+   * the specified interface, with the specified prefix prepended.
+   * @param prefix prepended to each name, e.g. "%%"
+   * @param interface a magic interface, e.g. classOf[LineMagic]
+   * @param classes a list of magic classes
+   * @return list of class names with prefix
+   */
+  protected[magic] def magicNames(prefix: String, interface: Class[_],
+  classes: List[Class[_]]) : List[String] = {
+val filteredClasses = classes.filter(_.getInterfaces.contains(interface))
+filteredClasses.map(c => s"${prefix}${c.getSimpleName}")
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel/src/main/scala/org/apache/toree/magic/builtin/RDD.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/magic/builtin/RDD.scala 
b/kernel/src/main/scala/org/apache/toree/magic/builtin/RDD.scala
new file mode 100644
index 000..dbee517
--- /dev/null
+++ b/kernel/src/main/scala/org/apache/toree/magic/builtin/RDD.scala
@@ -0,0 +1,61 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.magic.builtin
+
+import com.ibm.spark.interpreter.{ExecuteFailure, Results, ExecuteAborted, 
ExecuteError}
+import com.ibm.spark.kernel.protocol.v5.MIMEType
+import com.ibm.spark.magic._
+import com.ibm.spark.magic.dependencies.{IncludeKernelInterpreter, 
IncludeInterpreter}
+import com.ibm.spark.utils.LogLike
+import com.ibm.spark.utils.json.RddToJson
+import org.apache.spark.sql.SchemaRDD
+
+/**
+ * Temporary magic to show an RDD as JSON
+ */
+class RDD extends CellMagic with IncludeKernelInterpreter with LogLike {
+
+  private def convertToJson(code: String) = {
+val (result, message) = kernelInterpreter.interpret(code)
+result match {
+  case Results.Success =>
+val rddVarName = 
kernelInterpreter.lastExecutionVariableName.getOrElse("")
+kernelInterpreter.read(rddVarName).map(rddVal => {
+  try{
+CellMagicOutput(MIMEType.ApplicationJson -> 
RddToJson.convert(rddVal.asInstanceOf[SchemaRDD]))
+  } catch {
+case _: Throwable =>
+  CellMagicOutput(MIMEType.PlainText -> s"Could note convert RDD 
to 

[02/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/ReflectiveRBackend.scala
--
diff --git 
a/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/ReflectiveRBackend.scala
 
b/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/ReflectiveRBackend.scala
deleted file mode 100644
index 81cb86e..000
--- 
a/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/ReflectiveRBackend.scala
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *  http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.ibm.spark.kernel.interpreter.sparkr
-
-/**
- * Provides reflective access into the backend R component that is not
- * publically accessible.
- */
-class ReflectiveRBackend {
-  private val rBackendClass = Class.forName("org.apache.spark.api.r.RBackend")
-  private val rBackendInstance = rBackendClass.newInstance()
-
-  /**
-   * Initializes the underlying RBackend service.
-   *
-   * @return The port used by the service
-   */
-  def init(): Int = {
-val runMethod = rBackendClass.getDeclaredMethod("init")
-
-runMethod.invoke(rBackendInstance).asInstanceOf[Int]
-  }
-
-  /** Blocks until the service has finished. */
-  def run(): Unit = {
-val runMethod = rBackendClass.getDeclaredMethod("run")
-
-runMethod.invoke(rBackendInstance)
-  }
-
-  /** Closes the underlying RBackend service. */
-  def close(): Unit = {
-val runMethod = rBackendClass.getDeclaredMethod("close")
-
-runMethod.invoke(rBackendInstance)
-  }
-}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/SparkRBridge.scala
--
diff --git 
a/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/SparkRBridge.scala
 
b/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/SparkRBridge.scala
deleted file mode 100644
index 44fa203..000
--- 
a/sparkr-interpreter/src/main/scala/com/ibm/spark/kernel/interpreter/sparkr/SparkRBridge.scala
+++ /dev/null
@@ -1,78 +0,0 @@
-/*
- * Copyright 2015 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *  http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package com.ibm.spark.kernel.interpreter.sparkr
-
-import com.ibm.spark.interpreter.broker.producer.{StandardSQLContextProducer, 
StandardJavaSparkContextProducer, JavaSparkContextProducerLike, 
SQLContextProducerLike}
-import com.ibm.spark.interpreter.broker.{BrokerState, BrokerBridge}
-import com.ibm.spark.kernel.api.KernelLike
-import org.apache.spark.SparkContext
-
-/**
- * Represents constants for the SparkR bridge.
- */
-object SparkRBridge {
-  /** Represents the maximum amount of code that can be queued for Python. */
-  val MaxQueuedCode = 500
-
-  /** Contains the bridge used by the current R process. */
-  @volatile private var _sparkRBridge: Option[SparkRBridge] = None
-
-  /** Allows kernel to set bridge dynamically. */
-  private[sparkr] def sparkRBridge_=(newSparkRBridge: SparkRBridge): Unit = {
-_sparkRBridge = Some(newSparkRBridge)
-  }
-
-  /** Clears the bridge currently hosted statically. */
-  private[sparkr] def reset(): Unit = _sparkRBridge = None
-
-  /** Must be exposed in a static location for RBackend to access. */
-  def sparkRBridge: SparkRBridge = {
-assert(_sparkRBridge.nonEmpty, "SparkRBridge has not been initialized!")
-_sparkRBridge.get
-  }
-
-  /**
-   * Creates a new SparkRBridge instance.
-   *
-   * @param brokerState The container of broker state to expose
-   * @param kernel The kernel API to expose through the bridge
-   *
-   * @return The new SparkR bridge
-   */
-  def apply(
-brokerState: BrokerState,
-kernel: KernelLike
-): SparkRBridge = {
-new 

[20/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel-api/src/test/scala/org/apache/toree/utils/ScheduledTaskManagerSpec.scala
--
diff --git 
a/kernel-api/src/test/scala/org/apache/toree/utils/ScheduledTaskManagerSpec.scala
 
b/kernel-api/src/test/scala/org/apache/toree/utils/ScheduledTaskManagerSpec.scala
new file mode 100644
index 000..8db4535
--- /dev/null
+++ 
b/kernel-api/src/test/scala/org/apache/toree/utils/ScheduledTaskManagerSpec.scala
@@ -0,0 +1,129 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package com.ibm.spark.utils
+
+import java.util.Calendar
+import java.util.concurrent.atomic.AtomicBoolean
+import java.util.concurrent.{TimeUnit, CountDownLatch}
+
+import org.scalatest.concurrent.Eventually
+import org.scalatest.time.{Milliseconds, Span}
+import org.scalatest.{BeforeAndAfter, Matchers, FunSpec}
+
+class ScheduledTaskManagerSpec extends FunSpec with Matchers with 
BeforeAndAfter
+  with Eventually
+{
+  private val TestTimeInterval = 30
+  private val MaximumChecks = 3
+  private val TimeoutScale = 3
+  private var scheduledTaskManager: ScheduledTaskManager = _
+  private var scheduleVerifier: ScheduleVerifier = _
+
+  implicit override val patienceConfig = PatienceConfig(
+timeout = scaled(Span(
+  TestTimeInterval * MaximumChecks * TimeoutScale, Milliseconds)),
+interval = scaled(Span(TestTimeInterval / 2, Milliseconds))
+  )
+
+  private class ScheduleVerifier {
+@volatile private var checkinTimes: List[Long] = Nil
+private val taskRun = new AtomicBoolean(false)
+
+def task() = {
+  if (checkinTimes.length < MaximumChecks)
+checkinTimes = checkinTimes :+ Calendar.getInstance().getTimeInMillis
+  taskRun.set(true)
+}
+
+def shouldNotRunAnymore(milliseconds: Long) = eventually {
+  val offset: Int = (milliseconds * 0.5).toInt
+
+  // Clean the state and wait to see if the task is executed again
+  taskRun.set(false)
+  Thread.sleep(milliseconds + offset)
+
+  taskRun.get() should be (false)
+}
+
+def shouldRunEvery(milliseconds: Long) = {
+  // 50% +/-
+  val offset: Int = (milliseconds * 0.5).toInt
+
+  eventually {
+// Assert we have the desired number of checks
+checkinTimes.length should be (MaximumChecks)
+
+checkinTimes.take(checkinTimes.length - 1).zip(
+  checkinTimes.takeRight(checkinTimes.length - 1)
+).foreach({ times =>
+  val firstTime = times._1
+  val secondTime = times._2
+  (secondTime - firstTime) should (
+be >= milliseconds - offset and
+  be <= milliseconds + offset)
+})
+  }
+}
+  }
+
+  before {
+scheduledTaskManager = new ScheduledTaskManager
+scheduleVerifier = new ScheduleVerifier
+  }
+
+  after {
+scheduledTaskManager.stop()
+  }
+
+  describe("ScheduledTaskManager") {
+describe("#addTask") {
+  // TODO: This is failing frequently due to some sort of timing problem
+  ignore("should add a new task to be executed periodically") {
+scheduledTaskManager.addTask(timeInterval = TestTimeInterval,
+  task = scheduleVerifier.task())
+
+scheduleVerifier.shouldRunEvery(TestTimeInterval)
+  }
+}
+
+describe("#removeTask") {
+  it("should stop and remove the task if it exists") {
+val taskId = scheduledTaskManager.addTask(
+  timeInterval = TestTimeInterval,
+  task = scheduleVerifier.task()
+)
+
+scheduledTaskManager.removeTask(taskId)
+
+scheduleVerifier.shouldNotRunAnymore(TestTimeInterval)
+  }
+
+  it("should return true if the task was removed") {
+val taskId = scheduledTaskManager.addTask(
+  timeInterval = TestTimeInterval,
+  task = scheduleVerifier.task()
+)
+
+scheduledTaskManager.removeTask(taskId) should be (true)
+  }
+
+  it("should return false if the task does not exist") {
+scheduledTaskManager.removeTask("") should be (false)
+  }
+}
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel-api/src/test/scala/org/apache/toree/utils/TaskManagerSpec.scala
--
diff --git 

[43/51] [abbrv] incubator-toree git commit: Changes to license header on all files

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/kernel/src/test/scala/org/apache/toree/comm/KernelCommManagerSpec.scala
--
diff --git 
a/kernel/src/test/scala/org/apache/toree/comm/KernelCommManagerSpec.scala 
b/kernel/src/test/scala/org/apache/toree/comm/KernelCommManagerSpec.scala
index a8be55f..e7ae22a 100644
--- a/kernel/src/test/scala/org/apache/toree/comm/KernelCommManagerSpec.scala
+++ b/kernel/src/test/scala/org/apache/toree/comm/KernelCommManagerSpec.scala
@@ -1,9 +1,10 @@
 /*
- * Copyright 2014 IBM Corp.
- *
- *  Licensed under the Apache License, Version 2.0 (the "License");
- *  you may not use this file except in compliance with the License.
- *  You may obtain a copy of the License at
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
  *  http://www.apache.org/licenses/LICENSE-2.0
  *
@@ -11,7 +12,7 @@
  *  distributed under the License is distributed on an "AS IS" BASIS,
  *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  *  See the License for the specific language governing permissions and
- *  limitations under the License.
+ *  limitations under the License
  */
 
 package org.apache.toree.comm

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/kernel/src/test/scala/org/apache/toree/comm/KernelCommWriterSpec.scala
--
diff --git 
a/kernel/src/test/scala/org/apache/toree/comm/KernelCommWriterSpec.scala 
b/kernel/src/test/scala/org/apache/toree/comm/KernelCommWriterSpec.scala
index 44989eb..d129574 100644
--- a/kernel/src/test/scala/org/apache/toree/comm/KernelCommWriterSpec.scala
+++ b/kernel/src/test/scala/org/apache/toree/comm/KernelCommWriterSpec.scala
@@ -1,18 +1,21 @@
+
 /*
- * Copyright 2014 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
- * http://www.apache.org/licenses/LICENSE-2.0
+ *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
+
 package org.apache.toree.comm
 
 import java.util.UUID

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/kernel/src/test/scala/org/apache/toree/global/ExecutionCounterSpec.scala
--
diff --git 
a/kernel/src/test/scala/org/apache/toree/global/ExecutionCounterSpec.scala 
b/kernel/src/test/scala/org/apache/toree/global/ExecutionCounterSpec.scala
index c281d15..e22c784 100644
--- a/kernel/src/test/scala/org/apache/toree/global/ExecutionCounterSpec.scala
+++ b/kernel/src/test/scala/org/apache/toree/global/ExecutionCounterSpec.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2014 IBM Corp.
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * 

[23/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel-api/src/main/scala/org/apache/toree/interpreter/Results.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/interpreter/Results.scala 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/Results.scala
new file mode 100644
index 000..8bd12d0
--- /dev/null
+++ b/kernel-api/src/main/scala/org/apache/toree/interpreter/Results.scala
@@ -0,0 +1,38 @@
+/*
+ * Copyright 2015 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.interpreter
+
+/**
+ * Represents interpreter results, mostly taken from the
+ * tools.nsc.interpreter.Results object.
+ */
+object Results {
+  abstract sealed class Result
+
+  /** The line was interpreted successfully. */
+  case object Success extends Result { override def toString = "success" }
+
+  /** The line was erroneous in some way. */
+  case object Error extends Result { override def toString = "error" }
+
+  /** The input was incomplete.  The caller should request more input. */
+  case object Incomplete extends Result { override def toString = "incomplete" 
}
+
+  /** The line was aborted before completed. */
+  case object Aborted extends Result { override def toString = "aborted" }
+}
+

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerBridge.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerBridge.scala
 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerBridge.scala
new file mode 100644
index 000..94b9a24
--- /dev/null
+++ 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerBridge.scala
@@ -0,0 +1,46 @@
+/*
+ * Copyright 2015 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.interpreter.broker
+
+import com.ibm.spark.interpreter.broker.producer.{SQLContextProducerLike, 
JavaSparkContextProducerLike}
+import com.ibm.spark.kernel.api.KernelLike
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.sql.SQLContext
+import org.apache.spark.{SparkConf, SparkContext}
+
+/**
+ * Represents the API available to the broker to act as the bridge for data
+ * between the JVM and some external process.
+ *
+ * @param _brokerState The container of broker state to expose
+ * @param _kernel The kernel API to expose through the bridge
+ */
+class BrokerBridge(
+  private val _brokerState: BrokerState,
+  private val _kernel: KernelLike
+) extends BrokerName {
+  /**
+   * Represents the current state of the broker.
+   */
+  val state: BrokerState = _brokerState
+
+  /**
+   * Represents the kernel API available.
+   */
+  val kernel: KernelLike = _kernel
+}
+

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerCode.scala
--
diff --git 
a/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerCode.scala
 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerCode.scala
new file mode 100644
index 000..e480aa8
--- /dev/null
+++ 
b/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerCode.scala
@@ -0,0 +1,28 @@
+/*
+ * Copyright 2015 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, 

[49/51] [abbrv] incubator-toree git commit: Updates to README to reflect new name and Apache

[lbustelo]

Updates to README to reflect new name and Apache


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/bbece2db
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/bbece2db
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/bbece2db

Branch: refs/heads/master
Commit: bbece2db0d99426392fa1e37c5a22da1f47e08b7
Parents: c3b736a
Author: Gino Bustelo 
Authored: Thu Jan 21 14:32:06 2016 -0600
Committer: Gino Bustelo 
Committed: Thu Jan 21 15:13:22 2016 -0600

--
 README.md | 59 +-
 1 file changed, 34 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/bbece2db/README.md
--
diff --git a/README.md b/README.md
index 38f0a22..fe0e944 100644
--- a/README.md
+++ b/README.md
@@ -2,24 +2,24 @@
 [![License][license-badge]][license-url]
 [![Join the chat at 
https://gitter.im/ibm-et/spark-kernel][gitter-badge]][gitter-url]
 
-Spark Kernel
+Apache Toree
 
-The main goal of the Spark Kernel is to provide the foundation for interactive 
applications to connect to and use [Apache Spark][1].
+The main goal of the Toree is to provide the foundation for interactive 
applications to connect to and use [Apache Spark][1].
 
 Overview
 
-The Spark Kernel provides an interface that allows clients to interact with a 
Spark Cluster. Clients can send libraries and snippets of code that are 
interpreted and ran against a preconfigured Spark context. These snippets can 
do a variety of things:
+Toree provides an interface that allows clients to interact with a Spark 
Cluster. Clients can send libraries and snippets of code that are interpreted 
and ran against a preconfigured Spark context. These snippets can do a variety 
of things:
  1. Define and run spark jobs of all kinds
  2. Collect results from spark and push them to the client
  3. Load necessary dependencies for the running code
  4. Start and monitor a stream
  5. ...
 
-The kernel's main supported language is `Scala`, but it is also capable of 
processing both `Python` and `R`. It implements the latest Jupyter message 
protocol (5.0), so it can easily plug into the 3.x branch of Jupyter/IPython 
for quick, interactive data exploration.
+The main supported language is `Scala`, but it is also capable of processing 
both `Python` and `R`. It implements the latest Jupyter message protocol (5.0), 
so it can easily plug into the latest releases of Jupyter/IPython (3.2.x+ and 
4.x+) for quick, interactive data exploration.
 
 Try It
 ==
-A version of the Spark Kernel is deployed as part of the [Try 
Jupyter!][try-jupyter] site. Select `Scala 2.10.4 (Spark 1.4.1)` under the 
`New` dropdown. Note that this version only supports `Scala`.
+A version of Toree is deployed as part of the [Try Jupyter!][try-jupyter] 
site. Select `Scala 2.10.4 (Spark 1.4.1)` under the `New` dropdown. Note that 
this version only supports `Scala`.
 
 Develop
 ===
@@ -27,12 +27,12 @@ This project uses `make` as the entry point for build, 
test, and packaging. It s
 install `sbt`, `jupyter/ipython`, and other develoment requirements locally on 
your machine. The 2nd mode uses [Vagrant][vagrant] to simplify the development 
experience. In vagrant mode, all commands are sent to the vagrant box 
 that has all necessary dependencies pre-installed. To run in vagrant mode, run 
`export USE_VAGRANT=true`.  
 
-To build and interact with the Spark Kernel using Jupyter, run
+To build and interact with Toree using Jupyter, run
 ```
 make dev
 ```
 
-This will start a Jupyter notebook server. Depending on your mode, it will be 
accessible at `http://localhost:` or `http://192.168.44.44:`. From here 
you can create notebooks that use the Spark Kernel configured for local mode.
+This will start a Jupyter notebook server. Depending on your mode, it will be 
accessible at `http://localhost:` or `http://192.168.44.44:`. From here 
you can create notebooks that use Toree configured for Spark local mode.
 
 Tests can be run by doing `make test`.
 
@@ -40,51 +40,60 @@ Tests can be run by doing `make test`.
 
 Build & Package
 ===
-To build and package up the Spark Kernel, run
+To build and package up Toree, run
 ```
 make dist
 ```
 
-The resulting package of the kernel will be located at 
`./dist/spark-kernel-.tar.gz`. The uncompressed package is what is 
used is ran by Jupyter when doing `make dev`.
+The resulting package of the kernel will be located at 
`./dist//toree-kernel-.tar.gz`. The uncompressed package is what is 
used is ran by Jupyter when doing `make dev`.
 
+Reporting Issues
+
+Refer to and 

[51/51] [abbrv] incubator-toree git commit: Resetting version to start back at 0.1.0

[lbustelo]

Resetting version to start back at 0.1.0


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/84629223
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/84629223
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/84629223

Branch: refs/heads/master
Commit: 846292233c9f61bfc99579ec9a20a9d0ccfb3260
Parents: c7a0db1
Author: Gino Bustelo 
Authored: Thu Jan 21 15:21:23 2016 -0600
Committer: Gino Bustelo 
Committed: Thu Jan 21 15:21:23 2016 -0600

--
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/84629223/Makefile
--
diff --git a/Makefile b/Makefile
index 3d706df..1e90a4c 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
 
 .PHONY: help clean clean-dist build dev test test-travis
 
-VERSION?=0.1.5
+VERSION?=0.1.0
 IS_SNAPSHOT?=true
 APACHE_SPARK_VERSION?=1.5.1
 

[08/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
new file mode 100644
index 000..4ba6888
--- /dev/null
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ErrorContent.scala
@@ -0,0 +1,47 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.content
+
+import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import play.api.libs.json.Json
+
+import scala.language.implicitConversions
+
+case class ErrorContent(
+  ename: String,
+  evalue: String,
+  traceback: List[String]
+) extends KernelMessageContent{
+  override def content : String =
+Json.toJson(this)(ErrorContent.errorContentWrites).toString
+}
+
+object ErrorContent extends TypeString {
+  implicit val errorContentReads = Json.reads[ErrorContent]
+  implicit val errorContentWrites = Json.writes[ErrorContent]
+
+  implicit def ErrorContentToString(errorContent: ErrorContent): String ={
+Json.toJson(errorContent).toString
+  }
+
+  /**
+   * Returns the type string associated with this object.
+   *
+   * @return The type as a string
+   */
+  override def toTypeString: String = "error"
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
new file mode 100644
index 000..a1ec262
--- /dev/null
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteInput.scala
@@ -0,0 +1,40 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.content
+
+import com.ibm.spark.kernel.protocol.v5.KernelMessageContent
+import play.api.libs.json._
+
+case class ExecuteInput(
+  code: String,
+  execution_count: Int
+) extends KernelMessageContent {
+  override def content : String =
+Json.toJson(this)(ExecuteInput.executeInputWrites).toString
+}
+
+object ExecuteInput extends TypeString {
+  implicit val executeInputReads = Json.reads[ExecuteInput]
+  implicit val executeInputWrites = Json.writes[ExecuteInput]
+
+  /**
+   * Returns the type string associated with this object.
+   *
+   * @return The type as a string
+   */
+  override def toTypeString: String = "execute_input"
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
--
diff --git 
a/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
new file mode 100644
index 000..3462e73
--- /dev/null
+++ 
b/protocol/src/main/scala/org/apache/toree/kernel/protocol/v5/content/ExecuteReply.scala
@@ -0,0 +1,59 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" 

[27/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/HeartbeatClientSpec.scala
--
diff --git 
a/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/HeartbeatClientSpec.scala
 
b/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/HeartbeatClientSpec.scala
new file mode 100644
index 000..9fdd702
--- /dev/null
+++ 
b/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/HeartbeatClientSpec.scala
@@ -0,0 +1,48 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.client.socket
+
+import akka.actor.{ActorRef, ActorSystem, Props}
+import akka.testkit.{TestProbe, ImplicitSender, TestKit}
+import com.ibm.spark.communication.ZMQMessage
+import com.ibm.spark.kernel.protocol.v5.client.ActorLoader
+import org.scalatest.mock.MockitoSugar
+import org.scalatest.{Matchers, FunSpecLike}
+import org.mockito.Matchers._
+import org.mockito.Mockito._
+
+class HeartbeatClientSpec extends TestKit(ActorSystem("HeartbeatActorSpec"))
+  with ImplicitSender with FunSpecLike with Matchers with MockitoSugar {
+
+  describe("HeartbeatClientActor") {
+val socketFactory = mock[SocketFactory]
+val mockActorLoader = mock[ActorLoader]
+val probe : TestProbe = TestProbe()
+when(socketFactory.HeartbeatClient(any(classOf[ActorSystem]), 
any(classOf[ActorRef]))).thenReturn(probe.ref)
+
+val heartbeatClient = system.actorOf(Props(
+  classOf[HeartbeatClient], socketFactory, mockActorLoader, true
+))
+
+describe("send heartbeat") {
+  it("should send ping ZMQMessage") {
+heartbeatClient ! HeartbeatMessage
+probe.expectMsgClass(classOf[ZMQMessage])
+  }
+}
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/IOPubClientSpec.scala
--
diff --git 
a/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/IOPubClientSpec.scala
 
b/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/IOPubClientSpec.scala
new file mode 100644
index 000..b592dcd
--- /dev/null
+++ 
b/client/src/test/scala/org/apache/toree/kernel/protocol/v5/client/socket/IOPubClientSpec.scala
@@ -0,0 +1,300 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.client.socket
+
+import java.util.UUID
+
+import akka.actor.{ActorRef, ActorSystem, Props}
+import akka.pattern.ask
+import akka.testkit.{ImplicitSender, TestKit, TestProbe}
+import akka.util.Timeout
+import com.ibm.spark.comm.{CommCallbacks, CommRegistrar, CommStorage, 
CommWriter}
+import com.ibm.spark.communication.ZMQMessage
+import com.ibm.spark.kernel.protocol.v5
+import com.ibm.spark.kernel.protocol.v5._
+import com.ibm.spark.kernel.protocol.v5.client.Utilities._
+import com.ibm.spark.kernel.protocol.v5.client.execution.{DeferredExecution, 
DeferredExecutionManager}
+import com.ibm.spark.kernel.protocol.v5.client.{ActorLoader, Utilities}
+import com.ibm.spark.kernel.protocol.v5.content.{CommClose, CommMsg, CommOpen, 
StreamContent}
+import com.typesafe.config.ConfigFactory
+import org.mockito.Matchers.{eq => mockEq, _}
+import org.mockito.Mockito._
+import org.scalatest.concurrent.{Eventually, ScalaFutures}
+import org.scalatest.mock.MockitoSugar
+import org.scalatest.time.{Milliseconds, Span}
+import org.scalatest.{BeforeAndAfter, FunSpecLike, Matchers}
+import play.api.libs.json.Json
+
+import scala.concurrent.duration._
+import scala.concurrent.{Future, Promise}
+import scala.util.Failure
+
+object IOPubClientSpec {
+  val config ="""
+akka {
+  loglevel = 

[38/51] [abbrv] incubator-toree git commit: Internal rename of object and changes to entrypoint script

[lbustelo]

Internal rename of object and changes to entrypoint script


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/39cdd690
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/39cdd690
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/39cdd690

Branch: refs/heads/master
Commit: 39cdd690ded7f88f23514519a4704852c3ed4106
Parents: ac330ed
Author: Gino Bustelo 
Authored: Fri Jan 15 11:50:29 2016 -0600
Committer: Gino Bustelo 
Committed: Fri Jan 15 11:50:29 2016 -0600

--
 etc/bin/spark-kernel| 35 
 etc/bin/toree-kernel|  2 +-
 .../src/main/scala/org/apache/toree/Main.scala  |  2 +-
 3 files changed, 2 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/39cdd690/etc/bin/spark-kernel
--
diff --git a/etc/bin/spark-kernel b/etc/bin/spark-kernel
deleted file mode 100755
index 836201b..000
--- a/etc/bin/spark-kernel
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-
-#
-# Copyright 2015 IBM Corp.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-   ``
-PROG_HOME="$(cd "`dirname "$0"`"/..; pwd)"
-
-if [ -z "$SPARK_HOME" ]; then
-  echo "SPARK_HOME must be set to the location of a Spark distribution!"
-  exit 1
-fi
-
-echo "Starting Spark Kernel with SPARK_HOME=$SPARK_HOME"
-
-KERNEL_ASSEMBLY=`(cd ${PROG_HOME}/lib; ls -1 kernel-assembly-*.jar;)`
-
-# disable randomized hash for string in Python 3.3+
-export PYTHONHASHSEED=0
-
-exec "$SPARK_HOME"/bin/spark-submit \
-  ${SPARK_OPTS} \
-  --class org.apache.toree.SparkKernel $PROG_HOME/lib/${KERNEL_ASSEMBLY} "$@"

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/39cdd690/etc/bin/toree-kernel
--
diff --git a/etc/bin/toree-kernel b/etc/bin/toree-kernel
index 87b4ca1..71831bf 100755
--- a/etc/bin/toree-kernel
+++ b/etc/bin/toree-kernel
@@ -32,4 +32,4 @@ export PYTHONHASHSEED=0
 
 exec "$SPARK_HOME"/bin/spark-submit \
   ${SPARK_OPTS} \
-  --class org.apache.toree.SparkKernel $PROG_HOME/lib/${KERNEL_ASSEMBLY} "$@"
+  --class org.apache.toree.Main $PROG_HOME/lib/${KERNEL_ASSEMBLY} "$@"

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/39cdd690/kernel/src/main/scala/org/apache/toree/Main.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/Main.scala 
b/kernel/src/main/scala/org/apache/toree/Main.scala
index 1a8ac40..27e914d 100644
--- a/kernel/src/main/scala/org/apache/toree/Main.scala
+++ b/kernel/src/main/scala/org/apache/toree/Main.scala
@@ -20,7 +20,7 @@ import org.apache.toree.boot.layer._
 import org.apache.toree.boot.{CommandLineOptions, KernelBootstrap}
 import org.apache.toree.kernel.BuildInfo
 
-object SparkKernel extends App {
+object Main extends App {
   private val options = new CommandLineOptions(args)
 
   if (options.help) {

[16/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/InterpreterActor.scala
--
diff --git 
a/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/InterpreterActor.scala
 
b/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/InterpreterActor.scala
new file mode 100644
index 000..b5ae174
--- /dev/null
+++ 
b/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/InterpreterActor.scala
@@ -0,0 +1,99 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.ibm.spark.kernel.protocol.v5.interpreter
+
+import java.io.OutputStream
+
+import akka.actor.{Actor, ActorRef, Props}
+import akka.pattern.{ask, pipe}
+import akka.util.Timeout
+import com.ibm.spark.interpreter.Interpreter
+import com.ibm.spark.kernel.protocol.v5.KernelMessage
+import com.ibm.spark.kernel.protocol.v5.interpreter.tasks._
+import com.ibm.spark.kernel.protocol.v5.content._
+import com.ibm.spark.interpreter._
+import com.ibm.spark.utils.LogLike
+
+import scala.concurrent.duration._
+
+object InterpreterActor {
+  def props(interpreter: Interpreter): Props =
+Props(classOf[InterpreterActor], interpreter)
+}
+
+// TODO: Investigate restart sequence
+//
+// http://doc.akka.io/docs/akka/2.2.3/general/supervision.html
+//
+// "create new actor instance by invoking the originally provided factory 
again"
+//
+// Does this mean that the interpreter instance is not gc and is passed in?
+//
+class InterpreterActor(
+  interpreterTaskFactory: InterpreterTaskFactory
+) extends Actor with LogLike {
+  // NOTE: Required to provide the execution context for futures with akka
+  import context._
+
+  // NOTE: Required for ask (?) to function... maybe can define elsewhere?
+  implicit val timeout = Timeout(21474835.seconds)
+
+  //
+  // List of child actors that the interpreter contains
+  //
+  private var executeRequestTask: ActorRef = _
+  private var completeCodeTask: ActorRef = _
+
+  /**
+   * Initializes all child actors performing tasks for the interpreter.
+   */
+  override def preStart = {
+executeRequestTask = interpreterTaskFactory.ExecuteRequestTask(
+  context, InterpreterChildActorType.ExecuteRequestTask.toString)
+completeCodeTask = interpreterTaskFactory.CodeCompleteTask(
+  context, InterpreterChildActorType.CodeCompleteTask.toString)
+  }
+
+  override def receive: Receive = {
+case (executeRequest: ExecuteRequest, parentMessage: KernelMessage,
+  outputStream: OutputStream) =>
+  val data = (executeRequest, parentMessage, outputStream)
+  (executeRequestTask ? data) recover {
+case ex: Throwable =>
+  logger.error(s"Could not execute code ${executeRequest.code} because 
"
++ s"of exception: ${ex.getMessage}")
+  Right(ExecuteError(
+ex.getClass.getName,
+ex.getLocalizedMessage,
+ex.getStackTrace.map(_.toString).toList)
+  )
+  } pipeTo sender
+case (completeRequest: CompleteRequest) =>
+  logger.debug(s"InterpreterActor requesting code completion for code " +
+s"${completeRequest.code}")
+  (completeCodeTask ? completeRequest) recover {
+case ex: Throwable =>
+  logger.error(s"Could not complete code ${completeRequest.code}: " +
+s"${ex.getMessage}")
+  Right(ExecuteError(
+ex.getClass.getName,
+ex.getLocalizedMessage,
+ex.getStackTrace.map(_.toString).toList)
+  )
+  } pipeTo sender
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/package.scala
--
diff --git 
a/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/package.scala
 
b/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/package.scala
new file mode 100644
index 000..6738520
--- /dev/null
+++ 
b/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/interpreter/package.scala
@@ -0,0 +1,26 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of 

[37/51] [abbrv] incubator-toree git commit: Moving SparkKernel entry point to Main

[lbustelo]

Moving SparkKernel entry point to Main


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/ac330ed1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/ac330ed1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/ac330ed1

Branch: refs/heads/master
Commit: ac330ed1de044225bb9651a7315183b31a99226a
Parents: 9612a62
Author: Gino Bustelo 
Authored: Fri Jan 15 11:46:57 2016 -0600
Committer: Gino Bustelo 
Committed: Fri Jan 15 11:46:57 2016 -0600

--
 .../src/main/scala/org/apache/toree/Main.scala  | 43 
 .../scala/org/apache/toree/SparkKernel.scala| 43 
 2 files changed, 43 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/ac330ed1/kernel/src/main/scala/org/apache/toree/Main.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/Main.scala 
b/kernel/src/main/scala/org/apache/toree/Main.scala
new file mode 100644
index 000..1a8ac40
--- /dev/null
+++ b/kernel/src/main/scala/org/apache/toree/Main.scala
@@ -0,0 +1,43 @@
+/*
+ * Copyright 2014 IBM Corp.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.toree
+
+import org.apache.toree.boot.layer._
+import org.apache.toree.boot.{CommandLineOptions, KernelBootstrap}
+import org.apache.toree.kernel.BuildInfo
+
+object SparkKernel extends App {
+  private val options = new CommandLineOptions(args)
+
+  if (options.help) {
+options.printHelpOn(System.out)
+  } else if (options.version) {
+println(s"Kernel Version:   ${BuildInfo.version}")
+println(s"Build Date:   ${BuildInfo.buildDate}")
+println(s"Scala Version:${BuildInfo.scalaVersion}")
+println(s"Apache Spark Version: ${BuildInfo.sparkVersion}")
+  } else {
+(new KernelBootstrap(options.toConfig)
+  with StandardBareInitialization
+  with StandardComponentInitialization
+  with StandardHandlerInitialization
+  with StandardHookInitialization)
+  .initialize()
+  .waitForTermination()
+  .shutdown()
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/ac330ed1/kernel/src/main/scala/org/apache/toree/SparkKernel.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/SparkKernel.scala 
b/kernel/src/main/scala/org/apache/toree/SparkKernel.scala
deleted file mode 100644
index 1a8ac40..000
--- a/kernel/src/main/scala/org/apache/toree/SparkKernel.scala
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Copyright 2014 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.toree
-
-import org.apache.toree.boot.layer._
-import org.apache.toree.boot.{CommandLineOptions, KernelBootstrap}
-import org.apache.toree.kernel.BuildInfo
-
-object SparkKernel extends App {
-  private val options = new CommandLineOptions(args)
-
-  if (options.help) {
-options.printHelpOn(System.out)
-  } else if (options.version) {
-println(s"Kernel Version:   ${BuildInfo.version}")
-println(s"Build Date:   ${BuildInfo.buildDate}")
-println(s"Scala Version:${BuildInfo.scalaVersion}")
-println(s"Apache Spark Version: ${BuildInfo.sparkVersion}")
-  } else {
-(new KernelBootstrap(options.toConfig)
-  with StandardBareInitialization
-  with StandardComponentInitialization
-  with StandardHandlerInitialization
-  with StandardHookInitialization)
-  .initialize()
-  .waitForTermination()
-  .shutdown()
-  }
-}

[21/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel-api/src/test/scala/com/ibm/spark/utils/DynamicReflectionSupportSpec.scala
--
diff --git 
a/kernel-api/src/test/scala/com/ibm/spark/utils/DynamicReflectionSupportSpec.scala
 
b/kernel-api/src/test/scala/com/ibm/spark/utils/DynamicReflectionSupportSpec.scala
deleted file mode 100644
index fdfe637..000
--- 
a/kernel-api/src/test/scala/com/ibm/spark/utils/DynamicReflectionSupportSpec.scala
+++ /dev/null
@@ -1,181 +0,0 @@
-/*
- * Copyright 2014 IBM Corp.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package com.ibm.spark.utils
-
-import java.io.OutputStream
-
-import org.mockito.Matchers._
-import org.mockito.Mockito._
-import org.scalatest.mock.MockitoSugar
-import org.scalatest.{GivenWhenThen, BeforeAndAfter, FunSpec, Matchers}
-
-class DynamicReflectionSupportSpec
-  extends FunSpec with Matchers with MockitoSugar {
-
-  describe("DynamicReflectionSupport") {
-describe("with a class instance") {
-  describe("#selectDynamic") {
-it("should support accessing a normal field") {
-  class MyTestClass {
-val test = 3
-  }
-
-  val x: MyTestClass = new MyTestClass
-
-  val dynamicSupport = DynamicReflectionSupport(x.getClass, x)
-
-  dynamicSupport.test should be (3)
-}
-
-it("should support accessing a method with no arguments") {
-  class MyTestClass {
-def test = 3
-  }
-
-  val x: MyTestClass = new MyTestClass
-
-  val dynamicSupport = DynamicReflectionSupport(x.getClass, x)
-
-  dynamicSupport.test should be (3)
-}
-
-it("should throw an error if the field does not exist") {
-  class MyTestClass
-
-  val x: MyTestClass = new MyTestClass
-
-  val dynamicSupport = DynamicReflectionSupport(x.getClass, x)
-
-  intercept[NoSuchFieldException] {
-dynamicSupport.test
-  }
-}
-  }
-
-  describe("#applyDynamic") {
-it("should support executing a method with one argument") {
-  class MyTestClass {
-def test(x: Int) = x
-  }
-
-  val x: MyTestClass = new MyTestClass
-
-  val dynamicSupport = DynamicReflectionSupport(x.getClass, x)
-
-  dynamicSupport.test(5) should be (5)
-}
-
-it("should support executing a method with multiple arguments") {
-  class MyTestClass {
-def test(x: Int, y: String) = (x, y)
-  }
-
-  val x: MyTestClass = new MyTestClass
-
-  val dynamicSupport = DynamicReflectionSupport(x.getClass, x)
-
-  dynamicSupport.test(5, "test me") should be ((5, "test me"))
-}
-
-it("should throw an error if the method does not exist") {
-  class MyTestClass
-
-  val x: MyTestClass = new MyTestClass
-
-  val dynamicSupport = DynamicReflectionSupport(x.getClass, x)
-
-  intercept[NoSuchMethodException] {
-dynamicSupport.test(5, "test me")
-  }
-}
-  }
-}
-
-describe("with an object") {
-  describe("#selectDynamic") {
-it("should support accessing a normal field") {
-  object MyTestObject {
-val test = 3
-  }
-
-  val dynamicSupport =
-DynamicReflectionSupport(MyTestObject.getClass, MyTestObject)
-
-  dynamicSupport.test should be (3)
-}
-
-it("should support accessing a method with no arguments") {
-  object MyTestObject {
-def test = 3
-  }
-
-  val dynamicSupport =
-DynamicReflectionSupport(MyTestObject.getClass, MyTestObject)
-
-  dynamicSupport.test should be (3)
-}
-
-it("should throw an error if the field does not exist") {
-  object MyTestObject
-
-  val dynamicSupport =
-DynamicReflectionSupport(MyTestObject.getClass, MyTestObject)
-
-  intercept[NoSuchFieldException] {
-dynamicSupport.test
-  }
-}
-  }
-
-  describe("#applyDynamic") {
-it("should support executing a method with one argument") {
-  object MyTestObject {
-def test(x: Int) = x
-  }
-
-  val dynamicSupport =
-DynamicReflectionSupport(MyTestObject.getClass, MyTestObject)
-

[40/51] [abbrv] incubator-toree git commit: Changes to license header on all files

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelStatusSpec.scala
--
diff --git 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelStatusSpec.scala
 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelStatusSpec.scala
index 71fc9af..c354b73 100644
--- 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelStatusSpec.scala
+++ 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/KernelStatusSpec.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2014 IBM Corp.
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
 
 package org.apache.toree.kernel.protocol.v5.content

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReplySpec.scala
--
diff --git 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReplySpec.scala
 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReplySpec.scala
index 6ca748b..13f72c1 100644
--- 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReplySpec.scala
+++ 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownReplySpec.scala
@@ -1,17 +1,18 @@
 /*
- * Copyright 2014 IBM Corp.
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
  *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
+ *  http://www.apache.org/licenses/LICENSE-2.0
  *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
  */
 
 package org.apache.toree.kernel.protocol.v5.content

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/c3b736a4/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequestSpec.scala
--
diff --git 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequestSpec.scala
 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequestSpec.scala
index 5e9fec9..fa4caa6 100644
--- 
a/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequestSpec.scala
+++ 
b/protocol/src/test/scala/org/apache/toree/kernel/protocol/v5/content/ShutdownRequestSpec.scala
@@ -1,17 +1,18 @@
 

[11/51] [abbrv] incubator-toree git commit: Moved scala files to new locations based on new package

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/68f7ddd6/kernel/src/test/scala/org/apache/toree/kernel/protocol/v5/magic/PostProcessorSpec.scala
--
diff --git 
a/kernel/src/test/scala/org/apache/toree/kernel/protocol/v5/magic/PostProcessorSpec.scala
 
b/kernel/src/test/scala/org/apache/toree/kernel/protocol/v5/magic/PostProcessorSpec.scala
new file mode 100644
index 000..1557460
--- /dev/null
+++ 
b/kernel/src/test/scala/org/apache/toree/kernel/protocol/v5/magic/PostProcessorSpec.scala
@@ -0,0 +1,123 @@
+package com.ibm.spark.kernel.protocol.v5.magic
+
+import com.ibm.spark.interpreter.Interpreter
+import com.ibm.spark.kernel.protocol.v5._
+import com.ibm.spark.magic.{CellMagicOutput, LineMagicOutput}
+import org.mockito.Matchers._
+import org.mockito.Mockito._
+import org.scalatest.mock.MockitoSugar
+import org.scalatest.{FunSpec, Matchers}
+
+class PostProcessorSpec extends FunSpec with Matchers with MockitoSugar{
+  describe("#matchCellMagic") {
+it("should return the cell magic output when the Left contains a " +
+   "CellMagicOutput") {
+  val processor = new PostProcessor(mock[Interpreter])
+  val codeOutput = "some output"
+  val cmo = CellMagicOutput()
+  val left = Left(cmo)
+  processor.matchCellMagic(codeOutput, left) should be(cmo)
+}
+
+it("should package the original code when the Left does not contain a " +
+   "CellMagicOutput") {
+  val processor = new PostProcessor(mock[Interpreter])
+  val codeOutput = "some output"
+  val left = Left("")
+  val data = Data(MIMEType.PlainText -> codeOutput)
+  processor.matchCellMagic(codeOutput, left) should be(data)
+}
+  }
+
+  describe("#matchLineMagic") {
+it("should process the code output when the Right contains a " +
+   "LineMagicOutput") {
+  val processor = spy(new PostProcessor(mock[Interpreter]))
+  val codeOutput = "some output"
+  val lmo = LineMagicOutput
+  val right = Right(lmo)
+  processor.matchLineMagic(codeOutput, right)
+  verify(processor).processLineMagic(codeOutput)
+}
+
+it("should package the original code when the Right does not contain a " +
+   "LineMagicOutput") {
+  val processor = new PostProcessor(mock[Interpreter])
+  val codeOutput = "some output"
+  val right = Right("")
+  val data = Data(MIMEType.PlainText -> codeOutput)
+  processor.matchLineMagic(codeOutput, right) should be(data)
+}
+  }
+
+  describe("#processLineMagic") {
+it("should remove the result of the magic invocation if it is the last " +
+   "line") {
+  val processor = new PostProcessor(mock[Interpreter])
+  val x = "hello world"
+  val codeOutput = s"$x\nsome other output"
+  val data = Data(MIMEType.PlainText -> x)
+  processor.processLineMagic(codeOutput) should be(data)
+}
+  }
+
+  describe("#process") {
+it("should call matchCellMagic when the last variable is a Left") {
+  val intp = mock[Interpreter]
+  val left = Left("")
+  // Need to mock lastExecutionVariableName as it is being chained with
+  // the read method
+  doReturn(Some("")).when(intp).lastExecutionVariableName
+  doReturn(Some(left)).when(intp).read(anyString())
+
+  val processor = spy(new PostProcessor(intp))
+  val codeOutput = "hello"
+  processor.process(codeOutput)
+  verify(processor).matchCellMagic(codeOutput, left)
+}
+
+it("should call matchLineMagic when the last variable is a Right") {
+  val intp = mock[Interpreter]
+  val right = Right("")
+  // Need to mock lastExecutionVariableName as it is being chained with
+  // the read method
+  doReturn(Some("")).when(intp).lastExecutionVariableName
+  doReturn(Some(right)).when(intp).read(anyString())
+
+  val processor = spy(new PostProcessor(intp))
+  val codeOutput = "hello"
+  processor.process(codeOutput)
+  verify(processor).matchLineMagic(codeOutput, right)
+}
+
+it("should package the original code output when the Left is not a " +
+  "Left[CellMagicOutput, Nothing]") {
+  val intp = mock[Interpreter]
+  val left = Left("")
+  // Need to mock lastExecutionVariableName as it is being chained with
+  // the read method
+  doReturn(Some("")).when(intp).lastExecutionVariableName
+  doReturn(Some(left)).when(intp).read(anyString())
+
+  val processor = spy(new PostProcessor(intp))
+  val codeOutput = "hello"
+  val data = Data(MIMEType.PlainText -> codeOutput)
+  processor.process(codeOutput) should be(data)
+}
+
+it("should package the original code output when the Right is not a " +
+   "Right[LineMagicOutput, Nothing]") {
+  val intp = mock[Interpreter]
+  val right = Right("")
+  // Need to mock lastExecutionVariableName as it is being chained with
+  // the read method
+  

[2/2] incubator-toree git commit: Initial prototype of example notebooks.

[lbustelo]

Initial prototype of example notebooks.


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/3b0381ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/3b0381ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/3b0381ab

Branch: refs/heads/master
Commit: 3b0381ab76f92693c3a17519f8d7090fe35c75f8
Parents: 116d8bc
Author: Corey A. Stubbs 
Authored: Fri Feb 12 17:00:46 2016 -0600
Committer: Gino Bustelo 
Committed: Thu Feb 18 13:42:17 2016 -0600

--
 .gitignore  |2 +
 Makefile|3 +-
 .../notebooks/meetup-streaming-toree.ipynb  | 1469 ++
 3 files changed, 1473 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/3b0381ab/.gitignore
--
diff --git a/.gitignore b/.gitignore
index c5eead5..0bd7bb6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,6 +10,8 @@ target/
 .vagrant
 *.iml
 *.ipynb
+.ipynb_checkpoints/
+!etc/**/*.ipynb
 scratch/
 **/*ivy.xml
 test-output/

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/3b0381ab/Makefile
--
diff --git a/Makefile b/Makefile
index 688441e..8904850 100644
--- a/Makefile
+++ b/Makefile
@@ -121,5 +121,6 @@ release: pip-release bin-release
twine upload -r pypi toree-$(VERSION).tar.gz'
 
 jupyter: DOCKER_WORKDIR=/srv/toree/dist
+jupyter: EXTRA_CMD?=pip install jupyter_declarativewidgets==0.4.0; jupyter 
declarativewidgets install --user; jupyter declarativewidgets activate; apt-get 
update; apt-get install --yes curl; curl --silent --location 
https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install --yes 
nodejs; npm install -g bower;
 jupyter: pip-release
-   @$(DOCKER) -p : --user=root  $(IMAGE) bash -c   'pip install 
toree-$(VERSION).tar.gz && jupyter toree install && cd ~ && jupyter notebook 
--ip=* --no-browser'
+   @$(DOCKER) -p : --user=root  $(IMAGE) bash -c   '$(EXTRA_CMD) 
pip install toree-$(VERSION).tar.gz && jupyter toree install && cd 
/srv/toree/etc/examples/notebooks && jupyter notebook --ip=* --no-browser'

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/3b0381ab/etc/examples/notebooks/meetup-streaming-toree.ipynb
--
diff --git a/etc/examples/notebooks/meetup-streaming-toree.ipynb 
b/etc/examples/notebooks/meetup-streaming-toree.ipynb
new file mode 100644
index 000..a2efdec
--- /dev/null
+++ b/etc/examples/notebooks/meetup-streaming-toree.ipynb
@@ -0,0 +1,1469 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+"urth": {
+ "dashboard": {
+  "layout": {
+   "col": 4,
+   "height": 3,
+   "row": 0,
+   "width": 6
+  }
+ }
+}
+   },
+   "source": [
+"# Streaming Meetups Dashboard"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+"urth": {
+ "dashboard": {
+  "layout": {
+   "col": 6,
+   "height": 4,
+   "row": 3,
+   "width": 6
+  }
+ }
+}
+   },
+   "source": [
+"The purpose of this notebook is to give an all-in-one demo of streaming 
data from the [meetup.com RSVP 
API](http://www.meetup.com/meetup_api/docs/stream/2/rsvps/#websockets), through 
a local [Spark Streaming job](http://spark.apache.org/streaming/), and into 
[declarative widgets](https://github.com/jupyter-incubator/declarativewidgets) 
in a dashboard layout."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+"urth": {
+ "dashboard": {
+  "hidden": true
+ }
+}
+   },
+   "source": [
+"On your first visit to this notebook, we recommend that you execute one 
cell at a time as you read along. Later, if you  just want to see the demo, 
select *Cell > Run All* from the menu bar. Once you've run all of the cells, 
select *View > View Dashboard* and then click on the **Stream** toggle to start 
the data stream.\n",
+"\n",
+"**Table of Contents**\n",
+"\n",
+"1. [Create the Frontend Widgets](#Create-the-Widgets-Top) topic histogram, filter entry, user 
card, global heatmap\n",
+"2. [Define the Spark Streaming Job](#Define-the-Spark-Streaming-Job-Top) 
filter by topic, top topics, 
venue metadata\n",
+"3. [Publish Data to Widget 
Channels](#Publish-Data-to-Widget-Channels-Top)\n",
+"4. [Connect to the Data Source](#Connect-to-the-Data-Source-Top) \"custom receiver\", websocket 
connection, stream toggle\n",
+"5. [Arrange the Dashboard Layout](#Arrange-the-Dashboard-Layout-Top)"
+   ]
+  },
+  {
+   

incubator-toree git commit: Modified dev target to use docker and allow JVM debugger

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 69241c93a -> 9be21fa0d


Modified dev target to use docker and allow JVM debugger


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/9be21fa0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/9be21fa0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/9be21fa0

Branch: refs/heads/master
Commit: 9be21fa0d8e22ed1f2dfafb747c1e0221754bab1
Parents: 69241c9
Author: Gino Bustelo 
Authored: Tue Mar 15 17:07:03 2016 -0500
Committer: Gino Bustelo 
Committed: Mon Mar 21 08:49:34 2016 -0500

--
 Makefile| 13 ++---
 etc/kernel.json | 20 
 2 files changed, 30 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9be21fa0/Makefile
--
diff --git a/Makefile b/Makefile
index 3330457..530b41c 100644
--- a/Makefile
+++ b/Makefile
@@ -96,9 +96,16 @@ kernel/target/scala-2.10/$(ASSEMBLY_JAR): 
project/build.properties project/Build
 
 build: kernel/target/scala-2.10/$(ASSEMBLY_JAR)
 
-dev: VM_WORKDIR=~
-dev: dist
-   $(call RUN,ipython notebook --ip=* --no-browser)
+dev: DOCKER_WORKDIR=/srv/toree/etc/examples/notebooks
+dev: SUSPEND=n
+dev: DEBUG_PORT=5005
+dev: .example-image dist
+   @$(DOCKER) \
+   -e 
SPARK_OPTS="--driver-java-options=-agentlib:jdwp=transport=dt_socket,server=y,suspend=$(SUSPEND),address=5005"
 \
+   -v 
`pwd`/etc/kernel.json:/usr/local/share/jupyter/kernels/toree/kernel.json \
+   -p $(DEBUG_PORT):5005 -p : \
+   --user=root  $(EXAMPLE_IMAGE) \
+   bash -c "cp -r /srv/toree/dist/toree/* 
/usr/local/share/jupyter/kernels/toree/. && jupyter notebook --ip=* 
--no-browser"
 
 test: VM_WORKDIR=/src/toree-kernel
 test:

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9be21fa0/etc/kernel.json
--
diff --git a/etc/kernel.json b/etc/kernel.json
new file mode 100644
index 000..52a38a8
--- /dev/null
+++ b/etc/kernel.json
@@ -0,0 +1,20 @@
+{
+  "language_info": {
+"name": "scala"
+  },
+  "display_name": "Toree",
+  "env": {
+"PYTHONPATH": 
"/usr/local/spark/python:/usr/local/spark/python/lib/py4j-0.8.2.1-src.zip",
+"SPARK_HOME": "/usr/local/spark",
+"CAPTURE_STANDARD_ERR": "true",
+"MAX_INTERPRETER_THREADS": "16",
+"CAPTURE_STANDARD_OUT": "true",
+"SEND_EMPTY_OUTPUT": "false"
+  },
+  "argv": [
+"/usr/local/share/jupyter/kernels/toree/bin/run.sh",
+"--profile",
+"{connection_file}"
+  ],
+  "codemirror_mode": "scala"
+}
\ No newline at end of file

incubator-toree git commit: [ci skip] Bump to 0.1.0.dev5

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 61c4b81e6 -> d7ddc08ea


[ci skip] Bump to 0.1.0.dev5


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/d7ddc08e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/d7ddc08e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/d7ddc08e

Branch: refs/heads/master
Commit: d7ddc08ea65cf3964dfc59b443fe623b734167c8
Parents: 61c4b81
Author: Gino Bustelo 
Authored: Wed Mar 2 15:33:08 2016 -0600
Committer: Gino Bustelo 
Committed: Wed Mar 2 15:33:08 2016 -0600

--
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/d7ddc08e/Makefile
--
diff --git a/Makefile b/Makefile
index 6781adf..0a0d210 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
 
 .PHONY: help clean clean-dist build dev test test-travis release pip-release 
bin-release
 
-VERSION?=0.1.0.dev4
+VERSION?=0.1.0.dev5
 COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
 ifeq (, $(findstring dev, $(VERSION)))
 IS_SNAPSHOT?=false

incubator-toree git commit: Fix to sparkmagic example notebook due to def for sqlContext

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master bf6fc5056 -> d35292685


Fix to sparkmagic example notebook due to def for sqlContext


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/d3529268
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/d3529268
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/d3529268

Branch: refs/heads/master
Commit: d35292685bb0e9636176b9ecf353636b0cd2e280
Parents: bf6fc50
Author: Gino Bustelo 
Authored: Fri Apr 1 09:31:52 2016 -0500
Committer: Gino Bustelo 
Committed: Fri Apr 1 09:31:52 2016 -0500

--
 etc/examples/notebooks/magic-tutorial.ipynb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/d3529268/etc/examples/notebooks/magic-tutorial.ipynb
--
diff --git a/etc/examples/notebooks/magic-tutorial.ipynb 
b/etc/examples/notebooks/magic-tutorial.ipynb
index ba33bc3..ad69aa3 100644
--- a/etc/examples/notebooks/magic-tutorial.ipynb
+++ b/etc/examples/notebooks/magic-tutorial.ipynb
@@ -707,7 +707,8 @@
},
"outputs": [],
"source": [
-"import sqlContext.implicits._\n",
+"val sqlc = sqlContext\n",
+"import sqlc.implicits._\n",
 "case class Record(key: String, value: Int)\n",
 "val df = sc.parallelize(1 to 10).map(x => Record(x.toString, 
x)).toDF()\n",
 "df.registerTempTable(\"MYTABLE\")"

[1/2] incubator-toree git commit: Saved with correct kernel name

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master d35292685 -> 781b1e592


Saved with correct kernel name


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/3adc1897
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/3adc1897
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/3adc1897

Branch: refs/heads/master
Commit: 3adc1897eef0421809857766064ec2394706dd5c
Parents: d352926
Author: Gino Bustelo 
Authored: Fri Apr 1 14:40:08 2016 -0500
Committer: Gino Bustelo 
Committed: Fri Apr 1 14:40:08 2016 -0500

--
 .../notebooks/meetup-streaming-toree.ipynb  | 37 +---
 1 file changed, 24 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/3adc1897/etc/examples/notebooks/meetup-streaming-toree.ipynb
--
diff --git a/etc/examples/notebooks/meetup-streaming-toree.ipynb 
b/etc/examples/notebooks/meetup-streaming-toree.ipynb
index 0537f4a..eec9072 100644
--- a/etc/examples/notebooks/meetup-streaming-toree.ipynb
+++ b/etc/examples/notebooks/meetup-streaming-toree.ipynb
@@ -121,7 +121,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -194,7 +194,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -304,7 +304,9 @@
 "val fixed_event = default_event ++ 
(event).as[play.api.libs.json.JsObject] \n",
 "\n",
 "channel(\"meetups\").set(\"meetup\", fixed_event.value)\n",
-"} \n",
+"} catch {\n",
+"  case _ => print(\"No data to sample\")\n",
+"}\n",
 "}\n",
 "\n",
 "/**\n",
@@ -453,7 +455,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -470,7 +472,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -504,7 +506,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -531,7 +533,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -544,7 +546,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -576,7 +578,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -667,7 +669,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
-"collapsed": true
+"collapsed": false
},
"outputs": [],
"source": [
@@ -710,13 +712,22 @@
 "in a fresh notebook, the dashboard will only show cells with non-empty 
output. All other cells can be found in the *Hidden* section at the bottom of 
the dashboard layout page. You can quickly add all cell outputs or remove all 
cell outputs from the dashboard using the show / hide icons that appear in the 
notebook toolbar when you are in layout mode.\n",
 ""
]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+"collapsed": true
+   },
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Toree",
-   "language": "",
-   "name": "toree"
+   "display_name": "Toree - Scala",
+   "language": "scala",
+   "name": "toree_scala"
   },
   "language_info": {
"name": "scala"

[2/2] incubator-toree git commit: Converted magics to plugins

[lbustelo]

Converted magics to plugins

Fixed plugin test classpath to run script

Updated internal plugin loading to use only Toree-related classes

Updated compilation to use Java 1.6 to avoid issues on JDK 8

Fixed kernel hanging on shutdown

Added JVM forking options for sbt


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/9c8824ff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/9c8824ff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/9c8824ff

Branch: refs/heads/master
Commit: 9c8824fff063ff6e84b2e4298088954ae610612b
Parents: 429c74c
Author: Chip Senkbeil and Corey Stubbs 
Authored: Fri Mar 25 15:03:22 2016 -0500
Committer: Chip Senkbeil and Corey Stubbs 
Committed: Tue Mar 29 12:20:04 2016 -0500

--
 .jvmopts|  11 +
 Makefile|   1 +
 etc/bin/run.sh  |   5 +-
 .../toree/interpreter/broker/BrokerState.scala  |   4 +-
 .../scala/org/apache/toree/magic/Magic.scala|  24 +-
 .../org/apache/toree/magic/MagicExecutor.scala  |  51 ---
 .../org/apache/toree/magic/MagicLoader.scala| 138 
 .../org/apache/toree/magic/MagicManager.scala   | 111 +++
 .../toree/magic/MagicNotFoundException.scala|  22 ++
 .../magic/dependencies/DependencyMap.scala  | 171 --
 .../magic/dependencies/IncludeConfig.scala  |   8 +-
 .../IncludeDependencyDownloader.scala   |   9 +-
 .../magic/dependencies/IncludeInterpreter.scala |  10 +-
 .../magic/dependencies/IncludeKernel.scala  |   8 +-
 .../dependencies/IncludeKernelInterpreter.scala |  11 +-
 .../magic/dependencies/IncludeMagicLoader.scala |  31 --
 .../dependencies/IncludeOutputStream.scala  |  12 +-
 .../dependencies/IncludePluginManager.scala |  30 ++
 .../magic/dependencies/IncludeSQLContext.scala  |  12 +-
 .../dependencies/IncludeSparkContext.scala  |  13 +-
 .../apache/toree/magic/MagicLoaderSpec.scala| 184 ---
 .../apache/toree/magic/MagicManagerSpec.scala   | 324 +++
 .../src/test/scala/test/utils/SomeMagic.scala   |  25 ++
 .../org/apache/toree/boot/KernelBootstrap.scala |   5 +-
 .../boot/layer/ComponentInitialization.scala|  89 ++---
 .../boot/layer/HandlerInitialization.scala  |  28 +-
 .../toree/boot/layer/HookInitialization.scala   |   4 -
 .../org/apache/toree/kernel/api/Kernel.scala|  44 +--
 .../kernel/protocol/v5/magic/MagicParser.scala  |  23 +-
 .../protocol/v5/relay/ExecuteRequestRelay.scala |  12 +-
 .../apache/toree/magic/builtin/AddDeps.scala|   5 +-
 .../org/apache/toree/magic/builtin/AddJar.scala |  26 +-
 .../org/apache/toree/magic/builtin/Html.scala   |   6 +-
 .../apache/toree/magic/builtin/JavaScript.scala |   6 +-
 .../apache/toree/magic/builtin/LSMagic.scala|   4 +-
 .../org/apache/toree/magic/builtin/RDD.scala|   2 +
 .../apache/toree/magic/builtin/ShowTypes.scala  |   4 +-
 .../apache/toree/magic/builtin/Truncation.scala |   4 +-
 .../apache/toree/kernel/api/KernelSpec.scala|  15 +-
 .../protocol/v5/magic/MagicParserSpec.scala |  86 +++--
 .../v5/relay/ExecuteRequestRelaySpec.scala  |  29 +-
 .../apache/toree/magic/builtin/AddJarSpec.scala |  15 +-
 .../scala/system/KernelCommSpecForSystem.scala  |  13 +-
 .../src/test/scala/system/SuiteForSystem.scala  |   8 +-
 .../src/test/scala/system/TruncationTests.scala |  19 +-
 .../test/utils/NoArgSparkKernelTestKit.scala|   1 -
 .../scala/test/utils/SparkKernelDeployer.scala  |  19 +-
 .../scala/org/apache/toree/plugins/Plugin.scala |  28 +-
 .../apache/toree/plugins/PluginManager.scala|   4 +-
 .../org/apache/toree/plugins/PluginMethod.scala |   3 -
 .../apache/toree/plugins/PluginSearcher.scala   |  26 +-
 .../PluginManagerSpecForIntegration.scala   |   9 +
 .../toree/plugins/PluginManagerSpec.scala   |   8 +-
 .../org/apache/toree/plugins/PluginSpec.scala   |  28 +-
 project/Build.scala |  14 +-
 project/Common.scala|  29 +-
 project/plugins.sbt |   2 +-
 .../apache/toree/magic/builtin/PySpark.scala|   2 +
 resources/compile/log4j.properties  |   1 +
 .../org/apache/toree/magic/builtin/Scala.scala  |   4 +-
 .../org/apache/toree/magic/builtin/SparkR.scala |   3 +
 .../org/apache/toree/magic/builtin/Sql.scala|   3 +
 62 files changed, 964 insertions(+), 882 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9c8824ff/.jvmopts
--
diff --git a/.jvmopts b/.jvmopts
new file mode 100644
index 000..f22a607
--- /dev/null

[1/2] incubator-toree git commit: Converted magics to plugins

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 429c74c67 -> 9c8824fff


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9c8824ff/kernel/src/main/scala/org/apache/toree/magic/builtin/AddDeps.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/magic/builtin/AddDeps.scala 
b/kernel/src/main/scala/org/apache/toree/magic/builtin/AddDeps.scala
index c6771be..be36c8f 100644
--- a/kernel/src/main/scala/org/apache/toree/magic/builtin/AddDeps.scala
+++ b/kernel/src/main/scala/org/apache/toree/magic/builtin/AddDeps.scala
@@ -23,15 +23,15 @@ import java.net.URL
 import org.apache.toree.magic._
 import org.apache.toree.magic.dependencies._
 import org.apache.toree.utils.ArgumentParsingSupport
-
 import scala.util.Try
+import org.apache.toree.plugins.annotations.Event
 
 class AddDeps extends LineMagic with IncludeInterpreter
   with IncludeOutputStream with IncludeSparkContext with ArgumentParsingSupport
   with IncludeDependencyDownloader with IncludeKernel
 {
 
-  private lazy val printStream = new PrintStream(outputStream)
+  private def printStream = new PrintStream(outputStream)
 
   private val _transitive = parser.accepts(
 "transitive", "Retrieve dependencies recursively"
@@ -59,6 +59,7 @@ class AddDeps extends LineMagic with IncludeInterpreter
* @param code The single line of code
* @return The output of the magic
*/
+  @Event(name = "adddeps")
   override def execute(code: String): Unit = {
 val nonOptionArgs = parseArgs(code)
 dependencyDownloader.setPrintStream(printStream)

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9c8824ff/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala 
b/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
index 65abccc..125ee0a 100644
--- a/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
+++ b/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
@@ -26,6 +26,7 @@ import org.apache.toree.magic.builtin.AddJar._
 import org.apache.toree.magic.dependencies._
 import org.apache.toree.utils.{ArgumentParsingSupport, DownloadSupport, 
LogLike}
 import com.typesafe.config.Config
+import org.apache.toree.plugins.annotations.Event
 
 object AddJar {
 
@@ -47,7 +48,7 @@ object AddJar {
 class AddJar
   extends LineMagic with IncludeInterpreter with IncludeSparkContext
   with IncludeOutputStream with DownloadSupport with ArgumentParsingSupport
-  with IncludeKernel with IncludeMagicLoader with IncludeConfig with LogLike
+  with IncludeKernel with IncludePluginManager with IncludeConfig with LogLike
 {
   // Option to mark re-downloading of jars
   private val _force =
@@ -58,7 +59,7 @@ class AddJar
 parser.accepts("magic", "loads jar as a magic extension")
 
   // Lazy because the outputStream is not provided at construction
-  private lazy val printStream = new PrintStream(outputStream)
+  private def printStream = new PrintStream(outputStream)
 
   /**
* Retrieves file name from URL.
@@ -82,6 +83,7 @@ class AddJar
*
* @param code The line containing the location of the jar
*/
+  @Event(name = "addjar")
   override def execute(code: String): Unit = {
 val nonOptionArgs = parseArgs(code.trim)
 
@@ -103,12 +105,14 @@ class AddJar
 
 // Ensure the URL actually contains a jar or zip file
 if (!jarName.endsWith(".jar") && !jarName.endsWith(".zip")) {
-throw new IllegalArgumentException(s"The jar file $jarName must end in 
.jar or .zip.")
+throw new IllegalArgumentException(
+  s"The jar file $jarName must end in .jar or .zip."
+)
 }
 
 val downloadLocation = getJarDir(config) + "/" + jarName
 
-logger.debug( "Downloading jar to %s".format(downloadLocation) )
+logger.debug("Downloading jar to %s".format(downloadLocation))
 
 val fileDownloadLocation = new File(downloadLocation)
 
@@ -128,18 +132,12 @@ class AddJar
   printStream.println(s"Using cached version of $jarName")
 }
 
-
-if (_magic)
-{
-
-  magicLoader.addJar(fileDownloadLocation.toURI.toURL)
-
-}
-else
-{
+if (_magic) {
+  val plugins = pluginManager.loadPlugins(fileDownloadLocation)
+  pluginManager.initializePlugins(plugins)
+} else {
   interpreter.addJars(fileDownloadLocation.toURI.toURL)
   sparkContext.addJar(fileDownloadLocation.getCanonicalPath)
-
 }
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/9c8824ff/kernel/src/main/scala/org/apache/toree/magic/builtin/Html.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/magic/builtin/Html.scala 
b/kernel/src/main/scala/org/apache/toree/magic/builtin/Html.scala
index 8060a46..7975ce7 100644
--- 

incubator-toree git commit: [TOREE 256]: Add installation of all interpreters via pip

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 9c8824fff -> 2aa8ce12b


[TOREE 256]: Add installation of all interpreters via pip


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/2aa8ce12
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/2aa8ce12
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/2aa8ce12

Branch: refs/heads/master
Commit: 2aa8ce12bbf0aa08dfe4e52731aa5fb1a9d520eb
Parents: 9c8824f
Author: Corey A. Stubbs 
Authored: Fri Mar 18 13:41:57 2016 -0500
Committer: Corey A. Stubbs 
Committed: Tue Mar 29 15:02:46 2016 -0500

--
 Makefile  | 12 -
 etc/bin/run.sh| 14 +++--
 etc/pip_install/toree/toreeapp.py | 99 +-
 3 files changed, 70 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/2aa8ce12/Makefile
--
diff --git a/Makefile b/Makefile
index cdf525a..335de41 100644
--- a/Makefile
+++ b/Makefile
@@ -74,7 +74,7 @@ clean: clean-dist
$(call RUN,$(ENV_OPTS) sbt clean)
rm -r `find . -name target -type d`
 
-.example-image: EXTRA_CMD?=pip install jupyter_declarativewidgets==0.4.0; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs; npm install -g bower;
+.example-image: EXTRA_CMD?=printf "deb 
http://cran.rstudio.com/bin/linux/debian jessie-cran3/" >> 
/etc/apt/sources.list; apt-key adv --keyserver keys.gnupg.net --recv-key 
381BA480; apt-get update; pip install jupyter_declarativewidgets==0.4.0; 
jupyter declarativewidgets install --user; jupyter declarativewidgets activate; 
pip install jupyter_dashboards; jupyter dashboards install --user; jupyter 
dashboards activate; apt-get update; apt-get install --yes curl; curl --silent 
--location https://deb.nodesource.com/setup_0.12 | sudo bash -; apt-get install 
--yes nodejs r-base r-base-dev; npm install -g bower;
 .example-image:
@-docker rm -f examples_image
@docker run -it --user root --name examples_image \
@@ -145,6 +145,14 @@ release: pip-release bin-release
python setup.py register -r $(PYPI_REPO) && \
twine upload -r pypi toree-$(VERSION).tar.gz'
 
+define JUPYTER_COMMAND
+pip install toree-$(VERSION).tar.gz
+jupyter toree install --interpreters=PySpark,SQL,Scala,SparkR
+cd /srv/toree/etc/examples/notebooks
+jupyter notebook --ip=* --no-browser
+endef
+
+export JUPYTER_COMMAND
 jupyter: DOCKER_WORKDIR=/srv/toree/dist
 jupyter: .example-image pip-release
-   @$(DOCKER) -p : --user=root  $(EXAMPLE_IMAGE) bash -c   'pip 
install toree-$(VERSION).tar.gz && jupyter toree install && cd 
/srv/toree/etc/examples/notebooks && jupyter notebook --ip=* --no-browser'
+   @$(DOCKER) -p :  --user=root  $(EXAMPLE_IMAGE) bash -c 
"$$JUPYTER_COMMAND"

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/2aa8ce12/etc/bin/run.sh
--
diff --git a/etc/bin/run.sh b/etc/bin/run.sh
index 67b5a29..84ed43a 100755
--- a/etc/bin/run.sh
+++ b/etc/bin/run.sh
@@ -16,7 +16,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License
 #
-   ``
 PROG_HOME="$(cd "`dirname "$0"`"/..; pwd)"
 
 if [ -z "$SPARK_HOME" ]; then
@@ -31,7 +30,12 @@ KERNEL_ASSEMBLY=`(cd ${PROG_HOME}/lib; ls -1 
toree-kernel-assembly-*.jar;)`
 # disable randomized hash for string in Python 3.3+
 export PYTHONHASHSEED=0
 TOREE_ASSEMBLY=${PROG_HOME}/lib/${KERNEL_ASSEMBLY}
-exec "$SPARK_HOME"/bin/spark-submit \
-  ${SPARK_OPTS} \
-  --driver-class-path ${TOREE_ASSEMBLY} \
-  --class org.apache.toree.Main ${TOREE_ASSEMBLY} "$@"
+# The SPARK_OPTS values during installation are stored in 
__TOREE_SPARK_OPTS__. This allows values to be specified during
+# install, but also during runtime. The runtime options take precedence over 
the install options.
+if [ "${SPARK_OPTS}" = "" ]
+then
+   SPARK_OPTS=${__TOREE_SPARK_OPTS__}
+fi
+
+SPARK_OPTS="--driver-class-path=\"${TOREE_ASSEMBLY}\" ${SPARK_OPTS}"
+eval exec "${SPARK_HOME}/bin/spark-submit" "${SPARK_OPTS}" --class 
org.apache.toree.Main "${TOREE_ASSEMBLY}" "$@"

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/2aa8ce12/etc/pip_install/toree/toreeapp.py
--
diff --git 

[1/2] incubator-toree git commit: [ci skip] Bump to 0.1.0.dev6

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 855fb4100 -> 68247631b


[ci skip] Bump to 0.1.0.dev6


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/45e07093
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/45e07093
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/45e07093

Branch: refs/heads/master
Commit: 45e070936577f0d659e4cc7df3e424e7a24d3708
Parents: 855fb41
Author: Gino Bustelo 
Authored: Wed Apr 20 09:53:51 2016 -0500
Committer: Gino Bustelo 
Committed: Wed Apr 20 09:53:51 2016 -0500

--
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/45e07093/Makefile
--
diff --git a/Makefile b/Makefile
index a299e80..2d19877 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
 
 .PHONY: help clean clean-dist build dev test test-travis release pip-release 
bin-release dev-binder .binder-image
 
-VERSION?=0.1.0.dev5
+VERSION?=0.1.0.dev6
 COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
 ifeq (, $(findstring dev, $(VERSION)))
 IS_SNAPSHOT?=false

incubator-toree git commit: [skip ci] Bump to 0.1.0.dev7

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 392aaef81 -> 8c12e4d7b


[skip ci] Bump to 0.1.0.dev7


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/8c12e4d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/8c12e4d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/8c12e4d7

Branch: refs/heads/master
Commit: 8c12e4d7b2f1ab06a583c45184d39f9e38564a82
Parents: 392aaef
Author: Gino Bustelo 
Authored: Fri Apr 29 15:12:19 2016 -0500
Committer: Gino Bustelo 
Committed: Fri Apr 29 15:12:19 2016 -0500

--
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/8c12e4d7/Makefile
--
diff --git a/Makefile b/Makefile
index feaf7d7..a95b44b 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
 
 .PHONY: help clean clean-dist build dev test test-travis release pip-release 
bin-release dev-binder .binder-image audit audit-licenses
 
-BASE_VERSION=0.1.0.dev6
+BASE_VERSION=0.1.0.dev7
 VERSION=$(BASE_VERSION)-incubating
 COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
 ifeq (, $(findstring dev, $(VERSION)))

[2/2] incubator-toree git commit: [skip ci] Added missing license header

[lbustelo]

[skip ci] Added missing license header


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/5edd6d7d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/5edd6d7d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/5edd6d7d

Branch: refs/heads/master
Commit: 5edd6d7d1286576b5ac765d8810c70d6b0f1dbe2
Parents: 553ac74
Author: Gino Bustelo 
Authored: Tue Apr 26 13:26:28 2016 -0500
Committer: Gino Bustelo 
Committed: Tue Apr 26 13:26:28 2016 -0500

--
 kernel/src/test/scala/test/utils/package.scala | 16 
 1 file changed, 16 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/5edd6d7d/kernel/src/test/scala/test/utils/package.scala
--
diff --git a/kernel/src/test/scala/test/utils/package.scala 
b/kernel/src/test/scala/test/utils/package.scala
index b606d31..06468d8 100644
--- a/kernel/src/test/scala/test/utils/package.scala
+++ b/kernel/src/test/scala/test/utils/package.scala
@@ -1,3 +1,19 @@
+/*
+ *  Licensed to the Apache Software Foundation (ASF) under one or more
+ *  contributor license agreements.  See the NOTICE file distributed with
+ *  this work for additional information regarding copyright ownership.
+ *  The ASF licenses this file to You under the Apache License, Version 2.0
+ *  (the "License"); you may not use this file except in compliance with
+ *  the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License
+ */
 package test
 
 import scala.concurrent.duration._

[2/2] incubator-toree git commit: [skip ci] Added a disclaimer regarding LGPL dependency to README

[lbustelo]

[skip ci] Added a disclaimer regarding LGPL dependency to README


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/b59a600e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/b59a600e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/b59a600e

Branch: refs/heads/master
Commit: b59a600e6a0e7bf7698a1463954fcdb6e2426845
Parents: 7540760
Author: Gino Bustelo 
Authored: Fri May 20 15:02:21 2016 -0500
Committer: Gino Bustelo 
Committed: Fri May 20 15:02:21 2016 -0500

--
 README.md | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/b59a600e/README.md
--
diff --git a/README.md b/README.md
index 8d5ed66..b459dbf 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,12 @@ Toree provides an interface that allows clients to interact 
with a Spark Cluster
 
 The main supported language is `Scala`, but it is also capable of processing 
both `Python` and `R`. It implements the latest Jupyter message protocol (5.0), 
so it can easily plug into the latest releases of Jupyter/IPython (3.2.x+ and 
4.x+) for quick, interactive data exploration.
 
+ Disclaimer
+This project is currently not fully compliant with Apache release policy
+as it includes a runtime dependency that is licensed as LGPL v3 (plus a static
+linking exception). This package is currently under an effort to re-license
+(https://github.com/zeromq/jeromq/issues/327).
+
 Try It
 ==
 A version of Toree is deployed as part of the [Try Jupyter!][try-jupyter] 
site. Select `Scala 2.10.4 (Spark 1.4.1)` under the `New` dropdown. Note that 
this version only supports `Scala`.

[4/4] incubator-toree git commit: [skip ci] Changing the name of the kernel used sample notebook

[lbustelo]

[skip ci] Changing the name of the kernel used sample notebook


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/31b2039e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/31b2039e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/31b2039e

Branch: refs/heads/master
Commit: 31b2039ee762f8dd6dc1c9d5df25ffcc43d68e2f
Parents: f17b4ee
Author: Gino Bustelo 
Authored: Fri May 6 11:42:28 2016 -0500
Committer: Gino Bustelo 
Committed: Fri May 6 11:42:28 2016 -0500

--
 etc/examples/notebooks/magic-tutorial.ipynb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/31b2039e/etc/examples/notebooks/magic-tutorial.ipynb
--
diff --git a/etc/examples/notebooks/magic-tutorial.ipynb 
b/etc/examples/notebooks/magic-tutorial.ipynb
index c21e71e..8725868 100644
--- a/etc/examples/notebooks/magic-tutorial.ipynb
+++ b/etc/examples/notebooks/magic-tutorial.ipynb
@@ -783,9 +783,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Toree",
-   "language": "",
-   "name": "toree"
+   "display_name": "Apache Toree - Scala",
+   "language": "scala",
+   "name": "apache_toree_scala"
   },
   "language_info": {
"name": "scala"

[2/4] incubator-toree git commit: Update the magic-tutorial notebook to reflect the changes to make add deps dir unique

[lbustelo]

Update the magic-tutorial notebook to reflect the changes to make add deps dir 
unique


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/ddfa9028
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/ddfa9028
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/ddfa9028

Branch: refs/heads/master
Commit: ddfa90287614f54f254339c9f05b197ea07fd78d
Parents: 58ab0d8
Author: Michael Poplavski 
Authored: Tue May 3 16:25:29 2016 -0500
Committer: Michael Poplavski 
Committed: Wed May 4 14:56:11 2016 -0500

--
 etc/examples/notebooks/magic-tutorial.ipynb | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/ddfa9028/etc/examples/notebooks/magic-tutorial.ipynb
--
diff --git a/etc/examples/notebooks/magic-tutorial.ipynb 
b/etc/examples/notebooks/magic-tutorial.ipynb
index ad69aa3..c21e71e 100644
--- a/etc/examples/notebooks/magic-tutorial.ipynb
+++ b/etc/examples/notebooks/magic-tutorial.ipynb
@@ -443,7 +443,7 @@
  "text": [
   "Marking org.joda:joda-money:0.11 for download\n",
   "Preparing to fetch from:\n",
-  "-> file:/tmp/.ivy2/\n",
+  "-> file:/tmp/toree_add_deps5662724810625125387/\n",
   "-> https://repo1.maven.org/maven2\n;,
   "=> 1 (): Downloading 
https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.pom.sha1\n;,
   "=> 1 (): Downloading 
https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.pom\n;,
@@ -465,7 +465,7 @@
   "===> 2 (joda-money-0.11.jar): Downloaded 48481 bytes (76.08%)\n",
   "===> 2 (joda-money-0.11.jar): Downloaded 63725 bytes (100.00%)\n",
   "=> 2 (joda-money-0.11.jar): Finished downloading\n",
-  "-> New file at 
/tmp/.ivy2/https/repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar\n"
+  "-> New file at 
/tmp/toree_add_deps5662724810625125387/https/repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar\n"
  ]
 }
],
@@ -783,9 +783,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Toree - Scala",
-   "language": "scala",
-   "name": "toree_scala"
+   "display_name": "Toree",
+   "language": "",
+   "name": "toree"
   },
   "language_info": {
"name": "scala"

[3/4] incubator-toree git commit: Update jar dir location to toree from spark_kernel

[lbustelo]

Update jar dir location to toree from spark_kernel


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/f17b4ee7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/f17b4ee7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/f17b4ee7

Branch: refs/heads/master
Commit: f17b4ee77e0de5b02d6b85dc1ee9cc8d47298313
Parents: ddfa902
Author: Michael Poplavski 
Authored: Wed May 4 14:59:52 2016 -0500
Committer: Michael Poplavski 
Committed: Wed May 4 15:02:13 2016 -0500

--
 kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/f17b4ee7/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
--
diff --git a/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala 
b/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
index 125ee0a..a8bf14b 100644
--- a/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
+++ b/kernel/src/main/scala/org/apache/toree/magic/builtin/AddJar.scala
@@ -37,7 +37,7 @@ object AddJar {
 if(config.hasPath("jar_dir") && 
Files.exists(Paths.get(config.getString("jar_dir" {
   config.getString("jar_dir")
 } else {
-  
Files.createTempDirectory("spark_kernel_add_jars").toFile.getAbsolutePath
+  Files.createTempDirectory("toree_add_jars").toFile.getAbsolutePath
 }
   )
   jarDir.get

[1/4] incubator-toree git commit: Makes add deps installation directory either a specified path or a unique temp path per toree process.

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 0a9068a2f -> 31b2039ee


Makes add deps installation directory either a specified
path or a unique temp path per toree process.


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/58ab0d80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/58ab0d80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/58ab0d80

Branch: refs/heads/master
Commit: 58ab0d8037bbd2dd6b33bb565ea2dbec59cc5b7a
Parents: 0a9068a
Author: Michael Poplavski 
Authored: Fri Apr 22 12:23:32 2016 -0500
Committer: Michael Poplavski 
Committed: Wed May 4 14:52:45 2016 -0500

--
 .../toree/boot/layer/ComponentInitialization.scala| 14 ++
 resources/compile/reference.conf  |  5 ++---
 resources/test/reference.conf |  3 +--
 3 files changed, 13 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/58ab0d80/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
--
diff --git 
a/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
 
b/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
index a7194eb..fb7e76d 100644
--- 
a/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
+++ 
b/kernel/src/main/scala/org/apache/toree/boot/layer/ComponentInitialization.scala
@@ -18,6 +18,7 @@
 package org.apache.toree.boot.layer
 
 import java.io.File
+import java.nio.file.{Paths, Files}
 import java.util.concurrent.ConcurrentHashMap
 
 import akka.actor.ActorRef
@@ -116,12 +117,17 @@ trait StandardComponentInitialization extends 
ComponentInitialization {
   }
 
   private def initializeDependencyDownloader(config: Config) = {
-/*val dependencyDownloader = new IvyDependencyDownloader(
-  "http://repo1.maven.org/maven2/;, config.getString("ivy_local")
-)*/
+val depsDir = {
+  if(config.hasPath("deps_dir") && 
Files.exists(Paths.get(config.getString("deps_dir" {
+config.getString("deps_dir")
+  } else {
+Files.createTempDirectory("toree_add_deps").toFile.getAbsolutePath
+  }
+}
+
 val dependencyDownloader = new CoursierDependencyDownloader
 dependencyDownloader.setDownloadDirectory(
-  new File(config.getString("ivy_local"))
+  new File(depsDir)
 )
 
 dependencyDownloader

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/58ab0d80/resources/compile/reference.conf
--
diff --git a/resources/compile/reference.conf b/resources/compile/reference.conf
index 18efe52..1c23174 100644
--- a/resources/compile/reference.conf
+++ b/resources/compile/reference.conf
@@ -37,9 +37,6 @@ transport = "tcp"
 signature_scheme = "hmac-sha256"
 key = ""
 
-ivy_local = "/tmp/.ivy2"
-ivy_local = ${?IVY_LOCAL}
-
 interpreter_args = []
 
 magic_urls = []
@@ -53,6 +50,8 @@ send_empty_output = ${?SEND_EMPTY_OUTPUT}
 
 jar_dir = ${?JAR_DIR}
 
+deps_dir = ${?DEPS_DIR}
+
 default_interpreter = "Scala"
 default_interpreter = ${?DEFAULT_INTERPRETER}
 

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/58ab0d80/resources/test/reference.conf
--
diff --git a/resources/test/reference.conf b/resources/test/reference.conf
index 461d4e8..85e614c 100644
--- a/resources/test/reference.conf
+++ b/resources/test/reference.conf
@@ -38,8 +38,7 @@ signature_scheme = "hmac-sha256"
 key = ""
 spark.master = "local[*]"
 
-ivy_local = "/tmp/.ivy2"
-ivy_local = ${?IVY_LOCAL}
+deps_dir = ${?DEPS_DIR}
 
 interpreter_args = []
 

[1/2] incubator-toree git commit: [TOREE 278]: Replaced RDD Magic With DataFrame Added DataFrame magic to replace RDD magic Added support to output DataFrames in different types

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 31b2039ee -> 4463bf460


[TOREE 278]: Replaced RDD Magic With DataFrame
Added DataFrame magic to replace RDD magic
Added support to output DataFrames in different types


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/5a2b79e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/5a2b79e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/5a2b79e2

Branch: refs/heads/master
Commit: 5a2b79e25e75f24c64780361147aeae40bc13f47
Parents: 31b2039
Author: Corey A. Stubbs 
Authored: Wed May 4 15:06:00 2016 -0500
Committer: Corey A. Stubbs 
Committed: Fri May 6 12:58:36 2016 -0500

--
 etc/examples/notebooks/magic-tutorial.ipynb | 156 +++
 .../apache/toree/magic/builtin/Dataframe.scala  | 149 ++
 .../org/apache/toree/magic/builtin/RDD.scala|  64 --
 .../apache/toree/utils/DataFrameConverter.scala |  76 +++
 .../org/apache/toree/utils/json/RddToJson.scala |  42 
 .../toree/magic/builtin/DataFrameSpec.scala | 196 +++
 .../apache/toree/magic/builtin/RDDSpec.scala| 118 ---
 .../toree/utils/DataFrameConverterSpec.scala|  79 
 .../apache/toree/utils/json/RddToJsonSpec.scala |  56 --
 9 files changed, 656 insertions(+), 280 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/5a2b79e2/etc/examples/notebooks/magic-tutorial.ipynb
--
diff --git a/etc/examples/notebooks/magic-tutorial.ipynb 
b/etc/examples/notebooks/magic-tutorial.ipynb
index 8725868..3d3ca28 100644
--- a/etc/examples/notebooks/magic-tutorial.ipynb
+++ b/etc/examples/notebooks/magic-tutorial.ipynb
@@ -514,6 +514,162 @@
   },
   {
"cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"### %%DataFrame\n",
+"The `%%DataFrame` magic is used to convert a Spark SQL DataFrame into 
various formats. Currently, `json`, `html`, and `csv` are supported. The magic 
takes an expression, which evauluates to a dataframe, to perform the 
conversion. So, we first need to create a DataFrame object for reference."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+"collapsed": true
+   },
+   "outputs": [],
+   "source": [
+"case class DFRecord(key: String, value: Int)\n",
+"val sqlc = sqlContext\n",
+"import sqlc.implicits._\n",
+"val df = sc.parallelize(1 to 10).map(x => DFRecord(x.toString, x)).toDF()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"The default output is `html`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "data": {
+  "text/plain": [
+   "%%dataframe [arguments]\n",
+   "DATAFRAME_CODE\n",
+   "\n",
+   "DATAGRAME_CODE can be any numbered lines of code, as long as the\n",
+   "last line is a reference to a variable which is a DataFrame.\n",
+   "OptionDescription   \n",
+   "-----   \n",
+   "--helpDisplays the help and usage text for  \n",
+   "this magic. \n",
+   "--limit   The type of the output: html  \n",
+   "(default), csv, json (default: 10)  \n",
+   "--output  The type of the output: html  \n",
+   "(default), csv, json (default: html)\n"
+  ]
+ },
+ "execution_count": 1,
+ "metadata": {},
+ "output_type": "execute_result"
+}
+   ],
+   "source": [
+"%%dataframe"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "data": {
+  "text/html": [
+   
"keyvalue1122334455667788991010"
+  ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+}
+   ],
+   "source": [
+"%%dataframe\n",
+"df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"You can specify the `--output` argument to change the output type."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "data": {
+  "text/plain": [
+   "key,value\n",
+   "1,1\n",
+   "2,2\n",
+   "3,3\n",
+   "4,4\n",
+   "5,5\n",
+   "6,6\n",
+   "7,7\n",
+   "8,8\n",
+   "9,9\n",
+   "10,10"
+  ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"

[2/2] incubator-toree git commit: [skip ci] Added dataframe to the TOC of the sample notebook

[lbustelo]

[skip ci] Added dataframe to the TOC of the sample notebook


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/4463bf46
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/4463bf46
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/4463bf46

Branch: refs/heads/master
Commit: 4463bf460fd5bb04e2fe818e9127b4ec79678564
Parents: 5a2b79e
Author: Gino Bustelo 
Authored: Fri May 6 13:42:48 2016 -0500
Committer: Gino Bustelo 
Committed: Fri May 6 13:42:48 2016 -0500

--
 etc/examples/notebooks/magic-tutorial.ipynb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/4463bf46/etc/examples/notebooks/magic-tutorial.ipynb
--
diff --git a/etc/examples/notebooks/magic-tutorial.ipynb 
b/etc/examples/notebooks/magic-tutorial.ipynb
index 3d3ca28..e6d6a62 100644
--- a/etc/examples/notebooks/magic-tutorial.ipynb
+++ b/etc/examples/notebooks/magic-tutorial.ipynb
@@ -16,6 +16,7 @@
 "1. [AddJar](#addjar)\n",
 "1. [AddDeps](#adddeps)\n",
 "1. [Cell Magics](#cell-magics)\n",
+"1. [DataFrame](#dataframe)\n",
 "1. [Html](#html)\n",
 "1. [JavaScript](#javascript)\n",
 "1. [PySpark](#pyspark)\n",
@@ -516,7 +517,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
-"### %%DataFrame\n",
+"### %%DataFrameTop\n",
 "The `%%DataFrame` magic is used to convert a Spark SQL DataFrame into 
various formats. Currently, `json`, `html`, and `csv` are supported. The magic 
takes an expression, which evauluates to a dataframe, to perform the 
conversion. So, we first need to create a DataFrame object for reference."
]
   },

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x [created] 29d10b3e7

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/issue-123-make-imports-configurable [deleted] 36c1ace0e

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/ChangesToTravis [deleted] cea63655a

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/AddMacrosForDependencies [deleted] c3de594c7

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/ExtraStuff [deleted] db9429993

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/branch-0.1.4 [deleted] c4e459f63

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/AddZeppelinSupport [deleted] 96b4c0c32

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/branch-0.1.3 [deleted] 9bd066292

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/AddJRubySupport [deleted] 225eb62f9

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/AddNightlyBuildSupport [deleted] d68f07acd

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/MonitorMemory [deleted] f72176ae7

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/sqlInjection [deleted] ca1feaae8

incubator-toree git commit: Updated branch information in readme

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/master 01936c1da -> 87a9eb8ad


Updated branch information in readme


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/87a9eb8a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/87a9eb8a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/87a9eb8a

Branch: refs/heads/master
Commit: 87a9eb8ad08406ce0747e92f7714d4eb54153293
Parents: 01936c1
Author: Gino Bustelo 
Authored: Thu Sep 8 09:28:30 2016 -0500
Committer: Gino Bustelo 
Committed: Thu Sep 8 09:28:30 2016 -0500

--
 README.md | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/87a9eb8a/README.md
--
diff --git a/README.md b/README.md
index 8024ff1..148f2ed 100644
--- a/README.md
+++ b/README.md
@@ -94,9 +94,8 @@ As it stands, we maintain several branches for legacy 
versions of Spark. The tab
 
 Branch   | Apache Spark Version
  | 
-[master][master] | 1.5.1+
-[branch-0.1.4][branch-0.1.4] | 1.4.1
-[branch-0.1.3][branch-0.1.3] | 1.3.1
+[master][master] | 2.0
+[0.1.x][0.1.x]   | 1.6+
 
 Please note that for the most part, new features will mainly be added to the 
`master` branch.
 
@@ -124,5 +123,4 @@ We are working on porting our documentation into Apache. 
For the time being, you
 [mail-list]: mailto:d...@toree.incubator.apache.org
 
 [master]: https://github.com/apache/incubator-toree
-[branch-0.1.4]: https://github.com/apache/incubator-toree/tree/branch-0.1.4
-[branch-0.1.3]: https://github.com/apache/incubator-toree/tree/branch-0.1.3
+[0.1.x]: https://github.com/apache/incubator-toree/tree/0.1.x

[03/10] incubator-toree git commit: Squashed work for Apache Spark 2.0 and Scala 2.11

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/01936c1d/sparkr-interpreter/src/main/resources/R/pkg/R/sparkR.R
--
diff --git a/sparkr-interpreter/src/main/resources/R/pkg/R/sparkR.R 
b/sparkr-interpreter/src/main/resources/R/pkg/R/sparkR.R
deleted file mode 100644
index c445d1b..000
--- a/sparkr-interpreter/src/main/resources/R/pkg/R/sparkR.R
+++ /dev/null
@@ -1,360 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-.sparkREnv <- new.env()
-
-# Utility function that returns TRUE if we have an active connection to the
-# backend and FALSE otherwise
-connExists <- function(env) {
-  tryCatch({
-exists(".sparkRCon", envir = env) && isOpen(env[[".sparkRCon"]])
-  },
-  error = function(err) {
-return(FALSE)
-  })
-}
-
-#' Stop the Spark context.
-#'
-#' Also terminates the backend this R session is connected to
-sparkR.stop <- function() {
-  env <- .sparkREnv
-  if (exists(".sparkRCon", envir = env)) {
-# cat("Stopping SparkR\n")
-if (exists(".sparkRjsc", envir = env)) {
-  sc <- get(".sparkRjsc", envir = env)
-  callJMethod(sc, "stop")
-  rm(".sparkRjsc", envir = env)
-}
-
-if (exists(".backendLaunched", envir = env)) {
-  callJStatic("SparkRHandler", "stopBackend")
-}
-
-# Also close the connection and remove it from our env
-conn <- get(".sparkRCon", envir = env)
-close(conn)
-
-rm(".sparkRCon", envir = env)
-rm(".scStartTime", envir = env)
-  }
-
-  if (exists(".monitorConn", envir = env)) {
-conn <- get(".monitorConn", envir = env)
-close(conn)
-rm(".monitorConn", envir = env)
-  }
-
-  # Clear all broadcast variables we have
-  # as the jobj will not be valid if we restart the JVM
-  clearBroadcastVariables()
-
-  # Clear jobj maps
-  clearJobjs()
-}
-
-#' Initialize a new Spark Context.
-#'
-#' This function initializes a new SparkContext.
-#'
-#' @param master The Spark master URL.
-#' @param appName Application name to register with cluster manager
-#' @param sparkHome Spark Home directory
-#' @param sparkEnvir Named list of environment variables to set on worker 
nodes.
-#' @param sparkExecutorEnv Named list of environment variables to be used when 
launching executors.
-#' @param sparkJars Character string vector of jar files to pass to the worker 
nodes.
-#' @param sparkPackages Character string vector of packages from 
spark-packages.org
-#' @export
-#' @examples
-#'\dontrun{
-#' sc <- sparkR.init("local[2]", "SparkR", "/home/spark")
-#' sc <- sparkR.init("local[2]", "SparkR", "/home/spark",
-#'  list(spark.executor.memory="1g"))
-#' sc <- sparkR.init("yarn-client", "SparkR", "/home/spark",
-#'  list(spark.executor.memory="1g"),
-#'  list(LD_LIBRARY_PATH="/directory of JVM libraries 
(libjvm.so) on workers/"),
-#'  c("jarfile1.jar","jarfile2.jar"))
-#'}
-
-sparkR.init <- function(
-  master = "",
-  appName = "SparkR",
-  sparkHome = Sys.getenv("SPARK_HOME"),
-  sparkEnvir = list(),
-  sparkExecutorEnv = list(),
-  sparkJars = "",
-  sparkPackages = "") {
-
-  if (exists(".sparkRjsc", envir = .sparkREnv)) {
-cat(paste("Re-using existing Spark Context.",
-  "Please stop SparkR with sparkR.stop() or restart R to create a 
new Spark Context\n"))
-return(get(".sparkRjsc", envir = .sparkREnv))
-  }
-
-  jars <- suppressWarnings(normalizePath(as.character(sparkJars)))
-
-  # Classpath separator is ";" on Windows
-  # URI needs four /// as from http://stackoverflow.com/a/18522792
-  if (.Platform$OS.type == "unix") {
-uriSep <- "//"
-  } else {
-uriSep <- ""
-  }
-
-  existingPort <- Sys.getenv("EXISTING_SPARKR_BACKEND_PORT", "")
-  if (existingPort != "") {
-backendPort <- existingPort
-  } else {
-path <- tempfile(pattern = "backend_port")
-launchBackend(
-args = path,
-sparkHome = sparkHome,
-jars = jars,
-sparkSubmitOpts = Sys.getenv("SPARKR_SUBMIT_ARGS", "sparkr-shell"),
-packages = sparkPackages)
-# wait atmost 100 seconds for JVM to launch
-wait <- 0.1
-for (i in 1:25) {
-  Sys.sleep(wait)
-  if (file.exists(path)) {
-

[08/10] incubator-toree git commit: Squashed work for Apache Spark 2.0 and Scala 2.11

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/01936c1d/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
--
diff --git 
a/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
 
b/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
index b7d40f1..33fa861 100644
--- 
a/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
+++ 
b/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
@@ -17,18 +17,17 @@
 
 package org.apache.toree.kernel.interpreter.scala
 
-import java.io.{BufferedReader, ByteArrayOutputStream, InputStreamReader, 
PrintStream}
+import java.io.ByteArrayOutputStream
 import java.net.{URL, URLClassLoader}
 import java.nio.charset.Charset
 import java.util.concurrent.ExecutionException
 
-import com.typesafe.config.{ConfigFactory, Config}
+import com.typesafe.config.{Config, ConfigFactory}
 import org.apache.spark.SparkContext
-import org.apache.spark.repl.{SparkCommandLine, SparkIMain, 
SparkJLineCompletion}
-import org.apache.spark.sql.SQLContext
-import org.apache.toree.global.StreamState
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.repl.Main
+
 import org.apache.toree.interpreter._
-import org.apache.toree.interpreter.imports.printers.{WrapperConsole, 
WrapperSystem}
 import org.apache.toree.kernel.api.{KernelLike, KernelOptions}
 import org.apache.toree.utils.{MultiOutputStream, TaskManager}
 import org.slf4j.LoggerFactory
@@ -36,571 +35,306 @@ import org.slf4j.LoggerFactory
 import scala.annotation.tailrec
 import scala.concurrent.{Await, Future}
 import scala.language.reflectiveCalls
-import scala.tools.nsc.backend.JavaPlatform
-import scala.tools.nsc.interpreter.{IR, InputStream, JPrintWriter, 
OutputStream}
-import scala.tools.nsc.io.AbstractFile
-import scala.tools.nsc.util.{ClassPath, MergedClassPath}
-import scala.tools.nsc.{Global, Settings, io}
+import scala.tools.nsc.Settings
+import scala.tools.nsc.interpreter.{IR, OutputStream}
+import scala.tools.nsc.util.ClassPath
 import scala.util.{Try => UtilTry}
 
-class ScalaInterpreter(private val config:Config = ConfigFactory.load) extends 
Interpreter {
-
-  protected val logger = LoggerFactory.getLogger(this.getClass.getName)
-
-  private val ExecutionExceptionName = "lastException"
-  protected val _thisClassloader = this.getClass.getClassLoader
-
-  protected val _runtimeClassloader =
-new URLClassLoader(Array(), _thisClassloader) {
-  def addJar(url: URL) = this.addURL(url)
-}
-  protected val lastResultOut = new ByteArrayOutputStream()
+class ScalaInterpreter(private val config:Config = ConfigFactory.load) extends 
Interpreter with ScalaInterpreterSpecific {
+   protected val logger = LoggerFactory.getLogger(this.getClass.getName)
 
+   protected val _thisClassloader = this.getClass.getClassLoader
 
-  protected val multiOutputStream = MultiOutputStream(List(Console.out, 
lastResultOut))
-  private var taskManager: TaskManager = _
-  var sparkIMain: SparkIMain = _
-  protected var jLineCompleter: SparkJLineCompletion = _
+   protected val lastResultOut = new ByteArrayOutputStream()
 
-  protected var settings: Settings = newSettings(interpreterArgs())
+   protected val multiOutputStream = MultiOutputStream(List(Console.out, 
lastResultOut))
+   private[scala] var taskManager: TaskManager = _
 
-  settings.classpath.value = buildClasspath(_thisClassloader)
-  settings.embeddedDefaults(_runtimeClassloader)
-
-  private val maxInterpreterThreads: Int = {
-if(config.hasPath("max_interpreter_threads"))
-  config.getInt("max_interpreter_threads")
-else
-  TaskManager.DefaultMaximumWorkers
+  /** Since the ScalaInterpreter can be started without a kernel, we need to 
ensure that we can compile things.
+  Adding in the default classpaths as needed.
+*/
+  def appendClassPath(settings: Settings): Settings = {
+settings.classpath.value = buildClasspath(_thisClassloader)
+settings.embeddedDefaults(_runtimeClassloader)
+settings
   }
 
-  protected def newSparkIMain(
-settings: Settings, out: JPrintWriter
-  ): SparkIMain = {
-val s = new SparkIMain(settings, out)
-s.initializeSynchronous()
+  protected var settings: Settings = newSettings(List())
+  settings = appendClassPath(settings)
 
-s
-  }
 
-  protected def newTaskManager(): TaskManager =
-new TaskManager(maximumWorkers = maxInterpreterThreads)
+  private val maxInterpreterThreads: Int = {
+ if(config.hasPath("max_interpreter_threads"))
+   config.getInt("max_interpreter_threads")
+ else
+   TaskManager.DefaultMaximumWorkers
+   }
 
-  protected def newSettings(args: List[String]): Settings =
-new SparkCommandLine(args).settings
+   protected def newTaskManager(): 

[02/10] incubator-toree git commit: Squashed work for Apache Spark 2.0 and Scala 2.11

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/01936c1d/sparkr-interpreter/src/main/resources/R/pkg/inst/tests/test_rdd.R
--
diff --git a/sparkr-interpreter/src/main/resources/R/pkg/inst/tests/test_rdd.R 
b/sparkr-interpreter/src/main/resources/R/pkg/inst/tests/test_rdd.R
deleted file mode 100644
index 71aed2b..000
--- a/sparkr-interpreter/src/main/resources/R/pkg/inst/tests/test_rdd.R
+++ /dev/null
@@ -1,793 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-context("basic RDD functions")
-
-# JavaSparkContext handle
-sc <- sparkR.init()
-
-# Data
-nums <- 1:10
-rdd <- parallelize(sc, nums, 2L)
-
-intPairs <- list(list(1L, -1), list(2L, 100), list(2L, 1), list(1L, 200))
-intRdd <- parallelize(sc, intPairs, 2L)
-
-test_that("get number of partitions in RDD", {
-  expect_equal(numPartitions(rdd), 2)
-  expect_equal(numPartitions(intRdd), 2)
-})
-
-test_that("first on RDD", {
-  expect_equal(first(rdd), 1)
-  newrdd <- lapply(rdd, function(x) x + 1)
-  expect_equal(first(newrdd), 2)
-})
-
-test_that("count and length on RDD", {
-   expect_equal(count(rdd), 10)
-   expect_equal(length(rdd), 10)
-})
-
-test_that("count by values and keys", {
-  mods <- lapply(rdd, function(x) { x %% 3 })
-  actual <- countByValue(mods)
-  expected <- list(list(0, 3L), list(1, 4L), list(2, 3L))
-  expect_equal(sortKeyValueList(actual), sortKeyValueList(expected))
-
-  actual <- countByKey(intRdd)
-  expected <- list(list(2L, 2L), list(1L, 2L))
-  expect_equal(sortKeyValueList(actual), sortKeyValueList(expected))
-})
-
-test_that("lapply on RDD", {
-  multiples <- lapply(rdd, function(x) { 2 * x })
-  actual <- collect(multiples)
-  expect_equal(actual, as.list(nums * 2))
-})
-
-test_that("lapplyPartition on RDD", {
-  sums <- lapplyPartition(rdd, function(part) { sum(unlist(part)) })
-  actual <- collect(sums)
-  expect_equal(actual, list(15, 40))
-})
-
-test_that("mapPartitions on RDD", {
-  sums <- mapPartitions(rdd, function(part) { sum(unlist(part)) })
-  actual <- collect(sums)
-  expect_equal(actual, list(15, 40))
-})
-
-test_that("flatMap() on RDDs", {
-  flat <- flatMap(intRdd, function(x) { list(x, x) })
-  actual <- collect(flat)
-  expect_equal(actual, rep(intPairs, each=2))
-})
-
-test_that("filterRDD on RDD", {
-  filtered.rdd <- filterRDD(rdd, function(x) { x %% 2 == 0 })
-  actual <- collect(filtered.rdd)
-  expect_equal(actual, list(2, 4, 6, 8, 10))
-
-  filtered.rdd <- Filter(function(x) { x[[2]] < 0 }, intRdd)
-  actual <- collect(filtered.rdd)
-  expect_equal(actual, list(list(1L, -1)))
-
-  # Filter out all elements.
-  filtered.rdd <- filterRDD(rdd, function(x) { x > 10 })
-  actual <- collect(filtered.rdd)
-  expect_equal(actual, list())
-})
-
-test_that("lookup on RDD", {
-  vals <- lookup(intRdd, 1L)
-  expect_equal(vals, list(-1, 200))
-
-  vals <- lookup(intRdd, 3L)
-  expect_equal(vals, list())
-})
-
-test_that("several transformations on RDD (a benchmark on PipelinedRDD)", {
-  rdd2 <- rdd
-  for (i in 1:12)
-rdd2 <- lapplyPartitionsWithIndex(
-  rdd2, function(partIndex, part) {
-part <- as.list(unlist(part) * partIndex + i)
-  })
-  rdd2 <- lapply(rdd2, function(x) x + x)
-  actual <- collect(rdd2)
-  expected <- list(24, 24, 24, 24, 24,
-   168, 170, 172, 174, 176)
-  expect_equal(actual, expected)
-})
-
-test_that("PipelinedRDD support actions: cache(), persist(), unpersist(), 
checkpoint()", {
-  # RDD
-  rdd2 <- rdd
-  # PipelinedRDD
-  rdd2 <- lapplyPartitionsWithIndex(
-rdd2,
-function(partIndex, part) {
-  part <- as.list(unlist(part) * partIndex)
-})
-
-  cache(rdd2)
-  expect_true(rdd2@env$isCached)
-  rdd2 <- lapply(rdd2, function(x) x)
-  expect_false(rdd2@env$isCached)
-
-  unpersist(rdd2)
-  expect_false(rdd2@env$isCached)
-
-  persist(rdd2, "MEMORY_AND_DISK")
-  expect_true(rdd2@env$isCached)
-  rdd2 <- lapply(rdd2, function(x) x)
-  expect_false(rdd2@env$isCached)
-
-  unpersist(rdd2)
-  expect_false(rdd2@env$isCached)
-
-  tempDir <- tempfile(pattern = "checkpoint")
-  setCheckpointDir(sc, tempDir)
-  checkpoint(rdd2)
-  

[04/10] incubator-toree git commit: Squashed work for Apache Spark 2.0 and Scala 2.11

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/01936c1d/sparkr-interpreter/src/main/resources/R/pkg/R/generics.R
--
diff --git a/sparkr-interpreter/src/main/resources/R/pkg/R/generics.R 
b/sparkr-interpreter/src/main/resources/R/pkg/R/generics.R
deleted file mode 100644
index 43dd8d2..000
--- a/sparkr-interpreter/src/main/resources/R/pkg/R/generics.R
+++ /dev/null
@@ -1,985 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
- RDD Actions and Transformations 
-
-# @rdname aggregateRDD
-# @seealso reduce
-# @export
-setGeneric("aggregateRDD",
-   function(x, zeroValue, seqOp, combOp) { 
standardGeneric("aggregateRDD") })
-
-# @rdname cache-methods
-# @export
-setGeneric("cache", function(x) { standardGeneric("cache") })
-
-# @rdname coalesce
-# @seealso repartition
-# @export
-setGeneric("coalesce", function(x, numPartitions, ...) { 
standardGeneric("coalesce") })
-
-# @rdname checkpoint-methods
-# @export
-setGeneric("checkpoint", function(x) { standardGeneric("checkpoint") })
-
-# @rdname collect-methods
-# @export
-setGeneric("collect", function(x, ...) { standardGeneric("collect") })
-
-# @rdname collect-methods
-# @export
-setGeneric("collectAsMap", function(x) { standardGeneric("collectAsMap") })
-
-# @rdname collect-methods
-# @export
-setGeneric("collectPartition",
-   function(x, partitionId) {
- standardGeneric("collectPartition")
-   })
-
-# @rdname count
-# @export
-setGeneric("count", function(x) { standardGeneric("count") })
-
-# @rdname countByValue
-# @export
-setGeneric("countByValue", function(x) { standardGeneric("countByValue") })
-
-# @rdname statfunctions
-# @export
-setGeneric("crosstab", function(x, col1, col2) { standardGeneric("crosstab") })
-
-# @rdname distinct
-# @export
-setGeneric("distinct", function(x, numPartitions = 1) { 
standardGeneric("distinct") })
-
-# @rdname filterRDD
-# @export
-setGeneric("filterRDD", function(x, f) { standardGeneric("filterRDD") })
-
-# @rdname first
-# @export
-setGeneric("first", function(x) { standardGeneric("first") })
-
-# @rdname flatMap
-# @export
-setGeneric("flatMap", function(X, FUN) { standardGeneric("flatMap") })
-
-# @rdname fold
-# @seealso reduce
-# @export
-setGeneric("fold", function(x, zeroValue, op) { standardGeneric("fold") })
-
-# @rdname foreach
-# @export
-setGeneric("foreach", function(x, func) { standardGeneric("foreach") })
-
-# @rdname foreach
-# @export
-setGeneric("foreachPartition", function(x, func) { 
standardGeneric("foreachPartition") })
-
-# The jrdd accessor function.
-setGeneric("getJRDD", function(rdd, ...) { standardGeneric("getJRDD") })
-
-# @rdname glom
-# @export
-setGeneric("glom", function(x) { standardGeneric("glom") })
-
-# @rdname keyBy
-# @export
-setGeneric("keyBy", function(x, func) { standardGeneric("keyBy") })
-
-# @rdname lapplyPartition
-# @export
-setGeneric("lapplyPartition", function(X, FUN) { 
standardGeneric("lapplyPartition") })
-
-# @rdname lapplyPartitionsWithIndex
-# @export
-setGeneric("lapplyPartitionsWithIndex",
-   function(X, FUN) {
- standardGeneric("lapplyPartitionsWithIndex")
-   })
-
-# @rdname lapply
-# @export
-setGeneric("map", function(X, FUN) { standardGeneric("map") })
-
-# @rdname lapplyPartition
-# @export
-setGeneric("mapPartitions", function(X, FUN) { 
standardGeneric("mapPartitions") })
-
-# @rdname lapplyPartitionsWithIndex
-# @export
-setGeneric("mapPartitionsWithIndex",
-   function(X, FUN) { standardGeneric("mapPartitionsWithIndex") })
-
-# @rdname maximum
-# @export
-setGeneric("maximum", function(x) { standardGeneric("maximum") })
-
-# @rdname minimum
-# @export
-setGeneric("minimum", function(x) { standardGeneric("minimum") })
-
-# @rdname sumRDD
-# @export
-setGeneric("sumRDD", function(x) { standardGeneric("sumRDD") })
-
-# @rdname name
-# @export
-setGeneric("name", function(x) { standardGeneric("name") })
-
-# @rdname numPartitions
-# @export
-setGeneric("numPartitions", function(x) { standardGeneric("numPartitions") })
-
-# @rdname persist
-# @export
-setGeneric("persist", function(x, newLevel) { standardGeneric("persist") })
-
-# 

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/release [deleted] 70c2b7d55

incubator-toree git commit: [ci skip] 0.1.0 rc1 [Forced Update!]

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x 748abdb28 -> 3afb5eab3 (forced update)


[ci skip] 0.1.0 rc1


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/3afb5eab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/3afb5eab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/3afb5eab

Branch: refs/heads/0.1.x
Commit: 3afb5eab39f4dc8e8a95b84d42ff69281201fc09
Parents: 8801d62
Author: Gino Bustelo 
Authored: Wed Oct 5 15:38:54 2016 -0500
Committer: Gino Bustelo 
Committed: Thu Oct 6 13:15:20 2016 -0500

--
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/3afb5eab/Makefile
--
diff --git a/Makefile b/Makefile
index 5c033e9..c84a22f 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
 
 .PHONY: help clean clean-dist build dev test system-test test-travis release 
pip-release bin-release dev-binder .binder-image audit audit-licenses
 
-BASE_VERSION=0.1.0.dev9
+BASE_VERSION=0.1.0
 VERSION=$(BASE_VERSION)-incubating
 COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
 ifeq (, $(findstring dev, $(VERSION)))

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/0.1.0 [deleted] 71ef911a7

incubator-toree git commit: [TOREE-347] Add explicit guava dependency

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x 3afb5eab3 -> f68ffdcdb


[TOREE-347] Add explicit guava dependency


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/f68ffdcd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/f68ffdcd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/f68ffdcd

Branch: refs/heads/0.1.x
Commit: f68ffdcdb8da4b2206af0b441f77c159b8866599
Parents: 3afb5ea
Author: Gino Bustelo 
Authored: Thu Oct 6 13:26:04 2016 -0500
Committer: Gino Bustelo 
Committed: Thu Oct 6 13:26:04 2016 -0500

--
 kernel/build.sbt | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/f68ffdcd/kernel/build.sbt
--
diff --git a/kernel/build.sbt b/kernel/build.sbt
index 4ca4194..cc22ba5 100644
--- a/kernel/build.sbt
+++ b/kernel/build.sbt
@@ -18,6 +18,8 @@
 //
 // TEST DEPENDENCIES
 //
-libraryDependencies +=
-  "org.spark-project.akka" %% "akka-testkit" % "2.3.4-spark" % "test" // MIT
+libraryDependencies ++=
+  Seq("org.spark-project.akka" %% "akka-testkit" % "2.3.4-spark" % "test", // 
MIT
+"com.google.guava"  % "guava" % "19.0")
+
 

incubator-toree git commit: [ci skip] Release 0.1.0

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x 8801d6210 -> a5d76b380
Updated Tags:  refs/tags/0.1.0 [created] 71ef911a7


[ci skip] Release 0.1.0


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/a5d76b38
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/a5d76b38
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/a5d76b38

Branch: refs/heads/0.1.x
Commit: a5d76b380617b058b0f8b680d8e19f9c3bce0b71
Parents: 8801d62
Author: Gino Bustelo 
Authored: Wed Oct 5 15:38:54 2016 -0500
Committer: Gino Bustelo 
Committed: Wed Oct 5 15:39:09 2016 -0500

--
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/a5d76b38/Makefile
--
diff --git a/Makefile b/Makefile
index 5c033e9..c84a22f 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
 
 .PHONY: help clean clean-dist build dev test system-test test-travis release 
pip-release bin-release dev-binder .binder-image audit audit-licenses
 
-BASE_VERSION=0.1.0.dev9
+BASE_VERSION=0.1.0
 VERSION=$(BASE_VERSION)-incubating
 COMMIT=$(shell git rev-parse --short=12 --verify HEAD)
 ifeq (, $(findstring dev, $(VERSION)))

[3/4] incubator-toree git commit: Candidate Release 0.1.0

[lbustelo]

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/70c2b7d5/toree-pip/licenses/LICENSE-jline.txt
--
diff --git a/toree-pip/licenses/LICENSE-jline.txt 
b/toree-pip/licenses/LICENSE-jline.txt
new file mode 100644
index 000..0565aea
--- /dev/null
+++ b/toree-pip/licenses/LICENSE-jline.txt
@@ -0,0 +1,902 @@
+
+
+
+
+
+
+  http://ogp.me/ns# fb: http://ogp.me/ns/fb# object: 
http://ogp.me/ns/object# article: http://ogp.me/ns/article# profile: 
http://ogp.me/ns/profile#;>
+
+
+https://assets-cdn.github.com/assets/frameworks-473733caf98c6c528c9ddeb25f5830e456cb75553a54d79c8c3296648555f848.css;
 media="all" rel="stylesheet" />
+https://assets-cdn.github.com/assets/github-9366a97c2c7a861b73bf6d1bf3b461762a103f28da88b1a7612bd0a7d26c3542.css;
 media="all" rel="stylesheet" />
+
+
+
+https://assets-cdn.github.com/assets/site-2d5d1699983d8dd7c384c67dfc183fbace8e3a24866e8822069d599f68e71fb2.css;
 media="all" rel="stylesheet" />
+
+https://assets-cdn.github.com/assets/frameworks-9eb0ddac6d1c94c56980360abcade2abb10d27d7880f7ee7bf6640c2c2a06fc8.js;
 rel="preload" />
+
+https://assets-cdn.github.com/assets/github-4ddbe43ff9bf343ea4e58a83f44f72427d4c0e030722f47eb6ea4d231abfa161.js;
 rel="preload" />
+
+
+
+
+
+
+jline2/LICENSE.txt at master · jline/jline2 · GitHub
+
+https://github.com/fluidicon.png; 
title="GitHub">
+
+
+
+
+
+
+
+
+
+
+
+
+  https://avatars0.githubusercontent.com/u/711075?v=3s=400; 
name="twitter:image:src" />
+  https://avatars0.githubusercontent.com/u/711075?v=3s=400; 
property="og:image" />https://github.com/jline/jline2; 
property="og:url" />
+  https://api.github.com/_private/browser/stats;>
+https://api.github.com/_private/browser/errors;>
+https://assets-cdn.github.com/;>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  
+
+
+
+
+
+
+
+  
+
+  https://assets-cdn.github.com/pinned-octocat.svg; color="#4078c0">
+  https://assets-cdn.github.com/favicon.ico;>
+
+
+
+
+
+
+  
+  
+  https://github.com/jline/jline2.git;>
+
+  
+  https://github.com/jline/jline2/commits/master.atom; 
rel="alternate" title="Recent Commits to jline2:master" 
type="application/atom+xml">
+
+
+  https://github.com/jline/jline2/blob/master/LICENSE.txt; 
data-pjax-transient>
+  
+
+
+  
+
+Skip to content
+
+
+
+
+
+
+
+  
+  
+https://github.com/; 
aria-label="Homepage" data-ga-click="(Logged out) Header, go to homepage, 
icon:logo-wordmark">
+  
+
+
+
+  
+
+
+
+  
+
+  Personal
+
+  Open source
+
+  Business
+
+  Explore
+  
+
+  
+Sign up
+  Sign in
+  
+
+
+  Pricing
+  Blog
+  https://help.github.com;>Support
+  https://github.com/search;>Search GitHub
+  
+  
+
+  This repository
+  
+
+
+
+
+
+  
+
+
+
+
+
+
+  
+
+
+
+
+http://schema.org/SoftwareSourceCode;>
+
+  
+
+  
+
+
+
+
+
+  
+  
+
+Watch
+  
+  
+52
+  
+
+  
+
+  
+  
+
+Star
+  
+
+
+  516
+
+
+  
+
+  
+  
+
+Fork
+  
+
+
+  143
+
+  
+
+
+
+  
+  jline/jline2
+
+
+
+  
+  
+
+http://schema.org/BreadcrumbList;
+ role="navigation"
+ data-pjax="#js-repo-pjax-container">
+
+  http://schema.org/ListItem; 
itemprop="itemListElement">
+
+  
+  Code
+  
+  
+
+http://schema.org/ListItem; 
itemprop="itemListElement">
+  
+
+Issues
+42
+
+
+
+  http://schema.org/ListItem; 
itemprop="itemListElement">
+
+  
+  Pull requests
+  10
+  
+  
+
+
+  
+  Wiki
+
+  
+
+Pulse
+
+  
+
+Graphs
+
+
+
+
+  
+
+
+
+  
+
+
+
+Permalink
+
+
+
+
+  
+
+  
+Branch:
+master
+  
+
+  
+
+
+  
+
+Switch branches/tags
+  
+
+  
+
+  
+
+
+  
+
+  Branches
+
+
+  Tags
+
+  
+
+  
+
+  
+
+
+
+
+
+  
+  
+consistent-license-headers
+  
+
+
+  
+  
+gh-pages
+  
+
+
+  
+  
+jline3
+  
+
+
+  
+  
+master
+  
+
+
+  
+  
+non_bmp_characters
+  
+
+
+  
+  

[4/4] incubator-toree git commit: Candidate Release 0.1.0

[lbustelo]

Candidate Release 0.1.0


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/70c2b7d5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/70c2b7d5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/70c2b7d5

Branch: refs/heads/release
Commit: 70c2b7d55ed3f42400844c2c5c311a70d5afbefe
Parents: 
Author: Gino Bustelo 
Authored: Wed Oct 5 15:54:59 2016 -0500
Committer: Gino Bustelo 
Committed: Wed Oct 5 15:54:59 2016 -0500

--
 .gitignore  |   1 +
 DISCLAIMER  |  10 +
 LICENSE | 202 +
 NOTICE  |   5 +
 ...toree-0.1.0-incubating-binary-release.tar.gz | Bin 0 -> 23882513 bytes
 ...e-0.1.0-incubating-binary-release.tar.gz.asc |  17 +
 ...e-0.1.0-incubating-binary-release.tar.gz.md5 |   2 +
 ...e-0.1.0-incubating-binary-release.tar.gz.sha |   3 +
 toree-pip/DISCLAIMER|  10 +
 toree-pip/LICENSE   | 256 ++
 toree-pip/MANIFEST.in   |   5 +
 toree-pip/NOTICE|   7 +
 toree-pip/RELEASE_NOTES.md  |  17 +
 toree-pip/VERSION   |   2 +
 toree-pip/licenses/LICENSE-ammonite.txt |  25 +
 toree-pip/licenses/LICENSE-asm.txt  |  29 +
 toree-pip/licenses/LICENSE-clapper.txt  |  31 +
 toree-pip/licenses/LICENSE-jeromq.txt   | 373 
 toree-pip/licenses/LICENSE-jline.txt| 902 +++
 toree-pip/licenses/LICENSE-jopt-simple.txt  |  24 +
 toree-pip/licenses/LICENSE-scala-stm.txt|  24 +
 toree-pip/licenses/LICENSE-scala.txt|  11 +
 toree-pip/licenses/LICENSE-scalaz.txt   |  24 +
 toree-pip/licenses/LICENSE-slf4j.txt|  21 +
 toree-pip/setup.py  |  80 ++
 toree-pip/toree-0.1.0.tar.gz| Bin 0 -> 23875080 bytes
 toree-pip/toree-0.1.0.tar.gz.asc|  17 +
 toree-pip/toree-0.1.0.tar.gz.md5|   2 +
 toree-pip/toree-0.1.0.tar.gz.sha|   3 +
 toree-pip/toree.egg-info/PKG-INFO   |  29 +
 toree-pip/toree.egg-info/SOURCES.txt|  31 +
 toree-pip/toree.egg-info/dependency_links.txt   |   1 +
 toree-pip/toree.egg-info/entry_points.txt   |   3 +
 toree-pip/toree.egg-info/requires.txt   |   3 +
 toree-pip/toree.egg-info/top_level.txt  |   1 +
 toree-pip/toree/DISCLAIMER  |  10 +
 toree-pip/toree/LICENSE | 256 ++
 toree-pip/toree/NOTICE  |   7 +
 toree-pip/toree/RELEASE_NOTES.md|  17 +
 toree-pip/toree/VERSION |   2 +
 toree-pip/toree/__init__.py |  18 +
 toree-pip/toree/__main__.py |  21 +
 toree-pip/toree/_version.py |  20 +
 toree-pip/toree/bin/run.sh  |  45 +
 .../lib/toree-assembly-0.1.0-incubating.jar | Bin 0 -> 27054767 bytes
 toree-pip/toree/licenses/LICENSE-ammonite.txt   |  25 +
 toree-pip/toree/licenses/LICENSE-asm.txt|  29 +
 toree-pip/toree/licenses/LICENSE-clapper.txt|  31 +
 toree-pip/toree/licenses/LICENSE-jeromq.txt | 373 
 toree-pip/toree/licenses/LICENSE-jline.txt  | 902 +++
 .../toree/licenses/LICENSE-jopt-simple.txt  |  24 +
 toree-pip/toree/licenses/LICENSE-scala-stm.txt  |  24 +
 toree-pip/toree/licenses/LICENSE-scala.txt  |  11 +
 toree-pip/toree/licenses/LICENSE-scalaz.txt |  24 +
 toree-pip/toree/licenses/LICENSE-slf4j.txt  |  21 +
 toree-pip/toree/toreeapp.py | 167 
 ...toree-0.1.0-incubating-source-release.tar.gz | Bin 0 -> 2984428 bytes
 ...e-0.1.0-incubating-source-release.tar.gz.asc |  17 +
 ...e-0.1.0-incubating-source-release.tar.gz.md5 |   2 +
 ...e-0.1.0-incubating-source-release.tar.gz.sha |   3 +
 60 files changed, 4220 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/70c2b7d5/.gitignore
--
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 000..e43b0f9
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+.DS_Store

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/70c2b7d5/DISCLAIMER
--
diff --git a/DISCLAIMER b/DISCLAIMER
new file mode 100644
index 000..5aac7f4
--- /dev/null
+++ b/DISCLAIMER
@@ -0,0 +1,10 @@
+Apache Toree is an effort undergoing incubation at the Apache Software
+Foundation (ASF), sponsored by 

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.2 [deleted] 16919853d

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.2-jeromq [deleted] b673937de

incubator-toree git commit: [TOREE-334] java.util.NoSuchElementException on empty magic

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x 9e304948e -> b811f1025


[TOREE-334] java.util.NoSuchElementException on empty magic

Change magicRegex to handle `%` magic line


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/b811f102
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/b811f102
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/b811f102

Branch: refs/heads/0.1.x
Commit: b811f10251f156cd71686a620e3ab084cba59622
Parents: 9e30494
Author: Maciej Bryński 
Authored: Wed Aug 24 18:07:21 2016 +0200
Committer: Gino Bustelo 
Committed: Wed Aug 24 13:25:25 2016 -0500

--
 .../org/apache/toree/kernel/protocol/v5/magic/MagicParser.scala| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/b811f102/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/magic/MagicParser.scala
--
diff --git 
a/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/magic/MagicParser.scala
 
b/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/magic/MagicParser.scala
index bfe518d..9a912d7 100644
--- 
a/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/magic/MagicParser.scala
+++ 
b/kernel/src/main/scala/org/apache/toree/kernel/protocol/v5/magic/MagicParser.scala
@@ -22,7 +22,7 @@ import org.apache.toree.magic.MagicManager
 import scala.util.Try
 
 class MagicParser(private val magicManager: MagicManager) {
-  private val magicRegex = """^[%]{1,2}(\w+)""".r
+  private val magicRegex = """^[%]{1,2}(\w*)""".r
   protected[magic] val kernelObjectName = "kernel.magics"
 
   /**

svn commit: r16710 - in /dev/incubator/toree/0.1.0/rc3: ./ toree-bin/ toree-pip/ toree-src/

[lbustelo]

Author: lbustelo
Date: Wed Oct 26 20:13:05 2016
New Revision: 16710

Log:
0.1.0 rc3

Added:
dev/incubator/toree/0.1.0/rc3/
dev/incubator/toree/0.1.0/rc3/toree-bin/

dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz
   (with props)

dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc

dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5

dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
dev/incubator/toree/0.1.0/rc3/toree-pip/
dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz   (with 
props)
dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz.asc
dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz.md5
dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz.sha
dev/incubator/toree/0.1.0/rc3/toree-src/

dev/incubator/toree/0.1.0/rc3/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz
   (with props)

dev/incubator/toree/0.1.0/rc3/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz.asc

dev/incubator/toree/0.1.0/rc3/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz.md5

dev/incubator/toree/0.1.0/rc3/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz.sha

Added: 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz
==
Binary file - no diff available.

Propchange: 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz
--
svn:mime-type = application/octet-stream

Added: 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc
==
--- 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc
 (added)
+++ 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc
 Wed Oct 26 20:13:05 2016
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJYEQauAAoJEErQor8BMK2masUP/iO/yw6LqlSpkpkQOGWMLxA4
+glozqMjM4tBDbDHI5ZiZbE/vW8vx29hRVb/1yHkMiL1LewyMy8V/23TOZk/S3UxO
+aSbKVIBUFdq7VjcWO4RD5PREkOt/p9Du7HH9rHXyoMJgaCXsc5MGY1ESYNtp+sLJ
+MiMZpWMGaDPFRbeQtlLixwNdgNDQhwywhpL38qJ9G7zbDaBRU9XxJmMpEAQl8DY9
++vOgN397S4sb62GLpoKrdmKFaWO0FEoL3XQUE8q+MrYQ8NnO5SuMgZkJGWVLtJHr
+qqTWWNmvr5iJs4PuwPeaY3SczYSuMFp+KW9X1w3my47IvFvflJ32fDpSdUFU0Hnc
+hl0Jzd3fdvYmJwBGQaruRqx6JJVd3Wpkk+NLqnQhg3joqOm/v1TmBz+pS+xmvPXm
+C8jlJx7bZiuK9Gj64igm9WyKhbFaAFRQsIvju+h0UWwEMfHIwXkt03C4q23CF1UL
+Jv34Iw+1rkZ4fXKTDsRHqjL3hIDCSR/Vj/l9m9XKKZk+vhB/vnrYlm8N17NqDL3q
+GjBN+/LUUFTqoP0y8Nh3dnwBAu+byfpr9jk/vcT3WB0mCCqTCwA0ffaioiADJLWD
+bkglUVLGSX55CXQA9JSnVy7MogTWbFb2z/nwQ8OAvA66idYvbr/k212LhV7pt/+j
+sfgRadq0PAtCQTw6MW8y
+=76p6
+-END PGP SIGNATURE-

Added: 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5
==
--- 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5
 (added)
+++ 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5
 Wed Oct 26 20:13:05 2016
@@ -0,0 +1,2 @@
+/Users/ginobustelo/Dev/spark/toree/dist/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz:
 
+16 A1 A9 D3 38 A3 8E 69  31 BF BB 62 F5 30 29 D1

Added: 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
==
--- 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
 (added)
+++ 
dev/incubator/toree/0.1.0/rc3/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
 Wed Oct 26 20:13:05 2016
@@ -0,0 +1,3 @@
+/Users/ginobustelo/Dev/spark/toree/dist/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz:
 
+0B18C021 89E05CB5 BB8AEA3E F5E27A1C FAF33457 F9EFC0F4 002A7A53 CBD373B5 
B0009655
+ 6F43F623 DF0DEAD5 FDBFFE0B C2EC5162 021B2C90 277BA8E2 9191F90B

Added: dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz
==
Binary file - no diff available.

Propchange: dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/incubator/toree/0.1.0/rc3/toree-pip/apache-toree-0.1.0.tar.gz.asc
==
--- dev/incubator/toree/0.1.0/rc3/toree-pip

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.2 [deleted] 16919853d

incubator-toree git commit: make jackson a test-only dependency

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x 119bf3e2d -> cfc7883c1


make jackson a test-only dependency


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/cfc7883c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/cfc7883c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/cfc7883c

Branch: refs/heads/0.1.x
Commit: cfc7883c1ad805084ccbabaf457d4122db84ddcf
Parents: 119bf3e
Author: Oliver Köth 
Authored: Thu Oct 20 09:56:58 2016 +0200
Committer: Gino Bustelo 
Committed: Thu Oct 20 08:42:06 2016 -0500

--
 project/common.scala | 4 +++-
 protocol/build.sbt   | 3 +--
 2 files changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/cfc7883c/project/common.scala
--
diff --git a/project/common.scala b/project/common.scala
index 10f5a9a..b924859 100644
--- a/project/common.scala
+++ b/project/common.scala
@@ -138,7 +138,9 @@ object Common {
 // Test dependencies
 libraryDependencies ++= Seq(
   "org.scalatest" %% "scalatest" % "2.2.6" % "test", // Apache v2
-  "org.mockito" % "mockito-all" % "1.10.19" % "test"   // MIT
+  "org.mockito" % "mockito-all" % "1.10.19" % "test",   // MIT
+  // use the same jackson version in test than the one provided at runtime 
by Spark 2.0.0
+  "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5" % "test" // 
Apache v2
 ),
 ScoverageSbtPlugin.ScoverageKeys.coverageHighlighting := false,
 pomExtra :=

http://git-wip-us.apache.org/repos/asf/incubator-toree/blob/cfc7883c/protocol/build.sbt
--
diff --git a/protocol/build.sbt b/protocol/build.sbt
index 731761a..f82b465 100644
--- a/protocol/build.sbt
+++ b/protocol/build.sbt
@@ -29,6 +29,5 @@ libraryDependencies ++= Seq(
 // TEST DEPENDENCIES
 //
 libraryDependencies ++= Seq(
-  "org.scalactic" %% "scalactic" % "2.2.6" % "test", // Apache v2
-  "com.fasterxml.jackson.core" % "jackson-databind" % "2.7.4" // Apache v2
+  "org.scalactic" %% "scalactic" % "2.2.6" % "test" // Apache v2
 )

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.0rc3 [deleted] 73c60b208

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.2-jeromq [deleted] b673937de

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.0-rc3 [created] 73c60b208
  refs/tags/v0.1.2-jeromq [created] b673937de
  refs/tags/v0.1.5-assembly [created] ecfa5dd2a

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.5-assembly [deleted] ecfa5dd2a

incubator-toree git commit: [ci skip] 0.1.0 rc2

[lbustelo]

Repository: incubator-toree
Updated Branches:
  refs/heads/0.1.x 7aceef82e -> 119bf3e2d


[ci skip] 0.1.0 rc2


Project: http://git-wip-us.apache.org/repos/asf/incubator-toree/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-toree/commit/119bf3e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-toree/tree/119bf3e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-toree/diff/119bf3e2

Branch: refs/heads/0.1.x
Commit: 119bf3e2d1d16986f55802cf2323e8629ea25ef8
Parents: 7aceef8
Author: Gino Bustelo 
Authored: Tue Oct 11 13:08:27 2016 -0500
Committer: Gino Bustelo 
Committed: Tue Oct 11 13:08:27 2016 -0500

--

--

svn commit: r16474 - in /dev/incubator/toree/0.1.0/rc1: toree-bin/ toree-pip/ toree-src/

[lbustelo]

Author: lbustelo
Date: Tue Oct 11 18:30:17 2016
New Revision: 16474

Log:
Removing Toree 0.1.0 rc1

Removed:
dev/incubator/toree/0.1.0/rc1/toree-bin/
dev/incubator/toree/0.1.0/rc1/toree-pip/
dev/incubator/toree/0.1.0/rc1/toree-src/

svn commit: r16473 - in /dev/incubator/toree/0.1.0/rc2: ./ toree-bin/ toree-pip/ toree-src/

[lbustelo]

Author: lbustelo
Date: Tue Oct 11 18:26:29 2016
New Revision: 16473

Log:
Toree 0.1.0 rc2

Added:
dev/incubator/toree/0.1.0/rc2/
dev/incubator/toree/0.1.0/rc2/toree-bin/

dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz
   (with props)

dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc

dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5

dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
dev/incubator/toree/0.1.0/rc2/toree-pip/
dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz   (with 
props)
dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz.asc
dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz.md5
dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz.sha
dev/incubator/toree/0.1.0/rc2/toree-src/

dev/incubator/toree/0.1.0/rc2/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz
   (with props)

dev/incubator/toree/0.1.0/rc2/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz.asc

dev/incubator/toree/0.1.0/rc2/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz.md5

dev/incubator/toree/0.1.0/rc2/toree-src/apache-toree-0.1.0-incubating-source-release.tar.gz.sha

Added: 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz
==
Binary file - no diff available.

Propchange: 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz
--
svn:mime-type = application/octet-stream

Added: 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc
==
--- 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc
 (added)
+++ 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.asc
 Tue Oct 11 18:26:29 2016
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v1
+
+iQIcBAABAgAGBQJX/Sy8AAoJEErQor8BMK2mYrsP/RFg14QankwGPlRhTX2OUzbD
+hGAZOtmYG2zPubvitPOhLe81E0lemFglQyBjZ0/piSlrGyjko6v0gfevVwg+P44H
+Pm7CNCNIDNTHeTeecXfF+58Jl8oLPYDfwsSLaEohWK333NvrTxLBHQbBTshjVcPB
+HJgIw+75wGE9iAdMEnfdfyFlk4vMFhXn9RFye+loFJh80DVeFsV9AYd2OUFHrhV7
+ksEzM7Hr1LA+Bm2D4hoXoT8TamOs9qt5g+KjifJk7km3hKb4EI90zLsgQH6VMLqm
+46cpr7+1CE1qr0w9neKCaVaS+VPsoLYb0RUvhniwPjpPsm3okNDJIDu3p0cmatnM
+KTWAPZhyqAKNmzYkGRW9+bJ9oPq3xEsWIh4IQAF/gT/5SwLYL7TzT31uW5GbCwuF
+5w/A+QeW6d7hyburrttAwf7veKFN/nyZDZPpOvfakujuzK2l5IAt/2odv9ch9i5Z
+nUbmW12snUCIeA8ShjTykmV7YiSqRYsTQtCI36QNmeWWqrB0tNkwy7qlpzfJJdzI
+sRQDHAjm7YlqwdWxPk2dU0MplrYgQDHkM1p29vvIguyBlB5CO1Tolng6Xn/qKWQG
+Gws7eVE+fpXlHjRLeFdrb7hJpbQiHj4LQqxhCKtqq7lkk+hIEbQq5ZhBIB1UqSBF
+2Jah0V7JRLICVbDeGEHt
+=F335
+-END PGP SIGNATURE-

Added: 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5
==
--- 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5
 (added)
+++ 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.md5
 Tue Oct 11 18:26:29 2016
@@ -0,0 +1,2 @@
+/Users/ginobustelo/Dev/spark/toree/dist/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz:
 
+99 EA 7C 16 5D D8 DD 24  77 98 FF 20 4D DE A3 A2

Added: 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
==
--- 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
 (added)
+++ 
dev/incubator/toree/0.1.0/rc2/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz.sha
 Tue Oct 11 18:26:29 2016
@@ -0,0 +1,3 @@
+/Users/ginobustelo/Dev/spark/toree/dist/toree-bin/apache-toree-0.1.0-incubating-binary-release.tar.gz:
 
+A6A51920 613B0C13 BEAA78E8 D9069328 BC1338BF 64886D12 CBE46A4A 556AAE54 
D1DE8D71
+ 3A5F99B0 29544FC2 CB59BC73 96E81508 5C2330A2 4A241D74 48629968

Added: dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz
==
Binary file - no diff available.

Propchange: dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/incubator/toree/0.1.0/rc2/toree-pip/apache-toree-0.1.0.tar.gz.asc
==
--- dev/incubator/toree/0.1.0/rc2/toree-pip

[incubator-toree] Git Push Summary

[lbustelo]

Repository: incubator-toree
Updated Tags:  refs/tags/v0.1.0-rc2 [created] 9a11a3c8b
  refs/tags/v0.1.2 [created] 16919853d