Sorry for the late reply. Have you tried discussing the issue in the kernel gateway project? Don't see anything in your stack trace that points to Toree.
On Thu, May 12, 2016 at 11:47 AM, Sourav Mazumder < [email protected]> wrote: > Hi Gino, > > Any inputs on how to solve this issues ? Show I install a latest one if > this problem is already fixed ? > > Regards, > Sourav > > On Tue, May 10, 2016 at 3:03 PM, Sourav Mazumder < > [email protected]> wrote: > > > Hi Gino, > > > > Thanks for the same. > > > > I was trying to install the jupyter_ketnel_gateway and run the same. > > > > However, when I run the command 'jupyter kernelgateway' I get following > > error - > > > > File "/home/biadmin/.local/bin/jupyter-kernelgateway", line 7, in > <module> > > from kernel_gateway import launch_instance > > File > > > "/home/biadmin/.local/lib/python2.6/site-packages/kernel_gateway/__init__.py", > > line 4, in <module> > > from .gatewayapp import launch_instance > > File > > > "/home/biadmin/.local/lib/python2.6/site-packages/kernel_gateway/gatewayapp.py", > > line 9, in <module> > > import nbformat > > File > > "/home/biadmin/.local/lib/python2.6/site-packages/nbformat/__init__.py", > > line 11, in <module> > > from traitlets.log import get_logger > > File > > "/home/biadmin/.local/lib/python2.6/site-packages/traitlets/__init__.py", > > line 1, in <module> > > from .traitlets import * > > File > > > "/home/biadmin/.local/lib/python2.6/site-packages/traitlets/traitlets.py", > > line 1331 > > return {n: t for (n, t) in cls.class_traits(**metadata).items() > > > > Is there any issue with the current build ? > > > > Regards, > > Sourav > > > > > > > > On Fri, May 6, 2016 at 11:00 AM, Gino Bustelo <[email protected]> wrote: > > > >> http://s32.postimg.org/47f8a1qo5/toree_provisioned.png > >> > >> On Fri, May 6, 2016 at 9:54 AM, Gino Bustelo <[email protected]> wrote: > >> > >> > Ok... I'll try to hang it somewhere on the interwebs and send an url > >> > > >> > On Fri, May 6, 2016 at 9:29 AM, Luciano Resende <[email protected] > > > >> > wrote: > >> > > >> >> I believe the list will remove the image. > >> >> > >> >> On Thursday, May 5, 2016, Sourav Mazumder < > [email protected] > >> > > >> >> wrote: > >> >> > >> >> > Hi Gino, > >> >> > > >> >> > Thanks for the details. > >> >> > > >> >> > But I'm not able to see the image - it is coming as inline image. > >> >> > > >> >> > Could you please send the image once more ? > >> >> > > >> >> > Regards, > >> >> > Sourav > >> >> > > >> >> > On Thu, May 5, 2016 at 12:44 PM, Gino Bustelo <[email protected] > >> >> > <javascript:;>> wrote: > >> >> > > >> >> > > Sourav, > >> >> > > > >> >> > > The solution will look something like this picture > >> >> > > > >> >> > > [image: Inline image 1] > >> >> > > > >> >> > > There is no need for a separate Toree client if you are using > >> Jupyter. > >> >> > > Jupyter already knows how to talk to Toree. Now... there are > other > >> >> > > solutions that can sit on top of Toree that can expose REST or > web > >> >> > socket, > >> >> > > but are currently meant for custom client solutions. See > >> >> > > https://github.com/jupyter/kernel_gateway. > >> >> > > > >> >> > > Thanks, > >> >> > > Gino > >> >> > > > >> >> > > On Thu, May 5, 2016 at 11:46 AM, Sourav Mazumder < > >> >> > > [email protected] <javascript:;>> wrote: > >> >> > > > >> >> > >> Hi Gino, > >> >> > >> > >> >> > >> Thanks for explaining the scope of Toree. > >> >> > >> > >> >> > >> What I was looking for is a solution where Toree can play the > role > >> >> of a > >> >> > >> facade between the client application (in this case the > notebook) > >> and > >> >> > the > >> >> > >> underlying Spark cluster. So if the client application submit a > >> >> command > >> >> > it > >> >> > >> can accept it and execute it using underlying spark > infrastructure > >> >> (may > >> >> > be > >> >> > >> stand alone, on mesos, or on YARN) and return back the result. > >> >> > >> > >> >> > >> I someway like the option 2 too as I think it is in the similar > >> line > >> >> of > >> >> > my > >> >> > >> requirement. However, not sure whether I have got it fully. > >> >> > >> > >> >> > >> What essentially I'm looking for is a solution where the Jupyter > >> >> would > >> >> > be > >> >> > >> running on individual data scientists' laptop. The Jupyter will > >> issue > >> >> > the > >> >> > >> command from the laptop and the Toree client will accept it and > >> send > >> >> it > >> >> > to > >> >> > >> the Toree server running on the Spark Cluster. Toree server will > >> run > >> >> > that > >> >> > >> on Spark and return the results back. > >> >> > >> > >> >> > >> To achieve this requirement using option 2, can one potentially > >> >> change > >> >> > >> Jupyter (or add an extension) which can send the request to > Toree > >> >> > running > >> >> > >> on the provision layer over Zero MQ (or any other protocol like > >> >> REST) ? > >> >> > >> > >> >> > >> Regards, > >> >> > >> Sourav > >> >> > >> > >> >> > >> On Thu, May 5, 2016 at 6:47 AM, Gino Bustelo <[email protected] > >> >> > <javascript:;>> wrote: > >> >> > >> > >> >> > >> > >>>>>>>>>>>>>>>>>>> > >> >> > >> > Hi Gino, > >> >> > >> > > >> >> > >> > It does not solve the problem of running a Spark job (on > Yarn) > >> >> > remotely > >> >> > >> > from a Jupyter notebook which is running on say in a > laptop/some > >> >> > >> machine. > >> >> > >> > > >> >> > >> > The issue is in yarn-client mode the laptop needs to get > access > >> to > >> >> all > >> >> > >> the > >> >> > >> > slave nodes where the executors would be running. In a typical > >> >> > security > >> >> > >> > scenario of an organization the slave nodes are behind > firewall > >> and > >> >> > >> cannot > >> >> > >> > be accessed from any random machine outside. > >> >> > >> > > >> >> > >> > Regards, > >> >> > >> > Sourav > >> >> > >> > >>>>>>>>>>>>>>>>>>> > >> >> > >> > > >> >> > >> > > >> >> > >> > Sourav, I'm very much aware about the network implication of > >> Spark > >> >> > (not > >> >> > >> > exclusive to YARN). The typical way that I've seen this > problem > >> >> solved > >> >> > >> is: > >> >> > >> > > >> >> > >> > 1. You manages/host Jupyter in a privilege network space that > >> can > >> >> have > >> >> > >> > access to the Spark cluster. This involves no code changes on > >> >> either > >> >> > >> > Jupyter or Toree, but has the added cost for the service > >> provider > >> >> of > >> >> > >> > managing this frontend tool > >> >> > >> > > >> >> > >> > 2. You create a provisioner layer in a privilege network space > >> to > >> >> > manage > >> >> > >> > Kernels (Toree) and modify Jupyter through extensions to > >> understand > >> >> > how > >> >> > >> to > >> >> > >> > communicate with that provisioner layer. The pro of this is > that > >> >> you > >> >> > >> don't > >> >> > >> > have to manage the Notebooks, but the service provider still > >> need > >> >> to > >> >> > >> build > >> >> > >> > that provisioning layer and proxy the Kernels communication > >> >> channels. > >> >> > >> > > >> >> > >> > My preference is for #2. I think that frontend tools do not > >> need to > >> >> > live > >> >> > >> > close to Spark, but processes like Toree should be as close to > >> the > >> >> > >> compute > >> >> > >> > cluster as possible. > >> >> > >> > > >> >> > >> > Toree's scope is to be a Spark Driver program that allows > >> >> "interactive > >> >> > >> > computing". It is not it's scope to provide a full fledge > >> >> > >> > provisioning/hosting solution to access Spark. That is left to > >> the > >> >> > >> > implementers of Spark offerings to select the best way to > manage > >> >> Toree > >> >> > >> > kernels (i.e. Yarn, Mesos, Docker, etc...). > >> >> > >> > > >> >> > >> > Thanks, > >> >> > >> > Gino > >> >> > >> > > >> >> > >> > On Sat, Apr 30, 2016 at 9:53 PM, Gino Bustelo < > >> [email protected] > >> >> > <javascript:;>> > >> >> > >> wrote: > >> >> > >> > > >> >> > >> > > This is not possible without extending Jupyter. By default, > >> >> Jupyter > >> >> > >> start > >> >> > >> > > kernels as local processes. To be able to launch remote > >> kernels > >> >> you > >> >> > >> need > >> >> > >> > to > >> >> > >> > > provide an extension to the KernelManager and have some sort > >> of > >> >> > kernel > >> >> > >> > > provisioner to then manage the remote kernels. It is not > >> >> something > >> >> > >> hard > >> >> > >> > to > >> >> > >> > > do, but there is really nothing out there that I know of > that > >> you > >> >> > can > >> >> > >> use > >> >> > >> > > out of the box. > >> >> > >> > > > >> >> > >> > > Gino B. > >> >> > >> > > > >> >> > >> > > > On Apr 30, 2016, at 6:25 PM, Sourav Mazumder < > >> >> > >> > > [email protected] <javascript:;>> wrote: > >> >> > >> > > > > >> >> > >> > > > Hi, > >> >> > >> > > > > >> >> > >> > > > > >> >> > >> > > > is there any documentation which can be user to configure > a > >> >> local > >> >> > >> > Jupyter > >> >> > >> > > > process to talk remotely to a remote Apache Toree server ? > >> >> > >> > > > > >> >> > >> > > > Regards, > >> >> > >> > > > Sourav > >> >> > >> > > > >> >> > >> > > >> >> > >> > >> >> > > > >> >> > > > >> >> > > >> >> > >> >> > >> >> -- > >> >> Sent from my Mobile device > >> >> > >> > > >> > > >> > > > > >
