Hi Gino, Any inputs on how to solve this issues ? Show I install a latest one if this problem is already fixed ?
Regards, Sourav On Tue, May 10, 2016 at 3:03 PM, Sourav Mazumder < [email protected]> wrote: > Hi Gino, > > Thanks for the same. > > I was trying to install the jupyter_ketnel_gateway and run the same. > > However, when I run the command 'jupyter kernelgateway' I get following > error - > > File "/home/biadmin/.local/bin/jupyter-kernelgateway", line 7, in <module> > from kernel_gateway import launch_instance > File > "/home/biadmin/.local/lib/python2.6/site-packages/kernel_gateway/__init__.py", > line 4, in <module> > from .gatewayapp import launch_instance > File > "/home/biadmin/.local/lib/python2.6/site-packages/kernel_gateway/gatewayapp.py", > line 9, in <module> > import nbformat > File > "/home/biadmin/.local/lib/python2.6/site-packages/nbformat/__init__.py", > line 11, in <module> > from traitlets.log import get_logger > File > "/home/biadmin/.local/lib/python2.6/site-packages/traitlets/__init__.py", > line 1, in <module> > from .traitlets import * > File > "/home/biadmin/.local/lib/python2.6/site-packages/traitlets/traitlets.py", > line 1331 > return {n: t for (n, t) in cls.class_traits(**metadata).items() > > Is there any issue with the current build ? > > Regards, > Sourav > > > > On Fri, May 6, 2016 at 11:00 AM, Gino Bustelo <[email protected]> wrote: > >> http://s32.postimg.org/47f8a1qo5/toree_provisioned.png >> >> On Fri, May 6, 2016 at 9:54 AM, Gino Bustelo <[email protected]> wrote: >> >> > Ok... I'll try to hang it somewhere on the interwebs and send an url >> > >> > On Fri, May 6, 2016 at 9:29 AM, Luciano Resende <[email protected]> >> > wrote: >> > >> >> I believe the list will remove the image. >> >> >> >> On Thursday, May 5, 2016, Sourav Mazumder <[email protected] >> > >> >> wrote: >> >> >> >> > Hi Gino, >> >> > >> >> > Thanks for the details. >> >> > >> >> > But I'm not able to see the image - it is coming as inline image. >> >> > >> >> > Could you please send the image once more ? >> >> > >> >> > Regards, >> >> > Sourav >> >> > >> >> > On Thu, May 5, 2016 at 12:44 PM, Gino Bustelo <[email protected] >> >> > <javascript:;>> wrote: >> >> > >> >> > > Sourav, >> >> > > >> >> > > The solution will look something like this picture >> >> > > >> >> > > [image: Inline image 1] >> >> > > >> >> > > There is no need for a separate Toree client if you are using >> Jupyter. >> >> > > Jupyter already knows how to talk to Toree. Now... there are other >> >> > > solutions that can sit on top of Toree that can expose REST or web >> >> > socket, >> >> > > but are currently meant for custom client solutions. See >> >> > > https://github.com/jupyter/kernel_gateway. >> >> > > >> >> > > Thanks, >> >> > > Gino >> >> > > >> >> > > On Thu, May 5, 2016 at 11:46 AM, Sourav Mazumder < >> >> > > [email protected] <javascript:;>> wrote: >> >> > > >> >> > >> Hi Gino, >> >> > >> >> >> > >> Thanks for explaining the scope of Toree. >> >> > >> >> >> > >> What I was looking for is a solution where Toree can play the role >> >> of a >> >> > >> facade between the client application (in this case the notebook) >> and >> >> > the >> >> > >> underlying Spark cluster. So if the client application submit a >> >> command >> >> > it >> >> > >> can accept it and execute it using underlying spark infrastructure >> >> (may >> >> > be >> >> > >> stand alone, on mesos, or on YARN) and return back the result. >> >> > >> >> >> > >> I someway like the option 2 too as I think it is in the similar >> line >> >> of >> >> > my >> >> > >> requirement. However, not sure whether I have got it fully. >> >> > >> >> >> > >> What essentially I'm looking for is a solution where the Jupyter >> >> would >> >> > be >> >> > >> running on individual data scientists' laptop. The Jupyter will >> issue >> >> > the >> >> > >> command from the laptop and the Toree client will accept it and >> send >> >> it >> >> > to >> >> > >> the Toree server running on the Spark Cluster. Toree server will >> run >> >> > that >> >> > >> on Spark and return the results back. >> >> > >> >> >> > >> To achieve this requirement using option 2, can one potentially >> >> change >> >> > >> Jupyter (or add an extension) which can send the request to Toree >> >> > running >> >> > >> on the provision layer over Zero MQ (or any other protocol like >> >> REST) ? >> >> > >> >> >> > >> Regards, >> >> > >> Sourav >> >> > >> >> >> > >> On Thu, May 5, 2016 at 6:47 AM, Gino Bustelo <[email protected] >> >> > <javascript:;>> wrote: >> >> > >> >> >> > >> > >>>>>>>>>>>>>>>>>>> >> >> > >> > Hi Gino, >> >> > >> > >> >> > >> > It does not solve the problem of running a Spark job (on Yarn) >> >> > remotely >> >> > >> > from a Jupyter notebook which is running on say in a laptop/some >> >> > >> machine. >> >> > >> > >> >> > >> > The issue is in yarn-client mode the laptop needs to get access >> to >> >> all >> >> > >> the >> >> > >> > slave nodes where the executors would be running. In a typical >> >> > security >> >> > >> > scenario of an organization the slave nodes are behind firewall >> and >> >> > >> cannot >> >> > >> > be accessed from any random machine outside. >> >> > >> > >> >> > >> > Regards, >> >> > >> > Sourav >> >> > >> > >>>>>>>>>>>>>>>>>>> >> >> > >> > >> >> > >> > >> >> > >> > Sourav, I'm very much aware about the network implication of >> Spark >> >> > (not >> >> > >> > exclusive to YARN). The typical way that I've seen this problem >> >> solved >> >> > >> is: >> >> > >> > >> >> > >> > 1. You manages/host Jupyter in a privilege network space that >> can >> >> have >> >> > >> > access to the Spark cluster. This involves no code changes on >> >> either >> >> > >> > Jupyter or Toree, but has the added cost for the service >> provider >> >> of >> >> > >> > managing this frontend tool >> >> > >> > >> >> > >> > 2. You create a provisioner layer in a privilege network space >> to >> >> > manage >> >> > >> > Kernels (Toree) and modify Jupyter through extensions to >> understand >> >> > how >> >> > >> to >> >> > >> > communicate with that provisioner layer. The pro of this is that >> >> you >> >> > >> don't >> >> > >> > have to manage the Notebooks, but the service provider still >> need >> >> to >> >> > >> build >> >> > >> > that provisioning layer and proxy the Kernels communication >> >> channels. >> >> > >> > >> >> > >> > My preference is for #2. I think that frontend tools do not >> need to >> >> > live >> >> > >> > close to Spark, but processes like Toree should be as close to >> the >> >> > >> compute >> >> > >> > cluster as possible. >> >> > >> > >> >> > >> > Toree's scope is to be a Spark Driver program that allows >> >> "interactive >> >> > >> > computing". It is not it's scope to provide a full fledge >> >> > >> > provisioning/hosting solution to access Spark. That is left to >> the >> >> > >> > implementers of Spark offerings to select the best way to manage >> >> Toree >> >> > >> > kernels (i.e. Yarn, Mesos, Docker, etc...). >> >> > >> > >> >> > >> > Thanks, >> >> > >> > Gino >> >> > >> > >> >> > >> > On Sat, Apr 30, 2016 at 9:53 PM, Gino Bustelo < >> [email protected] >> >> > <javascript:;>> >> >> > >> wrote: >> >> > >> > >> >> > >> > > This is not possible without extending Jupyter. By default, >> >> Jupyter >> >> > >> start >> >> > >> > > kernels as local processes. To be able to launch remote >> kernels >> >> you >> >> > >> need >> >> > >> > to >> >> > >> > > provide an extension to the KernelManager and have some sort >> of >> >> > kernel >> >> > >> > > provisioner to then manage the remote kernels. It is not >> >> something >> >> > >> hard >> >> > >> > to >> >> > >> > > do, but there is really nothing out there that I know of that >> you >> >> > can >> >> > >> use >> >> > >> > > out of the box. >> >> > >> > > >> >> > >> > > Gino B. >> >> > >> > > >> >> > >> > > > On Apr 30, 2016, at 6:25 PM, Sourav Mazumder < >> >> > >> > > [email protected] <javascript:;>> wrote: >> >> > >> > > > >> >> > >> > > > Hi, >> >> > >> > > > >> >> > >> > > > >> >> > >> > > > is there any documentation which can be user to configure a >> >> local >> >> > >> > Jupyter >> >> > >> > > > process to talk remotely to a remote Apache Toree server ? >> >> > >> > > > >> >> > >> > > > Regards, >> >> > >> > > > Sourav >> >> > >> > > >> >> > >> > >> >> > >> >> >> > > >> >> > > >> >> > >> >> >> >> >> >> -- >> >> Sent from my Mobile device >> >> >> > >> > >> > >
