Hello Hitesh, In regards to one Spark Context per kernel: In general, the rule of thumb: One notebook/application -> One Kernel -> One Spark Context
There is no isolation of spark contexts within the kernel. You are, as you said, only able to create one context per kernel. We do have APIs and command line options to create the context if you do not want the one we supply. Typically, the application consuming from the kernel will need to implement access controls to isolate kernels from other users. It is entirely possible to connect multiple notebooks/applications to a kernel and "share" the context. I use the term "share" lightly because the interpretation of code is a serial process. This means users could potentially block one another by running some long running command. Kind Regards, Corey Stubbs On Thu, Feb 4, 2016 at 1:41 PM Hitesh Shah <hit...@apache.org> wrote: > Hello > > As far as I can find from various searches of the old docs on spark-kernel > wiki and issues on JIRA, atleast for jupyter, it seems like each notebook > has its own spark-kernel which in turns wraps its own spark context. I am > curious about how this isolation works considering that folks have > encountered problems when trying to hold multiple spark contexts within the > same jvm/process. Can someone help shed some light on this? > > I also took a look at the zeppelin work-in-progress branch. This however > seems to use a single spark kernel per interpreter ( the zeppelin folks > with their current spark integration also use a single spark context across > all notebooks ) which I believe would imply sharing of the single spark > kernel/spark context across all zeppelin notebooks? Is this assumption > correct? If yes, can anyone suggest how the zeppelin integration would > likely need to change? Would it be possible to say have one spark-kernel > client per notebook ( or re-use them as needed from a pool ) and somehow > have each one talk to a different spark kernel instance? > > thanks > — Hitesh > > > >
