It would also be great to see a page somewhere under https://zeppelin.incubator.apache.org/docs/index.html which clearly explains the relationship between Spark Contexts, notebooks, paragraphs and interpreters. As a new user to Zeppelin I’ve found this to be quite confusing! For example, I’ve just noticed it seems you can’t mix interpreters in the same paragraph, eg., do %dep and later on %spark, or %spark and then %angular….
From: tog [mailto:guillaume.all...@gmail.com] Sent: 25 September 2015 06:54 To: users@zeppelin.incubator.apache.org Subject: Re: Zeppelin Anatomy Hi Alex Yep, i think the multitenancy set-up has raised numerous questions recently. It might be interesting to dedicate a web page to your container approach in the doc. Thanks Guillaume On Friday, 25 September 2015, Alex <abezzu...@nflabs.com<mailto:abezzu...@nflabs.com>> wrote: Hi, Spark context is bounded to Spark interpreter instance, each running in a separate process. All notes that share the same interpreter - are sharing the context too (among other things) You can archive the desired behaviour in multiuser environment right now, I.e by creating a separate spark interpreter for each user in case all users share access to the same Zeppelin instance. Another approach that we use for our customers is to host a separate Zeppelin instance in container, one per-user, and have a balancing reverse-proxy in front of it. I can share more details on this multi-tenancy setup, if enough people from community are interested in it. Hole this helps! -- Kind regards, Alexander On 25 Sep 2015, at 00:54, Yian Shang <yian.sh...@gmail.com<javascript:_e(%7B%7D,'cvml','yian.sh...@gmail.com');>> wrote: Are there any plans to change this so that there will be a separate Spark context per Notebook? In a multi-user environment, it is hard to deal with the accidental overwriting of user variables. On Thu, Sep 24, 2015 at 7:19 AM, Rick Moritz <rah...@gmail.com<javascript:_e(%7B%7D,'cvml','rah...@gmail.com');>> wrote: Different instances of Zeppelin (even under the same user) are indeed separate, which is (currently) the only way to get any kind of independence into notebooks. In comparison, spark-notebook spawns one spark context per Notebook, which is somehwat better design, since concurrent useres of the same application aren't overwriting each other's variables accidentally, and each notebook is indeed "repeatable" and "stand-alone", which is a current deficit of Zeppelin, especially ina multi-user environment. So yes, closing one context in one instance of Zeppelin will not interefere with the other Spark context in the other instance of Zeppelin. On Thu, Sep 24, 2015 at 4:02 PM, Hammad <ham...@flexilogix.com<javascript:_e(%7B%7D,'cvml','ham...@flexilogix.com');>> wrote: Very useful indeed, Rick! If I have two zeppelin instances running as two different users with same Spark Master - I see them as two different applications in Spark Web UI. 1. will they have their own 'context' of execution in this case? If I understand, this would mean that closing a spark context in one user's zeppelin will have no impact on another user's zeppelin environment or its not true? On Thu, Sep 24, 2015 at 4:47 PM, Rick Moritz <rah...@gmail.com<javascript:_e(%7B%7D,'cvml','rah...@gmail.com');>> wrote: 1) Zeppelin uses the spark-shell REPL API. Therefore it behaves similarly to the scala shell. You do not write applications in the shell, in the technical sense, but instead evaluate individual expressions with the goal of interacting with a dataset. You can (manually) export some of the code that you find useful in Zeppelin to applications, for example to provide batch-pre-processing. I recommend you look at demos/descriptions of the interactive shell functionality to get an idea, of what Zeppelin offers over an application. Also: You still have to manage most of your imports ;) 2) There are two benefits: - You can import and export/share notebooks. This means it makes sense to split content. - You also reduce the load of the browser, by splitting heavy visualizations into multiple notebooks. Once you start rendering tens of thousands of points, you start reaching the limits of a browser's capability. Hopefully this helps you get started. On Thu, Sep 24, 2015 at 1:04 PM, Hammad <ham...@flexilogix.com<javascript:_e(%7B%7D,'cvml','ham...@flexilogix.com');>> wrote: Hi mates, I was struggling with anatomy of Zeppelin in context of Spark and could not find anywhere that could answer my questions in mind as below; 1. Usually a scala application structure is; import org.apache.<whatever> obect MyApp{ def main(args: Array[String]){ //something } } whereas, on zeppelin we only write //something. Does it mean that one zeppelin daemon is one application? What if I want to write multiple applications on one zeppelin daemon instance? 2. Related to (1), if same spark context is shared across all notebooks, whats the benefit of having multiple notebooks? I really appreciate if someone may help me understand above two. Thanks, Hmad -- Flexilogix Ph: +92 618090374<tel:%2B92%20618090374> Fax: +92 612011810<tel:%2B92%20612011810> http://www.flexilogix.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.flexilogix.com&d=BQMFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=c1CCNND4PG-Q_V2AJWDWrugZAXQ8Y3EE_f_mAHcpXcs&m=G2GWNaJlswVV3J8GlKQNwwxQ2tO_6M8UKrfQKxQbWDU&s=K59RiRJ1U5lCaT4vlluGKGa96LJIP4AVzIhlD8mynh8&e=> i...@flexilogix.com<javascript:_e(%7B%7D,'cvml','i...@flexilogix.com');> Disclaimer: This transmission (including any attachments) may contain confidential information, privileged material or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. -- PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net<https://urldefense.proofpoint.com/v2/url?u=http-3A__subkeys.pgp.net&d=BQMFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=c1CCNND4PG-Q_V2AJWDWrugZAXQ8Y3EE_f_mAHcpXcs&m=G2GWNaJlswVV3J8GlKQNwwxQ2tO_6M8UKrfQKxQbWDU&s=j6BGASFG1mEdXErqli7_oyuepWe7tNUQevV-t_99TRM&e=>
