Hi, The code is available here https://github.com/piyush-mukati/incubator-zeppelin/tree/parallel_scheduler_support_spark
some testing part is left. On Wed, Jan 13, 2016 at 11:47 PM, Dimp Bhat <dimp201...@gmail.com> wrote: > Hi Pranav, > When do you plan to send out the code for running notebooks in parallel ? > > Dimple > > On Tue, Nov 17, 2015 at 3:27 AM, Pranav Kumar Agarwal <praag...@gmail.com> > wrote: > >> Hi Rohit, >> >> We implemented the proposal and are able to run Zeppelin as a hosted >> service inside my organization. Our internal forked version has pluggable >> authentication and type ahead. >> >> I need to get the work ported to the latest and chop out the auth changes >> portion. We'll be submitting it soon. >> >> We'll target to get this out for review by 11/26. >> >> Regards, >> -Pranav. >> >> >> >> On 17/11/15 4:34 am, Rohit Agarwal wrote: >> >> Hey Pranav, >> >> Did you make any progress on this? >> >> -- >> Rohit >> >> On Sunday, August 16, 2015, moon soo Lee <m...@apache.org> wrote: >> >>> Pranav, proposal looks awesome! >>> >>> I have a question and feedback, >>> >>> You said you tested 1,2 and 3. To create SparkIMain per notebook, you >>> need information of notebook id. Did you get it from InterpreterContext? >>> Then how did you handle destroying of SparkIMain (when notebook is >>> deleting)? >>> As far as i know, interpreter not able to get information of notebook >>> deletion. >>> >>> >> 4. Build a queue inside interpreter to allow only one paragraph >>> execution >>> >> at a time per notebook. >>> >>> One downside of this approach is, GUI will display RUNNING instead of >>> PENDING for jobs inside of queue in interpreter. >>> >>> Best, >>> moon >>> >>> On Sun, Aug 16, 2015 at 12:55 AM IT CTO <goi....@gmail.com> wrote: >>> >>>> +1 for "to re-factor the Zeppelin architecture so that it can handle >>>> multi-tenancy easily" >>>> >>>> On Sun, Aug 16, 2015 at 9:47 AM DuyHai Doan <doanduy...@gmail.com> >>>> wrote: >>>> >>>>> Agree with Joel, we may think to re-factor the Zeppelin architecture >>>>> so that it can handle multi-tenancy easily. The technical solution >>>>> proposed >>>>> by Pranav is great but it only applies to Spark. Right now, each >>>>> interpreter has to manage multi-tenancy its own way. Ultimately Zeppelin >>>>> can propose a multi-tenancy contract/info (like UserContext, similar to >>>>> InterpreterContext) so that each interpreter can choose to use or not. >>>>> >>>>> >>>>> On Sun, Aug 16, 2015 at 3:09 AM, Joel Zambrano <djo...@gmail.com> >>>>> wrote: >>>>> >>>>>> I think while the idea of running multiple notes simultaneously is >>>>>> great. It is really dancing around the lack of true multi user support in >>>>>> Zeppelin. While the proposed solution would work if the applications >>>>>> resources are those of the whole cluster, if the app is limited (say they >>>>>> are 8 cores of 16, with some distribution in memory) then potentially >>>>>> your >>>>>> note can hog all the resources and the scheduler will have to throttle >>>>>> all >>>>>> other executions leaving you exactly where you are now. >>>>>> While I think the solution is a good one, maybe this question makes >>>>>> us think in adding true multiuser support. >>>>>> Where we isolate resources (cluster and the notebooks themselves), >>>>>> have separate login/identity and (I don't know if it's possible) share >>>>>> the >>>>>> same context. >>>>>> >>>>>> Thanks, >>>>>> Joel >>>>>> >>>>>> > On Aug 15, 2015, at 1:58 PM, Rohit Agarwal <mindpri...@gmail.com> >>>>>> wrote: >>>>>> > >>>>>> > If the problem is that multiple users have to wait for each other >>>>>> while >>>>>> > using Zeppelin, the solution already exists: they can create a new >>>>>> > interpreter by going to the interpreter page and attach it to their >>>>>> > notebook - then they don't have to wait for others to submit their >>>>>> job. >>>>>> > >>>>>> > But I agree, having paragraphs from one note wait for paragraphs >>>>>> from other >>>>>> > notes is a confusing default. We can get around that in two ways: >>>>>> > >>>>>> > 1. Create a new interpreter for each note and attach that >>>>>> interpreter to >>>>>> > that note. This approach would require the least amount of code >>>>>> changes but >>>>>> > is resource heavy and doesn't let you share Spark Context between >>>>>> different >>>>>> > notes. >>>>>> > 2. If we want to share the Spark Context between different notes, >>>>>> we can >>>>>> > submit jobs from different notes into different fairscheduler >>>>>> pools ( >>>>>> > >>>>>> https://spark.apache.org/docs/1.4.0/job-scheduling.html#scheduling-within-an-application >>>>>> ). >>>>>> > This can be done by submitting jobs from different notes in >>>>>> different >>>>>> > threads. This will make sure that jobs from one note are run >>>>>> sequentially >>>>>> > but jobs from different notes will be able to run in parallel. >>>>>> > >>>>>> > Neither of these options require any change in the Spark code. >>>>>> > >>>>>> > -- >>>>>> > Thanks & Regards >>>>>> > Rohit Agarwal >>>>>> > https://www.linkedin.com/in/rohitagarwal003 >>>>>> > >>>>>> > On Sat, Aug 15, 2015 at 12:01 PM, Pranav Kumar Agarwal < >>>>>> praag...@gmail.com> >>>>>> > wrote: >>>>>> > >>>>>> >> If someone can share about the idea of sharing single SparkContext >>>>>> through >>>>>> >>> multiple SparkILoop safely, it'll be really helpful. >>>>>> >> Here is a proposal: >>>>>> >> 1. In Spark code, change SparkIMain.scala to allow setting the >>>>>> virtual >>>>>> >> directory. While creating new instances of SparkIMain per notebook >>>>>> from >>>>>> >> zeppelin spark interpreter set all the instances of SparkIMain to >>>>>> the same >>>>>> >> virtual directory. >>>>>> >> 2. Start HTTP server on that virtual directory and set this HTTP >>>>>> server in >>>>>> >> Spark Context using classserverUri method >>>>>> >> 3. Scala generated code has a notion of packages. The default >>>>>> package name >>>>>> >> is "line$<linenumber>". Package name can be controlled using System >>>>>> >> Property scala.repl.name.line. Setting this property to "notebook >>>>>> id" >>>>>> >> ensures that code generated by individual instances of SparkIMain >>>>>> is >>>>>> >> isolated from other instances of SparkIMain >>>>>> >> 4. Build a queue inside interpreter to allow only one paragraph >>>>>> execution >>>>>> >> at a time per notebook. >>>>>> >> >>>>>> >> I have tested 1, 2, and 3 and it seems to provide isolation across >>>>>> >> classnames. I'll work towards submitting a formal patch soon - Is >>>>>> there any >>>>>> >> Jira already for the same that I can uptake? Also I need to >>>>>> understand: >>>>>> >> 1. How does Zeppelin uptake Spark fixes? OR do I need to first work >>>>>> >> towards getting Spark changes merged in Apache Spark github? >>>>>> >> >>>>>> >> Any suggestions on comments on the proposal are highly welcome. >>>>>> >> >>>>>> >> Regards, >>>>>> >> -Pranav. >>>>>> >> >>>>>> >>> On 10/08/15 11:36 pm, moon soo Lee wrote: >>>>>> >>> >>>>>> >>> Hi piyush, >>>>>> >>> >>>>>> >>> Separate instance of SparkILoop SparkIMain for each notebook while >>>>>> >>> sharing the SparkContext sounds great. >>>>>> >>> >>>>>> >>> Actually, i tried to do it, found problem that multiple >>>>>> SparkILoop could >>>>>> >>> generates the same class name, and spark executor confuses >>>>>> classname since >>>>>> >>> they're reading classes from single SparkContext. >>>>>> >>> >>>>>> >>> If someone can share about the idea of sharing single SparkContext >>>>>> >>> through multiple SparkILoop safely, it'll be really helpful. >>>>>> >>> >>>>>> >>> Thanks, >>>>>> >>> moon >>>>>> >>> >>>>>> >>> >>>>>> >>> On Mon, Aug 10, 2015 at 1:21 AM Piyush Mukati (Data Platform) < >>>>>> >>> piyush.muk...@flipkart.com <mailto:piyush.muk...@flipkart.com>> >>>>>> wrote: >>>>>> >>> >>>>>> >>> Hi Moon, >>>>>> >>> Any suggestion on it, have to wait lot when multiple people >>>>>> working >>>>>> >>> with spark. >>>>>> >>> Can we create separate instance of SparkILoop SparkIMain and >>>>>> >>> printstrems for each notebook while sharing theSparkContext >>>>>> >>> ZeppelinContext SQLContext and DependencyResolver and then use >>>>>> parallel >>>>>> >>> scheduler ? >>>>>> >>> thanks >>>>>> >>> >>>>>> >>> -piyush >>>>>> >>> >>>>>> >>> Hi Moon, >>>>>> >>> >>>>>> >>> How about tracking dedicated SparkContext for a notebook in >>>>>> Spark's >>>>>> >>> remote interpreter - this will allow multiple users to run >>>>>> their spark >>>>>> >>> paragraphs in parallel. Also, within a notebook only one >>>>>> paragraph is >>>>>> >>> executed at a time. >>>>>> >>> >>>>>> >>> Regards, >>>>>> >>> -Pranav. >>>>>> >>> >>>>>> >>> >>>>>> >>>> On 15/07/15 7:15 pm, moon soo Lee wrote: >>>>>> >>>> Hi, >>>>>> >>>> >>>>>> >>>> Thanks for asking question. >>>>>> >>>> >>>>>> >>>> The reason is simply because of it is running code statements. >>>>>> The >>>>>> >>>> statements can have order and dependency. Imagine i have two >>>>>> >>> paragraphs >>>>>> >>>> >>>>>> >>>> %spark >>>>>> >>>> val a = 1 >>>>>> >>>> >>>>>> >>>> %spark >>>>>> >>>> print(a) >>>>>> >>>> >>>>>> >>>> If they're not running one by one, that means they possibly runs >>>>>> in >>>>>> >>>> random order and the output will be always different. Either '1' >>>>>> or >>>>>> >>>> 'val a can not found'. >>>>>> >>>> >>>>>> >>>> This is the reason why. But if there are nice idea to handle this >>>>>> >>>> problem i agree using parallel scheduler would help a lot. >>>>>> >>>> >>>>>> >>>> Thanks, >>>>>> >>>> moon >>>>>> >>>> On 2015년 7월 14일 (화) at 오후 7:59 linxi zeng >>>>>> >>>> <linxizeng0...@gmail.com <mailto:linxizeng0...@gmail.com> >>>>>> >>> <mailto:linxizeng0...@gmail.com <mailto:linxizeng0...@gmail.com >>>>>> >>> >>>>>> >>> wrote: >>>>>> >>>> >>>>>> >>>> any one who have the same question with me? or this is not a >>>>>> >>> question? >>>>>> >>>> >>>>>> >>>> 2015-07-14 11:47 GMT+08:00 linxi zeng < >>>>>> linxizeng0...@gmail.com >>>>>> >>> <mailto:linxizeng0...@gmail.com> >>>>>> >>>> <mailto:linxizeng0...@gmail.com <mailto: >>>>>> >>> linxizeng0...@gmail.com>>>: >>>>>> >>>> >>>>>> >>>> hi, Moon: >>>>>> >>>> I notice that the getScheduler function in the >>>>>> >>>> SparkInterpreter.java return a FIFOScheduler which makes >>>>>> the >>>>>> >>>> spark interpreter run spark job one by one. It's not a >>>>>> good >>>>>> >>>> experience when couple of users do some work on zeppelin >>>>>> at >>>>>> >>>> the same time, because they have to wait for each other. >>>>>> >>>> And at the same time, SparkSqlInterpreter can chose what >>>>>> >>>> scheduler to use by "zeppelin.spark.concurrentSQL". >>>>>> >>>> My question is, what kind of consideration do you based on >>>>>> >>> to >>>>>> >>>> make such a decision? >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> ------------------------------------------------------------------------------------------------------------------------------------------ >>>>>> >>> >>>>>> >>> This email and any files transmitted with it are confidential >>>>>> and >>>>>> >>> intended solely for the use of the individual or entity to whom >>>>>> >>> they are addressed. If you have received this email in error >>>>>> >>> please notify the system manager. This message contains >>>>>> >>> confidential information and is intended only for the >>>>>> individual >>>>>> >>> named. If you are not the named addressee you should not >>>>>> >>> disseminate, distribute or copy this e-mail. Please notify the >>>>>> >>> sender immediately by e-mail if you have received this e-mail >>>>>> by >>>>>> >>> mistake and delete this e-mail from your system. If you are not >>>>>> >>> the intended recipient you are notified that disclosing, >>>>>> copying, >>>>>> >>> distributing or taking any action in reliance on the contents >>>>>> of >>>>>> >>> this information is strictly prohibited. Although Flipkart has >>>>>> >>> taken reasonable precautions to ensure no viruses are present >>>>>> in >>>>>> >>> this email, the company cannot accept responsibility for any >>>>>> loss >>>>>> >>> or damage arising from the use of this email or attachments >>>>>> >> >>>>>> >>>>> >>>>> >> >> -- >> Sent from a mobile device. Excuse my thumbs. >> >> >> >
