Hey Pranav,
Did you make any progress on this?
--
Rohit
On Sunday, August 16, 2015, moon soo Lee <m...@apache.org
<mailto:m...@apache.org>> wrote:
Pranav, proposal looks awesome!
I have a question and feedback,
You said you tested 1,2 and 3. To create SparkIMain per notebook,
you need information of notebook id. Did you get it from
InterpreterContext?
Then how did you handle destroying of SparkIMain (when notebook is
deleting)?
As far as i know, interpreter not able to get information of
notebook deletion.
>> 4. Build a queue inside interpreter to allow only one paragraph
execution
>> at a time per notebook.
One downside of this approach is, GUI will display RUNNING instead
of PENDING for jobs inside of queue in interpreter.
Best,
moon
On Sun, Aug 16, 2015 at 12:55 AM IT CTO <goi....@gmail.com
<javascript:_e(%7B%7D,'cvml','goi....@gmail.com');>> wrote:
+1 for "to re-factor the Zeppelin architecture so that it can
handle multi-tenancy easily"
On Sun, Aug 16, 2015 at 9:47 AM DuyHai Doan
<doanduy...@gmail.com
<javascript:_e(%7B%7D,'cvml','doanduy...@gmail.com');>> wrote:
Agree with Joel, we may think to re-factor the Zeppelin
architecture so that it can handle multi-tenancy easily.
The technical solution proposed by Pranav is great but it
only applies to Spark. Right now, each interpreter has to
manage multi-tenancy its own way. Ultimately Zeppelin can
propose a multi-tenancy contract/info (like UserContext,
similar to InterpreterContext) so that each interpreter
can choose to use or not.
On Sun, Aug 16, 2015 at 3:09 AM, Joel Zambrano
<djo...@gmail.com
<javascript:_e(%7B%7D,'cvml','djo...@gmail.com');>> wrote:
I think while the idea of running multiple notes
simultaneously is great. It is really dancing around
the lack of true multi user support in Zeppelin. While
the proposed solution would work if the applications
resources are those of the whole cluster, if the app
is limited (say they are 8 cores of 16, with some
distribution in memory) then potentially your note can
hog all the resources and the scheduler will have to
throttle all other executions leaving you exactly
where you are now.
While I think the solution is a good one, maybe this
question makes us think in adding true multiuser support.
Where we isolate resources (cluster and the notebooks
themselves), have separate login/identity and (I don't
know if it's possible) share the same context.
Thanks,
Joel
> On Aug 15, 2015, at 1:58 PM, Rohit Agarwal
<mindpri...@gmail.com
<javascript:_e(%7B%7D,'cvml','mindpri...@gmail.com');>> wrote:
>
> If the problem is that multiple users have to wait
for each other while
> using Zeppelin, the solution already exists: they
can create a new
> interpreter by going to the interpreter page and
attach it to their
> notebook - then they don't have to wait for others
to submit their job.
>
> But I agree, having paragraphs from one note wait
for paragraphs from other
> notes is a confusing default. We can get around that
in two ways:
>
> 1. Create a new interpreter for each note and
attach that interpreter to
> that note. This approach would require the least amount of
code changes but
> is resource heavy and doesn't let you share Spark
Context between different
> notes.
> 2. If we want to share the Spark Context between
different notes, we can
> submit jobs from different notes into different
fairscheduler pools (
>
https://spark.apache.org/docs/1.4.0/job-scheduling.html#scheduling-within-an-application).
> This can be done by submitting jobs from different
notes in different
> threads. This will make sure that jobs from one
note are run sequentially
> but jobs from different notes will be able to run
in parallel.
>
> Neither of these options require any change in the
Spark code.
>
> --
> Thanks & Regards
> Rohit Agarwal
> https://www.linkedin.com/in/rohitagarwal003
>
> On Sat, Aug 15, 2015 at 12:01 PM, Pranav Kumar
Agarwal <praag...@gmail.com
<javascript:_e(%7B%7D,'cvml','praag...@gmail.com');>>
> wrote:
>
>> If someone can share about the idea of sharing
single SparkContext through
>>> multiple SparkILoop safely, it'll be really helpful.
>> Here is a proposal:
>> 1. In Spark code, change SparkIMain.scala to allow
setting the virtual
>> directory. While creating new instances of
SparkIMain per notebook from
>> zeppelin spark interpreter set all the instances of
SparkIMain to the same
>> virtual directory.
>> 2. Start HTTP server on that virtual directory and
set this HTTP server in
>> Spark Context using classserverUri method
>> 3. Scala generated code has a notion of packages.
The default package name
>> is "line$<linenumber>". Package name can be
controlled using System
>> Property scala.repl.name.line. Setting this
property to "notebook id"
>> ensures that code generated by individual instances
of SparkIMain is
>> isolated from other instances of SparkIMain
>> 4. Build a queue inside interpreter to allow only
one paragraph execution
>> at a time per notebook.
>>
>> I have tested 1, 2, and 3 and it seems to provide
isolation across
>> classnames. I'll work towards submitting a formal
patch soon - Is there any
>> Jira already for the same that I can uptake? Also I
need to understand:
>> 1. How does Zeppelin uptake Spark fixes? OR do I
need to first work
>> towards getting Spark changes merged in Apache
Spark github?
>>
>> Any suggestions on comments on the proposal are
highly welcome.
>>
>> Regards,
>> -Pranav.
>>
>>> On 10/08/15 11:36 pm, moon soo Lee wrote:
>>>
>>> Hi piyush,
>>>
>>> Separate instance of SparkILoop SparkIMain for
each notebook while
>>> sharing the SparkContext sounds great.
>>>
>>> Actually, i tried to do it, found problem that
multiple SparkILoop could
>>> generates the same class name, and spark executor
confuses classname since
>>> they're reading classes from single SparkContext.
>>>
>>> If someone can share about the idea of sharing
single SparkContext
>>> through multiple SparkILoop safely, it'll be
really helpful.
>>>
>>> Thanks,
>>> moon
>>>
>>>
>>> On Mon, Aug 10, 2015 at 1:21 AM Piyush Mukati
(Data Platform) <
>>> piyush.muk...@flipkart.com
<javascript:_e(%7B%7D,'cvml','piyush.muk...@flipkart.com');>
<mailto:piyush.muk...@flipkart.com
<javascript:_e(%7B%7D,'cvml','piyush.muk...@flipkart.com');>>>
wrote:
>>>
>>> Hi Moon,
>>> Any suggestion on it, have to wait lot when
multiple people working
>>> with spark.
>>> Can we create separate instance of
SparkILoop SparkIMain and
>>> printstrems for each notebook while sharing
theSparkContext
>>> ZeppelinContext SQLContext and DependencyResolver
and then use parallel
>>> scheduler ?
>>> thanks
>>>
>>> -piyush
>>>
>>> Hi Moon,
>>>
>>> How about tracking dedicated SparkContext for a
notebook in Spark's
>>> remote interpreter - this will allow multiple
users to run their spark
>>> paragraphs in parallel. Also, within a notebook
only one paragraph is
>>> executed at a time.
>>>
>>> Regards,
>>> -Pranav.
>>>
>>>
>>>> On 15/07/15 7:15 pm, moon soo Lee wrote:
>>>> Hi,
>>>>
>>>> Thanks for asking question.
>>>>
>>>> The reason is simply because of it is running
code statements. The
>>>> statements can have order and dependency. Imagine
i have two
>>> paragraphs
>>>>
>>>> %spark
>>>> val a = 1
>>>>
>>>> %spark
>>>> print(a)
>>>>
>>>> If they're not running one by one, that means
they possibly runs in
>>>> random order and the output will be always
different. Either '1' or
>>>> 'val a can not found'.
>>>>
>>>> This is the reason why. But if there are nice
idea to handle this
>>>> problem i agree using parallel scheduler would
help a lot.
>>>>
>>>> Thanks,
>>>> moon
>>>> On 2015년 7월 14일 (화) at 오후 7:59 linxi zeng
>>>> <linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>
<mailto:linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>>
>>> <mailto:linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>
<mailto:linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>>>>
>>> wrote:
>>>>
>>>> any one who have the same question with me? or
this is not a
>>> question?
>>>>
>>>> 2015-07-14 11:47 GMT+08:00 linxi zeng
<linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>
>>> <mailto:linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>>
>>>> <mailto:linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>
<mailto:
>>> linxizeng0...@gmail.com
<javascript:_e(%7B%7D,'cvml','linxizeng0...@gmail.com');>>>>:
>>>>
>>>> hi, Moon:
>>>> I notice that the getScheduler function
in the
>>>> SparkInterpreter.java return a FIFOScheduler
which makes the
>>>> spark interpreter run spark job one by
one. It's not a good
>>>> experience when couple of users do some
work on zeppelin at
>>>> the same time, because they have to wait
for each other.
>>>> And at the same time, SparkSqlInterpreter
can chose what
>>>> scheduler to use by
"zeppelin.spark.concurrentSQL".
>>>> My question is, what kind of consideration
do you based on
>>> to
>>>> make such a decision?
>>>
>>>
>>>
>>>
>>>
------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> This email and any files transmitted with it
are confidential and
>>> intended solely for the use of the individual
or entity to whom
>>> they are addressed. If you have received this
email in error
>>> please notify the system manager. This message
contains
>>> confidential information and is intended only
for the individual
>>> named. If you are not the named addressee you
should not
>>> disseminate, distribute or copy this e-mail.
Please notify the
>>> sender immediately by e-mail if you have
received this e-mail by
>>> mistake and delete this e-mail from your
system. If you are not
>>> the intended recipient you are notified that
disclosing, copying,
>>> distributing or taking any action in reliance
on the contents of
>>> this information is strictly prohibited.
Although Flipkart has
>>> taken reasonable precautions to ensure no
viruses are present in
>>> this email, the company cannot accept
responsibility for any loss
>>> or damage arising from the use of this email or
attachments
>>
--
Sent from a mobile device. Excuse my thumbs.
