Re: user defined sessionId / URI for Livy sessions

2017-09-12 Thread Bikas Saha 
Resending to both aliases

Hi,

Would be good to understand how this feature would be useful other than 
providing users a handle that they can rely on as a 3rd party.

E.g. something like this may be useful in an active-active livy configuration 
because its not clear how sequential numeric id's would work in that context. 
Perhaps UUID's would also suffice for the active-active setup. Maybe the use 
case for user defined sessions could be to integrate with other software that 
could reliably use the user defined name for its internal operations instead of 
using a livy session id thats not guaranteed to be unique.

Curious as to what motivated choosing sequential numbers for livy sessions?

Bikas



From: Alex Bozarth <ajboz...@us.ibm.com>
Sent: Tuesday, September 12, 2017 2:44 PM
To: u...@livy.incubator.apache.org
Cc: dev@livy.incubator.apache.org
Subject: Re: user defined sessionId / URI for Livy sessions


That's fair, I'll still withhold judgement until I see an impl though.



Alex Bozarth
Software Engineer
Spark Technology Center


E-mail: ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>
GitHub: github.com/ajbozarth<https://github.com/ajbozarth>


505 Howard Street
San Francisco, CA 94105
United States




[Inactive hide details for Marcelo Vanzin ---09/12/2017 02:40:33 PM---BTW take 
my comment there with a grain of salt; at the tim]Marcelo Vanzin ---09/12/2017 
02:40:33 PM---BTW take my comment there with a grain of salt; at the time Livy 
was being targeted at mostly being

From: Marcelo Vanzin <van...@cloudera.com>
To: dev@livy.incubator.apache.org
Cc: "u...@livy.incubator.apache.org" <u...@livy.incubator.apache.org>
Date: 09/12/2017 02:40 PM
Subject: Re: user defined sessionId / URI for Livy sessions





BTW take my comment there with a grain of salt; at the time Livy was being 
targeted at mostly being hidden from users, making such a feature not make much 
sense in Livy itself. But things may have changed since then, especially as 
people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth 
<ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>> wrote:

I would agree with Marcelo's comment the JIRA that this isn't a good feature 
for livy, but I'll take a look at your impl if you open a PR and see if it 
changes my mind.


Alex Bozarth
Software Engineer
Spark Technology Center


E-mail: ajboz...@us.ibm.com<mailto:ajboz...@us.ibm.com>
GitHub: 
github.com/ajbozarth<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=VdRly7JyucmN1BnLWxZGFJY-iemUUfqUWOlQqaGAA54=047UIBZ1bxlXg56NfOnZ2UsJMzTJTSJ34UGFbLo3GbA=>


505 Howard Street
San Francisco, CA 94105
United States




Meisam Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding 
this feature to Livy? I can send a PR

From: Meisam Fathi <meisam.fa...@gmail.com<mailto:meisam.fa...@gmail.com>>
To: "u...@livy.incubator.apache.org<mailto:u...@livy.incubator.apache.org>" 
<u...@livy.incubator.apache.org<mailto:u...@livy.incubator.apache.org>>, 
"dev@livy.incubator.apache.org<mailto:dev@livy.incubator.apache.org>" 
<dev@livy.incubator.apache.org<mailto:dev@livy.incubator.apache.org>>
Date: 09/11/2017 10:23 AM
Subject: Re: user defined sessionId / URI for Livy sessions




+ dev
Is there any interest in adding this feature to Livy? I can send a PR

Ideally, it would be helpful if we could mint a session ID with a PUT
> request, something like PUT /sessions/foobar, where "foobar" is the newly
> created sessionId.
>
> I suggest we make session names unique and nonnumeric values (to guarantee
a session name does not clash with another session name or session ID).

Design doc:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
JIRA ticket: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=


Thanks,
Meisam






--
Marcelo

Re: user defined sessionId / URI for Livy sessions

2017-09-12 Thread Alex Bozarth 

That's fair, I'll still withhold judgement until I see an impl though.


   
 Alex Bozarth   
   
 Software Engineer  
   
 Spark Technology Center
   

   

 

 

 
 E-mail: ajboz...@us.ibm.com
 
 GitHub: github.com/ajbozarth   
 
   505 Howard 
Street 
 San Francisco, CA 
94105 
   United 
States 

 








From:   Marcelo Vanzin <van...@cloudera.com>
To: dev@livy.incubator.apache.org
Cc: "u...@livy.incubator.apache.org"
<u...@livy.incubator.apache.org>
Date:   09/12/2017 02:40 PM
Subject:    Re: user defined sessionId / URI for Livy sessions



BTW take my comment there with a grain of salt; at the time Livy was being
targeted at mostly being hidden from users, making such a feature not make
much sense in Livy itself. But things may have changed since then,
especially as people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:
  I would agree with Marcelo's comment the JIRA that this isn't a good
  feature for livy, but I'll take a look at your impl if you open a PR and
  see if it changes my mind.





   
 Alex Bozarth   
   
 Software Engineer  
   
 Spark Technology Center
   

   








 E-mail: ajboz...@us.ibm.com

 GitHub: github.com/ajbozarth   

  505 Howard 
Street 
San Francisco, CA 
94105 
  United 
States 







  Meisam Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in
  adding this feature to Livy? I can send a PR

  From: Meisam Fathi <meisam.fa...@gmail.com>
  To: "u...@livy.incubator.apache.org" <u...@livy.incubator.apache.org>, "
  dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
  Date: 09/11/2017 10:23 AM
  Subject: Re: user defined sessionId / URI for Livy sessions



  + dev
  Is there any interest in adding this feature to Livy? I can send a PR

  Ideally, it would be helpful if we could mint a session ID with a PUT
  > request, something like PUT /sessions/foobar, where "foobar" is the
  newly
  > created sessionId.
  >
  > I suggest we make session names unique and nonnumeric values (to
  guarantee
  a session name does not clash with another session name or session ID).

  Design doc:
  
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=

  JIRA ticket:
  
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=



  Thanks,
  Meisam








--
Marcelo

Re: user defined sessionId / URI for Livy sessions

2017-09-12 Thread Marcelo Vanzin 
BTW take my comment there with a grain of salt; at the time Livy was being
targeted at mostly being hidden from users, making such a feature not make
much sense in Livy itself. But things may have changed since then,
especially as people started using it more.

On Mon, Sep 11, 2017 at 1:51 PM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see if it changes my mind.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> San Francisco, CA 94105
> United States
>
>
>
> [image: Inactive hide details for Meisam Fathi ---09/11/2017 10:23:49
> AM---+ dev Is there any interest in adding this feature to Livy?]Meisam
> Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding
> this feature to Livy? I can send a PR
>
> From: Meisam Fathi <meisam.fa...@gmail.com>
> To: "u...@livy.incubator.apache.org" <u...@livy.incubator.apache.org>, "
> dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> Date: 09/11/2017 10:23 AM
> Subject: Re: user defined sessionId / URI for Livy sessions
> --
>
>
>
> + dev
> Is there any interest in adding this feature to Livy? I can send a PR
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> > request, something like PUT /sessions/foobar, where "foobar" is the newly
> > created sessionId.
> >
> > I suggest we make session names unique and nonnumeric values (to
> guarantee
> a session name does not clash with another session name or session ID).
>
> Design doc:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-
> 2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=
> xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
> JIRA ticket: https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-
> juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=
>
>
> Thanks,
> Meisam
>
>
>
>


-- 
Marcelo

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Saisai Shao 
I see. So based on this, we should manage a data structure in Livy Server
to keep all the live sessions' name. Also regarding to session recovery, we
should persist this structure to the reliable storage and recover after
restart.

I'm not pretty sure if it is a good feature or not. First because we
usually programmatically manage the session id, so from code level to
manager a session id or a session name there's no much difference; second,
usually it is hard for user to pick a unique name if one Livy Server has
many live sessions, by large chance the name will be conflicted, people
always like short, simple name.

Since I'm so familiar how people really use it, so it is just my two cents.

Thanks
Jerry


On Tue, Sep 12, 2017 at 8:46 AM, Meisam Fathi 
wrote:

> > If we're using session name, how do we guarantee the uniqueness of this
> > name?
> >
>
> If the requested session name already exist, Livy returns an error and does
> not create the session.
>
> Thanks,
> Meisam
>

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Meisam Fathi 
> If we're using session name, how do we guarantee the uniqueness of this
> name?
>

If the requested session name already exist, Livy returns an error and does
not create the session.

Thanks,
Meisam

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Saisai Shao 
If we're using session name, how do we guarantee the uniqueness of this
name?

Thanks
Jerry

On Tue, Sep 12, 2017 at 4:51 AM, Alex Bozarth <ajboz...@us.ibm.com> wrote:

> I would agree with Marcelo's comment the JIRA that this isn't a good
> feature for livy, but I'll take a look at your impl if you open a PR and
> see if it changes my mind.
>
>
> *Alex Bozarth*
> Software Engineer
> Spark Technology Center
> --
> *E-mail:* *ajboz...@us.ibm.com* <ajboz...@us.ibm.com>
> *GitHub: **github.com/ajbozarth* <https://github.com/ajbozarth>
>
>
> 505 Howard Street
> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
> San Francisco, CA 94105
> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
> United States
> <https://maps.google.com/?q=505+Howard+Street+San+Francisco,+CA+94105+United+States=gmail=g>
>
>
>
> [image: Inactive hide details for Meisam Fathi ---09/11/2017 10:23:49
> AM---+ dev Is there any interest in adding this feature to Livy?]Meisam
> Fathi ---09/11/2017 10:23:49 AM---+ dev Is there any interest in adding
> this feature to Livy? I can send a PR
>
> From: Meisam Fathi <meisam.fa...@gmail.com>
> To: "u...@livy.incubator.apache.org" <u...@livy.incubator.apache.org>, "
> dev@livy.incubator.apache.org" <dev@livy.incubator.apache.org>
> Date: 09/11/2017 10:23 AM
> Subject: Re: user defined sessionId / URI for Livy sessions
> --
>
>
>
> + dev
> Is there any interest in adding this feature to Livy? I can send a PR
>
> Ideally, it would be helpful if we could mint a session ID with a PUT
> > request, something like PUT /sessions/foobar, where "foobar" is the newly
> > created sessionId.
> >
> > I suggest we make session names unique and nonnumeric values (to
> guarantee
> a session name does not clash with another session name or session ID).
>
> Design doc:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_meisam_incubator-2Dlivy_wiki_Design-2Ddoc-2Dfor-
> 2DLivy-2D41-3A-2DAccessing-2Dsessions-2Dby-2Dname=
> DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-
> cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-juQkf5Q2FMYtA5kv5sqiMM=
> xTiY52FMWMdTRgCmiNRWe6yEoCchxKNxQrYPEkPupbw=
> JIRA ticket: https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_LIVY-2D41=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=S1_
> S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo=bUJg_csAaA5f2DPiMkjU-
> juQkf5Q2FMYtA5kv5sqiMM=lFed2hYlDA_wUo94RWUAw7N01lSN368P-ABmP_npWrM=
>
>
> Thanks,
> Meisam
>
>
>
>

Re: user defined sessionId / URI for Livy sessions

2017-09-11 Thread Meisam Fathi 
+ dev
Is there any interest in adding this feature to Livy? I can send a PR

Ideally, it would be helpful if we could mint a session ID with a PUT
> request, something like PUT /sessions/foobar, where "foobar" is the newly
> created sessionId.
>
> I suggest we make session names unique and nonnumeric values (to guarantee
a session name does not clash with another session name or session ID).

Design doc:
https://github.com/meisam/incubator-livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name
JIRA ticket: https://issues.apache.org/jira/browse/LIVY-41


Thanks,
Meisam