[ 
https://issues.apache.org/jira/browse/LIVY-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-471:
-----------------------------
    Description: 
Already post in mail list.

In our current API design to create interactive / batch session, we assume end 
user should upload jars, pyFiles and related dependencies to HDFS before 
creating the session, and we use one POST request to create session. But 
usually end user may not have the permission to access the HDFS in their 
submission machine, so it makes them hard to create new sessions. So the 
requirement here is that if Livy could offer APIs to upload resources during 
session creation. One implementation is proposed in

[https://github.com/apache/incubator-livy/pull/91.]

This add a field in session creation request to delay the session creation, 
then adding a bunch of APIs to support resource upload, finally adding an API 
to start creating the session. This seems a feasible solution, but also a 
little hack to support such scenario. So I was thinking if we could a set of 
new APIs to support such scenarios, rather than hack the existing APIs.

To borrow the concept from yarn application submission, we can have 3 APIs to 
create session.
 * requesting a new session id from Livy Server.
 * uploading resources associate with this session id.
 * submitting request to create session.

This is similar to YARN's process to submit application, and we can bump the 
supported API version for newly added APIs.

 

  was:
Already post in mail list.

In our current API design to create interactive / batch session, we assume end 
user should upload jars, pyFiles and related dependencies to HDFS before 
creating the session, and we use one POST request to create session. But 
usually end user may not have the permission to access the HDFS in their 
submission machine, so it makes them hard to create new sessions. So the 
requirement here is that if Livy could offer APIs to upload resources during 
session creation. One implementation is proposed in

[https://github.com/apache/incubator-livy/pull/91.]

This add a field in session creation request to delay the session creation, 
then adding a bunch of APIs to support resource upload, finally adding an API 
to start creating the session. This seems a feasible solution, but also a 
little hack to support such scenario. So I was thinking if we could a set of 
new APIs to support such scenarios, rather than hack the existing APIs.

To borrow the concept from yarn application submission, we can have 3 APIs to 
create session.
 * requesting a new session id from Livy Server.
 * uploading resources associate with this session id.
 * submitting request to create session. This is similar to YARN's process to 
submit application, and we can bump the supported API version for newly added 
APIs.

 


> New session creation API set to support resource uploading
> ----------------------------------------------------------
>
>                 Key: LIVY-471
>                 URL: https://issues.apache.org/jira/browse/LIVY-471
>             Project: Livy
>          Issue Type: Improvement
>          Components: Server
>    Affects Versions: 0.5.0
>            Reporter: Saisai Shao
>            Priority: Major
>
> Already post in mail list.
> In our current API design to create interactive / batch session, we assume 
> end user should upload jars, pyFiles and related dependencies to HDFS before 
> creating the session, and we use one POST request to create session. But 
> usually end user may not have the permission to access the HDFS in their 
> submission machine, so it makes them hard to create new sessions. So the 
> requirement here is that if Livy could offer APIs to upload resources during 
> session creation. One implementation is proposed in
> [https://github.com/apache/incubator-livy/pull/91.]
> This add a field in session creation request to delay the session creation, 
> then adding a bunch of APIs to support resource upload, finally adding an API 
> to start creating the session. This seems a feasible solution, but also a 
> little hack to support such scenario. So I was thinking if we could a set of 
> new APIs to support such scenarios, rather than hack the existing APIs.
> To borrow the concept from yarn application submission, we can have 3 APIs to 
> create session.
>  * requesting a new session id from Livy Server.
>  * uploading resources associate with this session id.
>  * submitting request to create session.
> This is similar to YARN's process to submit application, and we can bump the 
> supported API version for newly added APIs.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to