Hi Team, In our current API design to create interactive / batch session, we assume end user could upload jars, pyFiles and related dependencies to HDFS before creating the session, and we use one POST request to create session. But usually end user may not have the permission to access the HDFS in their submission machine, so it makes them hard to creating new sessions. So the requirement here is that if Livy could offer APIs to upload resources during session creation.
One implementation is proposed in https://github.com/apache/incubator-livy/pull/91. This add a field in session creation request to delay the session creation, then adding a bunch of APIs to support resource upload, finally adding an API to start creating the session. This seems a feasible solution, but also a little hack to support such scenario. So I was thinking if we could a set of new APIs to support such scenarios, rather than hack the existing APIs. To borrow the concept from yarn application submission, we can have 3 APIs to create session. 1. requesting a new session id from Livy Server. 2. uploading resources associate with this session id. 3. submitting request to create session. This is similar to YARN's process to submit application, and we can bump the supported API version for newly added APIs. What do you think about it? Any suggestion is greatly appreciated! Best regards
