[jira] [Commented] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932134#comment-16932134 ] Naman Mishra commented on LIVY-666: --- [~jerryshao] Thanks for reviewing the plan. As mentioned above, statement related APIs will need to be slightly changed as follows. Will keep the APIs backward compatible. * Statement creation (POST /statements)[Must have]: Optional parameter "interpreterGroup" to specify the context of execution of this statement. If unspecified, we can default to a "default" interpreter group. Backward compatibility is maintained with the default context. * Statement retrieval (GET /statements or /statement/)[Nice to have]: Additional attribute "interpreterGroup" in response to identify the execution context of statement. This should be backward compatible as well. * Listing of interpreter groups [Nice to have]: Implement new API to list all existing interpreter groups under a session to help with debugging purposes. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. > Similar ask has been put in LIVY-325 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931976#comment-16931976 ] Saisai Shao commented on LIVY-666: -- I think it looks good to me. With the above, I think the REST APIs should also be updated, especially statements related ones, would you please propose the changes here, mainly about the compatibility. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. > Similar ask has been put in LIVY-325 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931234#comment-16931234 ] Naman Mishra commented on LIVY-666: --- Proposed approach: Currently, a Livy session contains one "interpretergroup" which is a map of (Kind -> Interpreter objects) . In order to have multiple interpreter groups in a session, we can add another field that holds (intpGroupId -> intpGroup object) mapping. Interpreter groups will be created and added to this map lazily as and when a new scope is requested for execution. Effectively, a session will transform like below: !multiple_interpreter_groups.png|width=791,height=335! As can be seen from the above figure, all interpreter groups share a common SparkContext. Execute API (POST /statements) will accept an optional parameter "interpreterGroup" in the request body that can be passed to denote the interpreter group under which the code will be executed. Since the REPLs of one interpreter group are isolated from REPLs of other interpreter groups, variable declarations, imports and other code constructs are scoped to an interpreter group within a Livy session. CC [~jerryshao] [~tc0312] Can you guys please take a look at the proposed feature and approach and provide any feedback/suggestions? I can go ahead with implementation if I have a heads up. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
