[jira] [Commented] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932134#comment-16932134 ] Naman Mishra commented on LIVY-666: --- [~jerryshao] Thanks for reviewing the plan. As mentioned above, statement related APIs will need to be slightly changed as follows. Will keep the APIs backward compatible. * Statement creation (POST /statements)[Must have]: Optional parameter "interpreterGroup" to specify the context of execution of this statement. If unspecified, we can default to a "default" interpreter group. Backward compatibility is maintained with the default context. * Statement retrieval (GET /statements or /statement/)[Nice to have]: Additional attribute "interpreterGroup" in response to identify the execution context of statement. This should be backward compatible as well. * Listing of interpreter groups [Nice to have]: Implement new API to list all existing interpreter groups under a session to help with debugging purposes. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. > Similar ask has been put in LIVY-325 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (LIVY-657) Travis failed on should not create sessions with duplicate names
[ https://issues.apache.org/jira/browse/LIVY-657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-657. -- Fix Version/s: 0.7.0 Assignee: runzhiwang Resolution: Fixed Issue resolved by pull request 225 https://github.com/apache/incubator-livy/pull/225 > Travis failed on should not create sessions with duplicate names > > > Key: LIVY-657 > URL: https://issues.apache.org/jira/browse/LIVY-657 > Project: Livy > Issue Type: Bug > Components: Tests >Affects Versions: 0.6.0 >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Fix For: 0.7.0 > > Time Spent: 40m > Remaining Estimate: 0h > > should not create sessions with duplicate names *** FAILED *** (17 > milliseconds) > session2.stopped was false (SessionManagerSpec.scala:96) > > please reference to https://travis-ci.org/apache/incubator-livy/jobs/579604782 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931976#comment-16931976 ] Saisai Shao commented on LIVY-666: -- I think it looks good to me. With the above, I think the REST APIs should also be updated, especially statements related ones, would you please propose the changes here, mainly about the compatibility. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. > Similar ask has been put in LIVY-325 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected
[ https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931300#comment-16931300 ] Yiheng Wang commented on LIVY-664: -- session stop code: [https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L103] > Spark application still running when Livy session creating was rejected > > > Key: LIVY-664 > URL: https://issues.apache.org/jira/browse/LIVY-664 > Project: Livy > Issue Type: Bug >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: image-2019-09-08-20-38-50-195.png, > image-2019-09-08-20-39-18-569.png > > Time Spent: 10m > Remaining Estimate: 0h > > Steps for reproduce: > 1. Create a session with some name > 2. Create a second session with the same name > 2.1 Second session creating will be rejected since duplicated session name > is not allowed. > 2.2 Spark application will be submitted but Livy session won't be created > > Result: Spark application was submitted but Livy session is not created > Expected result: Livy session creating rejected AND Spark application should > be finished with failed or killed state or even should not be submitted. > !image-2019-09-08-20-38-50-195.png! > !image-2019-09-08-20-39-18-569.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Issue Comment Deleted] (LIVY-664) Spark application still running when Livy session creating was rejected
[ https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiheng Wang updated LIVY-664: - Comment: was deleted (was: Oh, I check the wrong branch... Sorry for the confusion. ) > Spark application still running when Livy session creating was rejected > > > Key: LIVY-664 > URL: https://issues.apache.org/jira/browse/LIVY-664 > Project: Livy > Issue Type: Bug >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: image-2019-09-08-20-38-50-195.png, > image-2019-09-08-20-39-18-569.png > > Time Spent: 10m > Remaining Estimate: 0h > > Steps for reproduce: > 1. Create a session with some name > 2. Create a second session with the same name > 2.1 Second session creating will be rejected since duplicated session name > is not allowed. > 2.2 Spark application will be submitted but Livy session won't be created > > Result: Spark application was submitted but Livy session is not created > Expected result: Livy session creating rejected AND Spark application should > be finished with failed or killed state or even should not be submitted. > !image-2019-09-08-20-38-50-195.png! > !image-2019-09-08-20-39-18-569.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected
[ https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931261#comment-16931261 ] Yiheng Wang commented on LIVY-664: -- Oh, I check the wrong branch... Sorry for the confusion. > Spark application still running when Livy session creating was rejected > > > Key: LIVY-664 > URL: https://issues.apache.org/jira/browse/LIVY-664 > Project: Livy > Issue Type: Bug >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: image-2019-09-08-20-38-50-195.png, > image-2019-09-08-20-39-18-569.png > > Time Spent: 10m > Remaining Estimate: 0h > > Steps for reproduce: > 1. Create a session with some name > 2. Create a second session with the same name > 2.1 Second session creating will be rejected since duplicated session name > is not allowed. > 2.2 Spark application will be submitted but Livy session won't be created > > Result: Spark application was submitted but Livy session is not created > Expected result: Livy session creating rejected AND Spark application should > be finished with failed or killed state or even should not be submitted. > !image-2019-09-08-20-38-50-195.png! > !image-2019-09-08-20-39-18-569.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (LIVY-684) Livy server support zookeeper service discovery
Zhefeng Wang created LIVY-684: - Summary: Livy server support zookeeper service discovery Key: LIVY-684 URL: https://issues.apache.org/jira/browse/LIVY-684 Project: Livy Issue Type: New Feature Reporter: Zhefeng Wang Livy server hasn't support service discovery, which is widely used in highly available. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naman Mishra updated LIVY-666: -- Affects Version/s: (was: 0.7.0) Description: Currently, a session can contain only one interpreter group. In order to support use case of multiple repls with the same spark application multiple interpreters with different variable scoping (something similar to scoped interpreter mode in Zeppelin: [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), I propose to have "named interpreter groups", i.e., multiple interpreter groups in a session all sharing a spark context. The interpreter group can be specified on which the execution is supposed to happen in the execution API. Similar ask has been put in LIVY-325 was:Currently, a session can contain only one interpreter group. In order to support use case of multiple repls with the same spark application multiple interpreters with different variable scoping (something similar to scoped interpreter mode in Zeppelin: [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), I propose to have "named interpreter groups", i.e., multiple interpreter groups in a session all sharing a spark context. The interpreter group can be specified on which the execution is supposed to happen in the execution API. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. > Similar ask has been put in LIVY-325 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931234#comment-16931234 ] Naman Mishra commented on LIVY-666: --- Proposed approach: Currently, a Livy session contains one "interpretergroup" which is a map of (Kind -> Interpreter objects) . In order to have multiple interpreter groups in a session, we can add another field that holds (intpGroupId -> intpGroup object) mapping. Interpreter groups will be created and added to this map lazily as and when a new scope is requested for execution. Effectively, a session will transform like below: !multiple_interpreter_groups.png|width=791,height=335! As can be seen from the above figure, all interpreter groups share a common SparkContext. Execute API (POST /statements) will accept an optional parameter "interpreterGroup" in the request body that can be passed to denote the interpreter group under which the code will be executed. Since the REPLs of one interpreter group are isolated from REPLs of other interpreter groups, variable declarations, imports and other code constructs are scoped to an interpreter group within a Livy session. CC [~jerryshao] [~tc0312] Can you guys please take a look at the proposed feature and approach and provide any feedback/suggestions? I can go ahead with implementation if I have a heads up. > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (LIVY-633) session should not be gc-ed for long running queries
[ https://issues.apache.org/jira/browse/LIVY-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-633. -- Fix Version/s: 0.7.0 Assignee: Yiheng Wang Resolution: Fixed Issue resolved by pull request 224 https://github.com/apache/incubator-livy/pull/224 > session should not be gc-ed for long running queries > > > Key: LIVY-633 > URL: https://issues.apache.org/jira/browse/LIVY-633 > Project: Livy > Issue Type: Bug > Components: Server >Affects Versions: 0.6.0 >Reporter: Liju >Assignee: Yiheng Wang >Priority: Major > Fix For: 0.7.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > If you have set a relatively small session timeout eg 15 mins and query > execution is taking > 15 mins , the session gets gc-ed , which is incorrect > wrt user experience as the user was still active on session and waiting for > result -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (LIVY-666) Support named interpreter groups
[ https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naman Mishra updated LIVY-666: -- Attachment: multiple_interpreter_groups.png > Support named interpreter groups > > > Key: LIVY-666 > URL: https://issues.apache.org/jira/browse/LIVY-666 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 >Reporter: Naman Mishra >Priority: Major > Attachments: multiple_interpreter_groups.png > > > Currently, a session can contain only one interpreter group. In order to > support use case of multiple repls with the same spark application multiple > interpreters with different variable scoping (something similar to scoped > interpreter mode in Zeppelin: > [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode > > |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), > I propose to have "named interpreter groups", i.e., multiple interpreter > groups in a session all sharing a spark context. The interpreter group can be > specified on which the execution is supposed to happen in the execution API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Issue Comment Deleted] (LIVY-661) POST /sessions API - Conf parameters
[ https://issues.apache.org/jira/browse/LIVY-661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated LIVY-661: Comment: was deleted (was: I'm working on it) > POST /sessions API - Conf parameters > > > Key: LIVY-661 > URL: https://issues.apache.org/jira/browse/LIVY-661 > Project: Livy > Issue Type: Bug > Components: API >Reporter: Arun Sethia >Priority: Critical > > The Livy POST /sessions API allows to pass conf (Spark configuration > properties). When we pass spark.driver.extraJavaOptions as conf, It overrides > the cluster default spark.driver.extraJavaOptions, Ideally it should append > with default conf like it is done for other conf values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
