[jira] [Resolved] (LIVY-533) Spark jobs submitted via programmatic API cannot always be canceled
[ https://issues.apache.org/jira/browse/LIVY-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved LIVY-533. - Resolution: Fixed Fix Version/s: 0.6.0 Issue resolved by pull request 128 [https://github.com/apache/incubator-livy/pull/128] > Spark jobs submitted via programmatic API cannot always be canceled > - > > Key: LIVY-533 > URL: https://issues.apache.org/jira/browse/LIVY-533 > Project: Livy > Issue Type: Bug > Components: RSC >Affects Versions: 0.5.0 >Reporter: Björn Lohrmann >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Running stages of Spark jobs submitted via Livy' programmatic API cannot > (always) be successfully cancelled. > The current implementation of .JobWrapper.cancel() interrupts the worker > thread on the Spark driver (via Future.cancel(true)): > [https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/rsc/src/main/java/org/apache/livy/rsc/driver/JobWrapper.java#L84] > This does not always cancel all activity in Spark, e.g. long-running stages > may remain unaffected. > The Spark-way of cancelling jobs seems to be via > SparkContext.setJobGroup()/cancelJobGroup(), which is also being used in > Livy's REPL Session: > [https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/repl/src/main/scala/org/apache/livy/repl/Session.scala#L164] > I have opened a PR that invokes setJobGroup()/cancelJobGroup() in addition to > interrupting the worker thread running on the driver: > [https://github.com/apache/incubator-livy/pull/128] > > It would be great if the fix could make it into the 0.6 release. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (LIVY-538) Add integration tests for the thrift server
[ https://issues.apache.org/jira/browse/LIVY-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido reassigned LIVY-538: Assignee: Marco Gaido > Add integration tests for the thrift server > --- > > Key: LIVY-538 > URL: https://issues.apache.org/jira/browse/LIVY-538 > Project: Livy > Issue Type: Sub-task > Components: Server, Tests >Affects Versions: 0.6.0 >Reporter: Marcelo Vanzin >Assignee: Marco Gaido >Priority: Major > > Currently there are no integration tests for the thrift server part. The UTs > are run on a different environment than ITs, which are more similar to a real > cluster, and having ITs would allow us to catch issues in the code that are > masked in UTs, such as serialization issues or referencing unavailable > classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LIVY-538) Add integration tests for the thrift server
[ https://issues.apache.org/jira/browse/LIVY-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707357#comment-16707357 ] Marco Gaido commented on LIVY-538: -- Created PR: https://github.com/apache/incubator-livy/pull/131. > Add integration tests for the thrift server > --- > > Key: LIVY-538 > URL: https://issues.apache.org/jira/browse/LIVY-538 > Project: Livy > Issue Type: Sub-task > Components: Server, Tests >Affects Versions: 0.6.0 >Reporter: Marcelo Vanzin >Assignee: Marco Gaido >Priority: Major > > Currently there are no integration tests for the thrift server part. The UTs > are run on a different environment than ITs, which are more similar to a real > cluster, and having ITs would allow us to catch issues in the code that are > masked in UTs, such as serialization issues or referencing unavailable > classes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
