GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/5288
[SPARK-6629] cancelJobGroup() may not work for jobs whose job groups are
inherited from parent threads
When a job is submitted with a job group and that job group is inherited
from a parent thread, there are multiple bugs that may prevent this job from
being cancelable via `SparkContext.cancelJobGroup()`:
- When filtering jobs based on their job group properties, DAGScheduler
calls `get()` instead of `getProperty()`, which does not respect inheritance,
so it will skip over jobs whose job group properties were inherited.
- `Properties` objects are mutable, but we do not make defensive copies /
snapshots, so modifications of the parent thread's job group will cause running
jobs' groups to change; this also breaks cancelation.
Both of these issues are easy to fix: use `getProperty()` and perform
defensive copying.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
localProperties-mutability-race
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5288.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5288
----
commit b376114a333f4a1d3b2153f75b14856c9ba72875
Author: Josh Rosen <[email protected]>
Date: 2015-03-31T08:00:54Z
Fix bug that prevented jobs with inherited job group properties from being
cancelled.
commit 707e4170c97aa38e529c457a6760b15e2e75e02b
Author: Josh Rosen <[email protected]>
Date: 2015-03-31T08:10:58Z
Clone local properties to prevent mutations from breaking job cancellation.
commit 3f7b9e83dc199d8ed772a7822ae4b012c888ca2a
Author: Josh Rosen <[email protected]>
Date: 2015-03-31T08:19:02Z
Add JIRA reference; move clone into DAGScheduler
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
