GitHub user lirui-intel opened a pull request:
https://github.com/apache/spark/pull/12775
[SPARK-14958][Core] Failed task not handled when there's error
deserializing failure reason
## What changes were proposed in this pull request?
TaskResultGetter tries to deserialize the TaskEndReason before handling the
failed task. If an error is thrown during deserialization, the failed task
won't be handled, which leaves the job hanging.
The PR proposes to handle the failed task in a finally block.
## How was this patch tested?
In my case I hit a NoClassDefFoundError and the job hangs. Manually
verified the patch can fix it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lirui-intel/spark SPARK-14958
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12775.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12775
----
commit d302df2a361b248091198472c68c294c87db3483
Author: Rui Li <[email protected]>
Date: 2016-04-29T01:57:13Z
SPARK-14958: fix
commit 3dd5dd9642761053b0edd32615e1563813a1162d
Author: Rui Li <[email protected]>
Date: 2016-04-29T05:12:31Z
Revert "SPARK-14958: fix"
This reverts commit d302df2a361b248091198472c68c294c87db3483.
commit ee34cd2d67a98ef48b0453d6c0a77b88c9db12fb
Author: Rui Li <[email protected]>
Date: 2016-04-29T05:15:25Z
handle in finally
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
