Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16639#discussion_r99950321
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -405,6 +415,13 @@ private[spark] class Executor(
setTaskFinishedAndClearInterruptStatus()
execBackend.statusUpdate(taskId, TaskState.FAILED,
ser.serialize(reason))
+ case t: Throwable if hasFetchFailure =>
--- End diff --
yeah, as this is now, you could eliminate this -- I left it separate for
now just to highlight that we can differentiate two special cases, which we
could handle in a few different ways.
1) FetchFailed is thrown, and the task fails, but its not the outer-most
exception
It seems clear in this case, we should fail the task with a FetchFailure.
But do we also want to log an error or something indicating bad user code?
Kinda minor, but might be a good idea. (Suggested by @mridulm above as well,
I think.)
1a) or the FetchFailed isn't part of the thrown exception at all.
As I mentioned in my response to your other question, I'd like to consider
this exactly the same as (1).
2) FetchFailed is thrown, but totally swallowed so the task succeeds
Should we succeed the task, or fail it? I don't really know how this would
happen. It seems *really* unlikely the user meant to do this. But then again,
maybe the user did? I chose to just log an error but still succeed the task.
(@markhamstra commented about this on the jira as well.)
its pretty easy to change the code for whatever the desired behavior is,
just waiting for a clear decision.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
