Ngone51 commented on pull request #34461:
URL: https://github.com/apache/spark/pull/34461#issuecomment-963748246
> IIUC, for a retried map stage, it would go with below code path due to
dep.shuffleMergeEnabled=false:
> ...
That's what we have today. But I'm thinking the current behavior may be
wrong. (Your fix based on the current behavior is correct). For example,
assuming we have stage 1 and stage 2, and stage 2 hits fetch failure and leads
to stage 1 rerun.
So, with the current behavior, when stage 2 reruns, it'd have
`stage1.dep.shuffleMergeEnabled=false`, and stage 2 can't fetch merge data of
stage 1. However, I think we do can fetch the merge data, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
