Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/2127#issuecomment-53520416
That code looks like it's just setting the output locations for the map
stage... what about the following case:
(1) map stage runs
(2) reduce stage starts
(3) reduce task fails because map output A is lost
(4) map stage is restarted , with a single task for output A
(5) scheduler gets another message that a second reduce task failed because
output B was missing.
(6) map stage finishes, and new reduce stage is started
(7) when the reduce stage tries to get the output locations, won't it get
an exception because there's no location for output B?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
