Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/9214#issuecomment-153830804
I talked to Imran offline, and if the assumption that all attempts generate
the same set of files holds, then this should be ok. It feels a little weird,
because if the attempt outputs are non-deterministic, this change would be
adding yet another source of non-determinism (i.e. you could mix outputs of
different stage attempts in the subsequent shuffle), but if people feel the
smaller diff is worth that, then so be it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
