Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4066#issuecomment-73625517
I'm going to try to pick this up now to finish a few remaining changes,
mostly variable-renaming to avoid re-introducing the confusing "splits"
terminology.
I want to quickly draw attention to a comment up-thread, though:
https://github.com/apache/spark/pull/4066#discussion_r24315166. In a nutshell,
we might want to enable this even if speculation is turned off because it's
possible for concurrent copies of a task to occur if fetch failures trigger
re-execution of earlier stages. This might be a rarer occurrence than
speculation races, though.
I also still have to test this on EC2 and need to run some performance
benchmarks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
