GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/10969
[SPARK-13053] Unignore tests in InternalAccumulatorSuite
These were ignored because they are incorrectly written; they don't
actually trigger stage retries, which is what the tests are testing. These
tests are now rewritten to induce stage retries through fetch failures.
Note: there were 2 tests before and now there's only 1. What happened? It
turns out that the case where we only resubmit a subset of of the original
missing partitions is very difficult to simulate in tests without potentially
introducing flakiness. This is because the `DAGScheduler` removes all map
outputs associated with a given executor when this happens, and we will need
multiple executors to trigger this case, and sometimes the scheduler still
removes map outputs from all executors.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark unignore-accum-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10969.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10969
----
commit 147f062699fafbb814bed6c48abc663b3057d925
Author: Andrew Or <[email protected]>
Date: 2016-01-28T00:39:08Z
Rewrite test to actually trigger stage retries
This commit actually removes one of the tests, which test stage
retries when only a subset of the original partitions are
submitted. This scenario is difficult to simulate in tests
without introducing flakiness.
commit 708f6bc3cd9d7b563d282644b3ac2172e48e6edb
Author: Andrew Or <[email protected]>
Date: 2016-01-28T00:50:56Z
Fix potential race condition in job completion callback
There's a chance that we don't actually wait until we finish
calling the callback since currently we don't wait for it to
happen. This commit adds a way to do that.
commit 66ab94f7b244133f5ea3edbfa0a50e28494ddb50
Author: Andrew Or <[email protected]>
Date: 2016-01-28T00:53:39Z
Merge branch 'master' of github.com:apache/spark into unignore-accum-test
commit b0a1f511eb550484ce60b3f14755c4163e93feaf
Author: Andrew Or <[email protected]>
Date: 2016-01-28T18:43:49Z
Merge branch 'master' of github.com:apache/spark into unignore-accum-test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
