GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/10759
[SPARK-12174] Speed up BlockManagerSuite getRemoteBytes() test
This patch significantly speeds up the BlockManagerSuite's "SPARK-9591:
getRemoteBytes from another location when Exception throw" test, reducing the
test time from 45s to ~250ms. The key change was to set
`spark.shuffle.io.maxRetries` to 0 (the code previously set
`spark.network.timeout` to `2s`, but this didn't make a difference because the
slowdown was not due to this timeout).
Along the way, I also cleaned up the way that we handle SparkConf in
BlockManagerSuite: previously, each test would mutate a shared SparkConf
instance, while now each test gets a fresh SparkConf.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-12174
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10759.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10759
----
commit 0d0114c739d42d38bbf758f1c2578b46b02e94e8
Author: Josh Rosen <[email protected]>
Date: 2016-01-14T21:54:06Z
Use new conf in each test case.
commit 5a4916882a004e721d001f7cb85ff3c4a483b1a2
Author: Josh Rosen <[email protected]>
Date: 2016-01-14T21:57:20Z
Remove conf reset code now that it's managed in afterEach().
commit 1c67e99c369811b111a1c1d8aa6c9f35b76deece
Author: Josh Rosen <[email protected]>
Date: 2016-01-14T21:59:19Z
Remove some unnecessary vars in test case
commit af7a117b38ab803113052b269ff958b7a7d2bab2
Author: Josh Rosen <[email protected]>
Date: 2016-01-14T22:04:44Z
Change IO retries instead of network timeout
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
