GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/9124
[SPARK-11078] Ensure spilling tests actually spill
#9084 uncovered that many tests that test spilling don't actually spill.
This is a follow-up patch to fix that to ensure our unit tests actually catch
potential bugs in spilling. The size of this patch is inflated by the
refactoring of `ExternalSorterSuite`, which had a lot of duplicate code and
logic.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark spilling-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9124.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9124
----
commit fd34c25d7ab6d3506d42b2f0cd8c8f26b7726e50
Author: Andrew Or <and...@databricks.com>
Date: 2015-10-14T20:05:42Z
Fix and clean up ExternalSorterSuite
This commit does several things:
- remove noisy warning in GrantEverythingMemoryManager
- remove duplciate code in ExternalSorterSuite
- add a force spill threshold to make it easier to verify spilling
- ensure spilling tests actually spill in ExternalSorterSuite
commit 285c81c26b210bcbe88f055475b2a53ac61bb5c1
Author: Andrew Or <and...@databricks.com>
Date: 2015-10-14T21:33:51Z
Fix spilling tests in ExternalAppendOnlyMapSuite
commit 7226933d2a40896b9c4d606eb2c0ab6437507431
Author: Andrew Or <and...@databricks.com>
Date: 2015-10-14T22:25:15Z
Fix DistributedSuite
commit 1b7fa3d6d32b1e254a47706db7eccd915c7368aa
Author: Andrew Or <and...@databricks.com>
Date: 2015-10-14T22:28:25Z
Merge branch 'master' of github.com:apache/spark into spilling-tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
