GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/8511
[SPARK-10341] [SQL] fix memory starving in unsafe SMJ
In SMJ, the first ExternalSorter could consume all the memory before
spilling, then the second can not even acquire the first page.
Before we have a better memory allocator, SMJ should call prepare() before
call any compute() of it's children.
cc @rxin @JoshRosen
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark smj_memory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8511.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8511
----
commit 1afb4f35e70b1d91b073dfe03e801d7570b7a97a
Author: Davies Liu <[email protected]>
Date: 2015-08-28T20:47:50Z
fix memory starving in unsafe SMJ
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
