GitHub user caneGuy opened a pull request:
https://github.com/apache/spark/pull/19168
[SPARK-21956][CORE] Fetch up to max bytes when buf really been released
## What changes were proposed in this pull request?
Right now,ShuffleBlockFetcherIterator will decrease bytesInFlight
immediatily when take from results queue.But currentresult's bytebuf has not
been released at that time.Then direct memory may be a little out of control.
We should decrease bytesInFlight when currentresult's bytebuf really been
released.
## How was this patch tested?
Exsisted Unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/caneGuy/spark zhoukang/fix-fetch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19168.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19168
----
commit 0a473bbf905874006711126f5b7375845df802db
Author: zhoukang <[email protected]>
Date: 2017-09-08T09:39:38Z
[SPARK-MI][CORE] Fetch up to max bytes when buf really released
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
