GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/860
[SPARK-1912] fix compress memory issue during reduce
When we need to read a compressed block, we will first create a compress
stream instance(LZF or Snappy) and use it to wrap that block.
Let's say a reducer task need to read 1000 local shuffle blocks, it will
first prepare to read that 1000 blocks, which means create 1000 compression
stream instance to wrap them. But the initialization of compression instance
will allocate some memory and when we have many compression instance at the
same time, it is a problem.
Actually reducer reads the shuffle blocks one by one, so we can do the
compression instance initialization lazily.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark fix-compress
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/860.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #860
----
commit 8ebff777f940fd440d882999bcd5b2e771d65a3e
Author: Wenchen Fan(Cloud) <[email protected]>
Date: 2014-05-23T10:12:53Z
fix compress memory issue during reduce
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
