GitHub user shimingfei opened a pull request:
https://github.com/apache/spark/pull/5908
Tachyon integration improvement
Two main changes:
1. Add two functions in ExternalBlockManager, which are putValues and
getValues
because the implementation may not rely on the putBytes and getBytes
2. improve Tachyon integration.
1. Currently, when putting data into Tachyon, Spark first serialize all
data in one partition into a ByteBuffer, and then write into Tachyon, this will
uses much memory and increase GC overhead
2. when get data from Tachyon, getValues depends on getBytes, which also
read all data into On heap byte arry, and result in much memory usage.
This PR changes the approach of the two functions, make them read / write
data by stream to reduce memory usage.
In our testing, when data size is huge, this patch reduces about 30% GC
time and 70% full GC time, and total execution time reduces about 10%
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shimingfei/spark Tachyon-integration-rebase
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5908.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5908
----
commit b9d7c3818fe6e0a7ef705211c67554514e130036
Author: Mingfei <[email protected]>
Date: 2015-05-05T09:09:53Z
add getValues and putValues in ExternalBlockManager's Interface
commit a6ce31e2456cefe3ea362b76fcf7ff4bfad003ea
Author: Mingfei <[email protected]>
Date: 2015-05-05T09:10:43Z
Make getValues read data from Tachyon by stream
Make putValues write data to Tachyon by stream
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
