GitHub user liyezhang556520 opened a pull request:
https://github.com/apache/spark/pull/2134
[SPARK-3000][CORE] drop old blocks to disk in parallel when memory is no...
...t large enough for caching new blocks
Currently, old blocks dropping for new blocks' caching are processed by one
thread at the same time. Which can not fully utilize the disk throughput. If
the to be dropped block size is huge, then the dropping time will be very long.
We need to make it processed in parallel. In this patch, dropping blocks
operation are processed in multiple threads, before dropping, each thread will
select the blocks that to be dropped for itself.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyezhang556520/spark spark-3000-v0.4.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2134.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2134
----
commit 357dae839034490bf83b8fdadb413cdef32f2e8b
Author: Zhang, Liye <[email protected]>
Date: 2014-08-26T10:20:30Z
[SPARK-3000][CORE] drop old blocks to disk in parallel when memory is not
large enough for caching new blocks
Currently, old blocks dropping for new blocks' caching are processed by one
thread at the same time. Which can not fully utilize the disk throughput. If
the to be dropped block size is huge, then the dropping time will be very long.
We need to make it processed in parallel. In this patch, dropping blocks
operation are processed in multiple threads, before dropping, each thread will
select the blocks that to be dropped for itself.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
