[
https://issues.apache.org/jira/browse/SPARK-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209624#comment-15209624
]
He Tianyi commented on SPARK-14068:
-----------------------------------
For example:
In my deployment, both SSDs and HDDs are installed on each node. In production
I've encountered 'No space left on device' many times since SSDs does not
always fit all data blocks during shuffle phase.
In this case one may want to implement a 'ssd-first' strategy. That is, use
SSDs if possible, otherwise fallback to HDDs.
Generally, strategies may be highly customized per user.
> Pluggable DiskBlockManager
> --------------------------
>
> Key: SPARK-14068
> URL: https://issues.apache.org/jira/browse/SPARK-14068
> Project: Spark
> Issue Type: New Feature
> Components: Block Manager
> Reporter: He Tianyi
> Priority: Minor
>
> Currently DiskBlockManager places file by hashing strategy, this can be
> non-optimal in some scenario.
> Maybe we make it pluggable. That is, DiskBlockManager can be replaced with
> another implementation with different strategy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
