[
https://issues.apache.org/jira/browse/SPARK-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012081#comment-14012081
]
Raymond Liu commented on SPARK-1957:
------------------------------------
Initial pull request at : https://github.com/apache/spark/pull/907
The current approaching as below :
A diskStoreManager layer is added to manage multiple layered disk stores, each
layer of store employ a quota on volume. except the last store, which should be
the fall back media and thus do not have hard quota limitation. On each Store
layer, when the quota is reached, the further data is written to lower layer.
The diskStoreManager is the common interface that BlockManager call into.
The shuffle realted staffs take the assumption that disk store is a pure file
based store, and take short cut path to write file directly, and also has no
easy way to control quota, especially when shuffle consolidate is involved. So
in the solution, the shuffle related code is pick out, and all shuffle data
goto the last fallback data store. say most probably HDDs.
Should have shuffle related code not access file directly, and working with
current several proposals on shuffle framework to make it fit in pluggable
store infrastructure in the future.
> Pluggable disk store for BlockManager
> -------------------------------------
>
> Key: SPARK-1957
> URL: https://issues.apache.org/jira/browse/SPARK-1957
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Reporter: Raymond Liu
>
> As the first step toward the goal of SPAK-1733, support a pluggable disk
> store to allow different disk storage to be plug into the BlockManager's
> DiskStore layer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
