GitHub user zhzhan opened a pull request:
https://github.com/apache/spark/pull/6449
[SPARK-6112][Block Manager] Provide external block store support through
HDFS RAM_DISK
This is the follow up for SPARK-6479 to add external block store by HDFS.
It provides similar functionalities of Tachyon store. All configurations are
also similar. To really leverage the hdfs ramdisk capacity, the user need to
set the directory to be lazy persist. Due to this is new feature introduced
into hdfs, the code does not do it programmatically to avoid any compatibility
issue. Without the configuration, the block may persist to hdfs disk, instead
of ramdisk.
hdfs dfsadmin -setStoragePolicy /memory1 LAZY_PERSIST
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhzhan/spark SPARK-6112
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6449.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6449
----
commit 6bfeb07dd2dad39e7e1fda921db1f52fcffb5b68
Author: Zhan Zhang <[email protected]>
Date: 2015-05-26T16:39:13Z
hdfs block store support initial commit
add hdfs block manager
external block store hdfs support
hdfs support for external block store
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
