[
https://issues.apache.org/jira/browse/SPARK-9203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634106#comment-14634106
]
Konstantin Boudnik commented on SPARK-9203:
-------------------------------------------
Huge +1 from the downstream aggregation's standpoint: say in Bigtop we are
avoid locking to just one solution of something. The case in point is HCFS
which allows you to replace HDFS with another client API compatible
implementation. Same goes to the in-memory caching: why limiting the
integration points of in-memory technologies artificially if more collaboration
is possible.
> Make filesystem pluggable in BlockStore and BlockManager
> --------------------------------------------------------
>
> Key: SPARK-9203
> URL: https://issues.apache.org/jira/browse/SPARK-9203
> Project: Spark
> Issue Type: Improvement
> Reporter: Alexey Goncharuk
>
> I was looking through the code in order to understand better how RDD is
> persisted to Tachyon off-heap filesystem. It looks like that the Tachyon
> filesystem is hard-coded and there is no way to switch to another in-memory
> filesystem. I think it would be great if the implementation of the
> BlockManager and BlockStore would be able to plug in another filesystem.
> For example, Apache Ignite also has an implementation of in-memory filesystem
> which can store data in on-heap and off-heap formats. It would be great if it
> could integrate with Spark.
> Apache Ignite In-Memory Filesystem:
> https://ignite.incubator.apache.org/features/igfs.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
