[
https://issues.apache.org/jira/browse/SPARK-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346499#comment-14346499
]
kishorekumar neelamegam edited comment on SPARK-4368 at 3/4/15 6:58 AM:
------------------------------------------------------------------------
If no changes are needed in Spark, can it be documented to show how Ceph
storage can be used instead of HDFS. I read the above link on GlusterFS,
doesn't provide much info. May be we'll play with Ceph storage and study its
workings. Any pointers will be greatly appreciated.
We have a Ceph cluster, and the concepts relate to HDFS. (distributed
replicated storage) on block level.
http://docs.megam.io/v1.0/docs/megam_ceph_works
We are thinking on testing storage using Ceph as VMs use Ceph. HDFS hogs more
core/memory as its JVM, where ceph is native. This is more of unification of
storage for a customer and reduces cores/memory.
was (Author: indykish):
If no changes or needed in Spark, can it be documented to show how Ceph storage
can be used instead of HDFS. I read the above link on GlusterFS, doesn't
provide much info. May be we'll play with Ceph storage and study its workings.
Any pointers will be greatly appreciated.
We have a Ceph cluster, and the concepts relate to HDFS. (distributed
replicated storage) on block level.
http://docs.megam.io/v1.0/docs/megam_ceph_works
We are thinking on testing storage using Ceph as VMs use Ceph. HDFS hogs more
core/memory as its JVM, where ceph is native. This is more of unification of
storage for a customer and reduces cores/memory.
> Ceph integration?
> -----------------
>
> Key: SPARK-4368
> URL: https://issues.apache.org/jira/browse/SPARK-4368
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Reporter: Serge Smertin
>
> There is a use-case of storing big number of relatively small BLOB objects
> (2-20Mb), which has to have some ugly workarounds in HDFS environments. There
> is a need to process those BLOBs close to data themselves, so that's why
> MapReduce paradigm is good, as it guarantees data locality.
> Ceph seems to be one of the systems that maintains both of the properties
> (small files and data locality) -
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/032119.html. I
> know already that Spark supports GlusterFS -
> http://mail-archives.apache.org/mod_mbox/spark-user/201404.mbox/%3ccf657f2b.5b3a1%[email protected]%3E
> So i wonder, could there be an integration with this storage solution and
> what could be the effort of doing that?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
