Re: Make off-heap store pluggable

2015-07-21 Thread Alexey Goncharuk
2015-07-20 21:32 GMT-07:00 Prashant Sharma scrapco...@gmail.com: +1 Looks like a nice idea(I do not see any harm). Would you like to work on the patch to support it ? Prashant Sharma Yes, I would like to contribute to it once we clarify the appropriate path. --Alexey On Tue, Jul 21,

Re: Make off-heap store pluggable

2015-07-21 Thread Alexey Goncharuk
2015-07-20 23:29 GMT-07:00 Matei Zaharia matei.zaha...@gmail.com: I agree with this -- basically, to build on Reynold's point, you should be able to get almost the same performance by implementing either the Hadoop FileSystem API or the Spark Data Source API over Ignite in the right way. This

Re: Make off-heap store pluggable

2015-07-21 Thread Zhan Zhang
Hi Alexey, SPARK-6479https://issues.apache.org/jira/browse/SPARK-6479 is for the plugin API, and SPARK-6112https://issues.apache.org/jira/browse/SPARK-6112 is for hdfs plugin. Thanks. Zhan Zhang On Jul 21, 2015, at 10:56 AM, Alexey Goncharuk

Re: Make off-heap store pluggable

2015-07-21 Thread Alexey Goncharuk
2015-07-20 21:40 GMT-07:00 Reynold Xin r...@databricks.com: I sent it prematurely. They are already pluggable, or at least in the process to be more pluggable. In 1.4, instead of calling the external system's API directly, we added an API for that. There is a patch to add support for HDFS

Re: Make off-heap store pluggable

2015-07-21 Thread Matei Zaharia
I agree with this -- basically, to build on Reynold's point, you should be able to get almost the same performance by implementing either the Hadoop FileSystem API or the Spark Data Source API over Ignite in the right way. This would let people save data persistently in Ignite in addition to

Re: Make off-heap store pluggable

2015-07-21 Thread Sean Owen
(Related, not important comment: it would also be nice to separate out the Tachyon dependency from core, as it's conceptually pluggable but is still hard-coded into several places in the code, and a lot of the comments/docs in the code.) On Tue, Jul 21, 2015 at 5:40 AM, Reynold Xin

Make off-heap store pluggable

2015-07-20 Thread Alexey Goncharuk
Hello Spark community, I was looking through the code in order to understand better how RDD is persisted to Tachyon off-heap filesystem. It looks like that the Tachyon filesystem is hard-coded and there is no way to switch to another in-memory filesystem. I think it would be great if the

Re: Make off-heap store pluggable

2015-07-20 Thread Prashant Sharma
+1 Looks like a nice idea(I do not see any harm). Would you like to work on the patch to support it ? Prashant Sharma On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk alexey.goncha...@gmail.com wrote: Hello Spark community, I was looking through the code in order to understand better how

Re: Make off-heap store pluggable

2015-07-20 Thread Reynold Xin
They are already pluggable. On Mon, Jul 20, 2015 at 9:32 PM, Prashant Sharma scrapco...@gmail.com wrote: +1 Looks like a nice idea(I do not see any harm). Would you like to work on the patch to support it ? Prashant Sharma On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk

Re: Make off-heap store pluggable

2015-07-20 Thread Reynold Xin
I sent it prematurely. They are already pluggable, or at least in the process to be more pluggable. In 1.4, instead of calling the external system's API directly, we added an API for that. There is a patch to add support for HDFS in-memory cache. Somewhat orthogonal to this, longer term, I am