2015-07-20 21:32 GMT-07:00 Prashant Sharma scrapco...@gmail.com:
+1 Looks like a nice idea(I do not see any harm). Would you like to work
on the patch to support it ?
Prashant Sharma
Yes, I would like to contribute to it once we clarify the appropriate path.
--Alexey
On Tue, Jul 21,
2015-07-20 23:29 GMT-07:00 Matei Zaharia matei.zaha...@gmail.com:
I agree with this -- basically, to build on Reynold's point, you should be
able to get almost the same performance by implementing either the Hadoop
FileSystem API or the Spark Data Source API over Ignite in the right way.
This
Hi Alexey,
SPARK-6479https://issues.apache.org/jira/browse/SPARK-6479 is for the plugin
API, and SPARK-6112https://issues.apache.org/jira/browse/SPARK-6112 is for
hdfs plugin.
Thanks.
Zhan Zhang
On Jul 21, 2015, at 10:56 AM, Alexey Goncharuk
2015-07-20 21:40 GMT-07:00 Reynold Xin r...@databricks.com:
I sent it prematurely.
They are already pluggable, or at least in the process to be more
pluggable. In 1.4, instead of calling the external system's API directly,
we added an API for that. There is a patch to add support for HDFS
I agree with this -- basically, to build on Reynold's point, you should be able
to get almost the same performance by implementing either the Hadoop FileSystem
API or the Spark Data Source API over Ignite in the right way. This would let
people save data persistently in Ignite in addition to
(Related, not important comment: it would also be nice to separate out the
Tachyon dependency from core, as it's conceptually pluggable but is still
hard-coded into several places in the code, and a lot of the comments/docs
in the code.)
On Tue, Jul 21, 2015 at 5:40 AM, Reynold Xin
Hello Spark community,
I was looking through the code in order to understand better how RDD is
persisted to Tachyon off-heap filesystem. It looks like that the Tachyon
filesystem is hard-coded and there is no way to switch to another in-memory
filesystem. I think it would be great if the
+1 Looks like a nice idea(I do not see any harm). Would you like to work on
the patch to support it ?
Prashant Sharma
On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk
alexey.goncha...@gmail.com wrote:
Hello Spark community,
I was looking through the code in order to understand better how
They are already pluggable.
On Mon, Jul 20, 2015 at 9:32 PM, Prashant Sharma scrapco...@gmail.com
wrote:
+1 Looks like a nice idea(I do not see any harm). Would you like to work
on the patch to support it ?
Prashant Sharma
On Tue, Jul 21, 2015 at 2:46 AM, Alexey Goncharuk
I sent it prematurely.
They are already pluggable, or at least in the process to be more
pluggable. In 1.4, instead of calling the external system's API directly,
we added an API for that. There is a patch to add support for HDFS
in-memory cache.
Somewhat orthogonal to this, longer term, I am
10 matches
Mail list logo