Hi Dmitry,
Ignite provides better data distribution and better performance if there are
more partitions than nodes in topology. 1024 is the default number of
partitions, but you can change it by providing custom affinity function
configuration:
CacheConfiguration cfg = new CacheConfiguration("hello-world-cache").
setAffinity(new RendezvousAffinityFunction(false, 32)); // 32 partitions
instead of 1024.
final IgniteRDD igniteRDD = igniteContext.fromCache(cfg);
You can try this and see if it gets better.
Actually, I think that methods like isEmpty should be overridden in
IgniteRDD to use native IgniteCache API, it will be much faster. I created a
ticket for this task [1], feel free to provide your comments there. Are
there any other methods that should be optimized?
[1] https://issues.apache.org/jira/browse/IGNITE-2538
-Val
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Sharing-Spark-RDDs-with-Ignite-tp2805p2808.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
