Vladimir First, thank you for your reply - it was very quick and insightful!
I agree with you that comparing to in-JVM is not a fair comparison. I'm only using that as the absolute upper-bound for performance; in other words, the closer we are to that the better. I have tried your (very good) suggestions, with mixed results. *1)* Setting *CacheConfiguration.copyOnRead* flag to "false". This helped quite a bit - almost 2X performance improvement. *2)* I have the recommended Ignite JVM optimizations and they seem to be working well, no major GC pauses. *3)* *Using Compute Tasks as opposed to broadcast()*. Great suggestion, and I must say Ignite's API is super intuitive and powerful for map-reduce. I implemented this as follows: each server node now gets a number of jobs (I made the degree of parallelism is configurable through a jvm property to make it easy to test). Each job will compute the value based on the local cache partitions it gets assigned (I use the affinity API to get the cache partitions on each node). I can see this as being very powerful for big-data caches. The optimal number of jobs per node is between 2 and 4 for my tests (If I increase it to say 8-16 there's a lot of resource contention and performance degrades). The only minor disadvantage over broadcast() is that I notice the messages take more to get back to the client node (in the order of 100 millis as opposed to 10 millis with broadcast). *4) *Unfortunately, *the ScanQuery API is much slower than the Entry iteration* for me. I can't understand why. I did some profiling and it looks like the ScanQuery does go through a lot more layers of code and it does a lot more work than the iterator. Can you tell me what can cause this? Also, when I tried the iterator with multiple tasks on the same node, there was a lot of blocking, so actually broadcast() (i.e. 1 task per node) was much faster. It would be super nice if there were a way to iterate through the entries on a given partition, just like there is a way to assign a partition to the ScanQuery. How can I accomplish this? I'm attaching the updated source code and classes jar so you can try this on your end. Again, thanks so much for your help, and I hope you can provide me some further insight regarding the ScanQuery / Iteration over partitions. Andres ignite_test_src.jar <http://apache-ignite-users.70518.x6.nabble.com/file/n3748/ignite_test_src.jar> inventory_cache_compute.jar <http://apache-ignite-users.70518.x6.nabble.com/file/n3748/inventory_cache_compute.jar> -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-Cache-and-Compute-Performance-tp3722p3748.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
