On 2014-10-10, 3:03 AM, Dan Berindei wrote: > > The problem is that the intermediate keys aren't in the same segment: > we want the reduce phase to access only keys local to the reducing > node, and keys in different input segments can yield values for the > same intermediate key. So like you say, we'd have to retry on every > topology change in the intermediary cache, not just the ones affecting > segment _i_. > If we have to retry for all segments on every topology change than I am not sure why would it make sense to work on this optimization and topology handling mechanism at all. We have to handle the cases where one node might have completed map phase and inserted deltas, while the other only started inserting deltas, and the third one is still doing map phase and has not inserted any deltas at all. The same thing with reduce portion. It seems to me that in the end any algorithm we come up with will not be not much better than: detect topology change, retry map/reduce job.
Vladimir _______________________________________________ infinispan-dev mailing list [email protected] https://lists.jboss.org/mailman/listinfo/infinispan-dev
