Dear all, In my test programer, there are 3 partitions for each RDD, the iteration procedure is as follows: var rdd_0 = ... // init for (...) { *rdd_1* = *rdd_0*.reduceByKey(...).partitionBy(p) // calculate rdd_1 from rdd_0 *rdd_0* = *rdd_0*.partitionBy(p).join(*rdd_1*)... // update rdd_0 by rdd_1 *rdd_0*./action/() } I thought rdd_0 and rdd_1 are part by the same partitioner, and their corresponding partitions are on the same node. for example, rdd_0's partition_0 and rdd_1's partiiton_0 are on the same node in each iteration. But in fact, rdd_0's partition_0 changes its location between workers. Any way to make rdd_0 and rdd_1's partitions not changing their locations, and their corresponding partitions are on the same node for fast join() ? Best Regards, randy
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/why-do-RDD-s-partitions-migrate-between-worker-nodes-in-different-iterations-tp16669.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org