Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r212195284
--- Diff:
core/src/main/scala/org/apache/spark/rdd/ZippedPartitionsRDD.scala ---
@@ -95,6 +99,18 @@ private[spark] class ZippedPartitionsRDD2[A: ClassTag,
B: ClassTag, V: ClassTag]
rdd2 = null
f = null
}
+
+ private def isRandomOrder(rdd: RDD[_]): Boolean = {
+ rdd.computingRandomLevel == RDD.RandomLevel.UNORDERED
--- End diff --
Instead of this, check if order is not IDEMPOTENT.
In both unordered and indeterminate case, we should return indeterminate
for this zip rdd from computingRandomLevel.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
