Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22112#discussion_r213397293
--- Diff: core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
---
@@ -32,12 +32,16 @@ import org.apache.spark.{Partition, TaskContext}
* doesn't modify the keys.
* @param isFromBarrier Indicates whether this RDD is transformed from an
RDDBarrier, a stage
* containing at least one RDDBarrier shall be turned
into a barrier stage.
+ * @param orderSensitiveFunc whether or not the function is
order-sensitive. If it's order
+ * sensitive, it may return totally different
result if the input order
+ * changed. Mostly stateful functions are
order-sensitive.
*/
private[spark] class MapPartitionsRDD[U: ClassTag, T: ClassTag](
var prev: RDD[T],
f: (TaskContext, Int, Iterator[T]) => Iterator[U], // (TaskContext,
partition index, iterator)
preservesPartitioning: Boolean = false,
- isFromBarrier: Boolean = false)
+ isFromBarrier: Boolean = false,
+ orderSensitiveFunc: Boolean = false)
--- End diff --
`orderSensitiveFunc` -> `isOrderSensitive`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
