Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11242#discussion_r56892490
--- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
@@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
var rdds: Seq[RDD[T]])
extends RDD[T](sc, Nil) { // Nil since we implement getDependencies
+ // Evaluate partitions in parallel. Partitions of each rdd will be
cached by the `partitions`
+ // val in `RDD`.
+ private[spark] lazy val parallelPartitionEval: Boolean = {
--- End diff --
Such an implementation would break all over in, say, MapReduce. Can it be
likely, like is there a known instance of this? I must say that fixing and
redeploying the code strikes me as a great recommendation, since it is going to
break a bunch of other places in Spark that actually read data and maybe rely
on the correctness of this shared state, and where a failure could be silent
corruption. Making a flag signals that it's a valid design choice.
I wouldn't normally object to a hidden undocumented flag as a safety valve
but I can't see a case where it's safe to recommend?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
