Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/11242#discussion_r56890135
--- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
@@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
var rdds: Seq[RDD[T]])
extends RDD[T](sc, Nil) { // Nil since we implement getDependencies
+ // Evaluate partitions in parallel. Partitions of each rdd will be
cached by the `partitions`
+ // val in `RDD`.
+ private[spark] lazy val parallelPartitionEval: Boolean = {
--- End diff --
I agree that it's a flawed implementation, but it's reasonable to say they
exist without causing problems today because split planning isn't done in
parallel. I think your point about how someone would find this setting is
valid, but I'm more concerned that someone that hits the problem should have a
way around it. Without a way to turn this off, a user would be forced to either
roll back their Spark version or fix and redeploy their InputFormat code. Both
are expensive compared to a config option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
