Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11242#discussion_r59061019
--- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
@@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
var rdds: Seq[RDD[T]])
extends RDD[T](sc, Nil) { // Nil since we implement getDependencies
+ // visible for testing
+ private[spark] val isPartitionEvalParallel: Boolean =
+ rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
--- End diff --
One JVM can definitely have many instances of one `InputFormat` and get
partitions from them at the same time. If this is where we disagree, let's talk
about that.
You're arguing that it becomes more likely to trigger with this change --
yes. That argues against making this change at all with the default to 'on'. If
this is likely to be a real problem, then we're opting in to possibly subtle
correctness bugs by default. A safety valve only helps if someone even knows
there's a problem and knows this is the source, which is unlikely for most
callers.
I'm further asserting that even if this behavior can be disabled, this is a
bug that's going to be triggered by other Spark usage already. Current usage is
already going to cause this type of problem; it's not even that rare. Although
I can't be sure we aren't seeing bugs due to this, I can't think of any
evidence of this in the wild.
Examples are good, but, these aren't actual bugs at this point right?
they're instances where the format implementation had a bug (at least, made a
much too aggressive assumption) and was fixed. That's the right outcome, since
it's the only thing that actually fixes the problem.
I myself will not merge this change for these reasons. It's hard to feel
too too strongly about an undocumented flag, so I would not veto it if someone
else wants to merge it. My concern is that it does a little harm: this may help
someone think they've avoided a particular type of problem when they have not,
in Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
