JoshRosen commented on a change in pull request #34265:
URL: https://github.com/apache/spark/pull/34265#discussion_r728560126
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -732,6 +732,35 @@ private[spark] class DAGScheduler(
missing.toList
}
+ /** Invoke `.partitions` on the given RDD and all of its ancestors */
+ private def eagerlyComputePartitionsForRddAndAncestors(rdd: RDD[_]): Unit = {
+ val startTime = System.nanoTime
+ val visitedRdds = new HashSet[RDD[_]]
+ // We are manually maintaining a stack here to prevent StackOverflowError
+ // caused by recursively visiting
+ val waitingForVisit = new ListBuffer[RDD[_]]
+ waitingForVisit += rdd
+
+ def visit(rdd: RDD[_]): Unit = {
+ if (!visitedRdds(rdd)) {
+ visitedRdds += rdd
+
+ // Eagerly compute:
+ rdd.partitions
+
+ for (dep <- rdd.dependencies) {
+ waitingForVisit.prepend(dep.rdd)
+ }
+ }
+ }
+
+ while (waitingForVisit.nonEmpty) {
+ visit(waitingForVisit.remove(0))
+ }
+ logDebug("eagerlyComputePartitionsForRddAndAncestors for RDD %d took %f
seconds"
Review comment:
I agree that user-facing INFO/WARN messages for slow `getPartitions()`
calls might be useful for attributing long gaps in the driver logs.
I also wonder whether users might be better served by placing that logging
closer to the actual source of the slowness: for example, `getPartitions()` is
most likely going to be slow due to file listing in `HadoopRDD`, so if we place
logging in `HadoopRDD` then we can add more information (e.g. "file listing
took x seconds and listed y files").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
