cloud-fan commented on code in PR #57347:
URL: https://github.com/apache/spark/pull/57347#discussion_r3690713745
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala:
##########
@@ -71,7 +71,21 @@ case class LogicalQueryStage(
physicalStats.getOrElse(logicalPlan.stats)
}
- override def maxRows: Option[Long] =
stats.rowCount.map(_.min(Long.MaxValue).toLong)
+ override def maxRows: Option[Long] = {
+ // A query stage's `rowCount` is an exact, valid upper bound only when it
comes from the
+ // runtime statistics of a materialized stage. Checking `isMaterialized`
alone is not enough:
+ // `computeStats()` can still fall back to `logicalPlan.stats` (a cost
estimate, with
+ // `isRuntime = false`) when the physical-stage lookup yields no
statistics, and that estimate
+ // can under-count (e.g. return 0). Treating such an estimate as a hard
upper bound lets rules
Review Comment:
Use the grammatically parallel gerund here; `e.g.` also conventionally takes
a comma.
```suggestion
// can under-count (e.g., returning 0). Treating such an estimate as a
hard upper bound lets rules
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
