sunchao commented on code in PR #55839:
URL: https://github.com/apache/spark/pull/55839#discussion_r3650444366
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEPropagateEmptyRelation.scala:
##########
@@ -52,19 +53,37 @@ object AQEPropagateEmptyRelation extends
PropagateEmptyRelationBase {
// - positive value means an estimated row count which can be
over-estimated
// - none means the plan has not materialized or the plan can not be
estimated
private def getEstimatedRowCount(plan: LogicalPlan): Option[BigInt] = plan
match {
- case LogicalQueryStage(_, stage: QueryStageExec) if stage.isMaterialized =>
+ case LogicalQueryStage(_, physicalPlan) =>
+ getEstimatedRowCount(physicalPlan)
+
+ case _: EmptyRelation => Some(0)
+
+ case _ => None
+ }
+
+ private def getEstimatedRowCount(plan: SparkPlan): Option[BigInt] = plan
match {
Review Comment:
Good point - fixed in `5a4b32a0b6d`. The rule now handles `BaseLimitExec`,
covering both `LocalLimitExec` and `GlobalLimitExec`. `LIMIT 0` is
unconditionally empty; for other limits, emptiness is propagated only when the
child is proven empty, so offset and unknown/nonempty statistics cannot be
misclassified. The regression covers zero and positive limits, materialized
empty and nonempty stages, unknown statistics, offsets, and `limit = -1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
