This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 0d29d5e777cf [SPARK-57956][4.2][SQL] Fix AQE incorrectly eliminating a
global LIMIT over a non-materialized query stage
0d29d5e777cf is described below
commit 0d29d5e777cfe550f928fa967f71c0b75f176f31
Author: raviston <[email protected]>
AuthorDate: Thu Nov 5 14:11:14 2026 +0800
[SPARK-57956][4.2][SQL] Fix AQE incorrectly eliminating a global LIMIT over
a non-materialized query stage
### What changes were proposed in this pull request?
Backport of #57347 (SPARK-57956) to `branch-4.2`.
`LogicalQueryStage.maxRows` now uses the stage's `rowCount` only when the
stage is materialized and its statistics are runtime statistics. Otherwise it
falls back to `logicalPlan.maxRows`, so a cost estimate is never used as a hard
upper bound by `EliminateLimits` during AQE re-optimization.
The `LogicalQueryStage` change applies cleanly. `AdaptiveQueryExecSuite`
needed a manual resolution of the import block only, because `master` imports
names used by later tests that are not on `branch-4.2`. The added test is
identical to the one in #57347.
### Why are the changes needed?
SPARK-57956 was merged into `master`, `branch-4.x` and `branch-4.3`, but
not into `branch-4.2` or `branch-4.1`, which still have the old `maxRows`:
```scala
override def maxRows: Option[Long] =
stats.rowCount.map(_.min(Long.MaxValue).toLong)
```
#59434 reports a wrong result on 4.1.3 from the same cause: with AQE on, a
nested `LIMIT ... OFFSET` over a `LEFT JOIN` sometimes returns more rows than
its outer `LIMIT`.
```sql
SELECT x FROM (
SELECT x FROM (
SELECT p.id AS x FROM p LEFT JOIN (
SELECT p.id AS anchor FROM p INNER JOIN k ON p.id = k.a INNER JOIN p
AS q ON k.b = q.id
) o ON p.id = o.anchor
LIMIT 6 OFFSET 2
) LIMIT 3 OFFSET 1
)
```
With the tables from #59434 (RDD-backed, `local[1]`,
`spark.sql.shuffle.partitions=2`), running this query 200 times on `branch-4.2`:
| | AQE on | AQE off |
|---|---|---|
| Before this PR | 3 rows: 181 runs, **5 rows: 19 runs** | 3 rows: 200 runs
|
| After this PR | 3 rows: 200 runs | 3 rows: 200 runs |
### Does this PR introduce _any_ user-facing change?
Yes. It fixes a correctness bug where, with AQE enabled, a query with a
LIMIT over certain plans could return more rows than the LIMIT.
### How was this patch tested?
- The new test `SPARK-57956: unmaterialized query stage exposes structural
maxRows, not the estimate` passes.
- The existing test `SPARK-36424: Support eliminate limits in AQE
Optimizer` still passes.
- The #59434 query above, run 200 times, as shown in the table.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #60017 from Raviston6296/SPARK-57956-4.2.
Lead-authored-by: raviston <[email protected]>
Co-authored-by: Vaibhav Garg <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.../sql/execution/adaptive/LogicalQueryStage.scala | 16 ++++++++-
.../adaptive/AdaptiveQueryExecSuite.scala | 42 ++++++++++++++++++++--
2 files changed, 54 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
index 62e00d1ea6ed..09fc97bb46fa 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
@@ -71,7 +71,21 @@ case class LogicalQueryStage(
physicalStats.getOrElse(logicalPlan.stats)
}
- override def maxRows: Option[Long] =
stats.rowCount.map(_.min(Long.MaxValue).toLong)
+ override def maxRows: Option[Long] = {
+ // A query stage's `rowCount` is an exact, valid upper bound only when it
comes from the
+ // runtime statistics of a materialized stage. Checking `isMaterialized`
alone is not enough:
+ // `computeStats()` can still fall back to `logicalPlan.stats` (a cost
estimate, with
+ // `isRuntime = false`) when the physical-stage lookup yields no
statistics, and that estimate
+ // can under-count (e.g., returning 0). Treating such an estimate as a
hard upper bound lets
+ // rules such as EliminateLimits wrongly drop a LIMIT and change the
result cardinality
+ // (SPARK-57956). Only trust `rowCount` when the stats are materialized
runtime stats;
+ // otherwise fall back to `logicalPlan.maxRows`, which is always a sound
upper bound.
+ if (isMaterialized && stats.isRuntime) {
+ stats.rowCount.map(_.min(Long.MaxValue).toLong)
+ } else {
+ logicalPlan.maxRows
+ }
+ }
override def isMaterialized: Boolean = physicalPlan.exists {
case s: QueryStageExec => s.isMaterialized
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index a004e40268a6..8187058b1851 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -29,10 +29,10 @@ import org.apache.spark.scheduler.{SparkListener,
SparkListenerEvent, SparkListe
import org.apache.spark.shuffle.sort.SortShuffleManager
import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference, EqualTo, IsNull, Or}
-import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference, EqualTo, IsNull, Literal, Or}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight,
EliminateLimits}
import org.apache.spark.sql.catalyst.plans.{Inner, LeftAnti}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join, JoinHint,
LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, GlobalLimit,
Join, JoinHint, LeafNode, LocalRelation, LogicalPlan, Statistics}
import org.apache.spark.sql.classic.Strategy
import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
@@ -59,6 +59,16 @@ import org.apache.spark.tags.SlowSQLTest
import org.apache.spark.util.ArrayImplicits._
import org.apache.spark.util.Utils
+/**
+ * A leaf whose cost estimate under-counts its structural row bound. Used by
SPARK-57956 to verify
+ * that an unmaterialized query stage exposes the structural `maxRows` instead
of an estimate.
+ */
+private case class UnderCountLeaf(output: Seq[Attribute]) extends LeafNode {
+ override def maxRows: Option[Long] = Some(2L)
+ override def computeStats(): Statistics =
+ Statistics(sizeInBytes = BigInt(1), rowCount = Some(BigInt(0)))
+}
+
@SlowSQLTest
class AdaptiveQueryExecSuite
extends SharedSparkSession
@@ -2746,6 +2756,32 @@ class AdaptiveQueryExecSuite
}
}
+ test("SPARK-57956: unmaterialized query stage exposes structural maxRows,
not the estimate") {
+ // Build a LogicalQueryStage whose underlying stage is never materialized.
Its computeStats()
+ // falls back to the logical plan's cost estimate - here an under-count of
0 rows - which must
+ // NOT be promoted to a hard maxRows bound. Otherwise EliminateLimits
would drop a LIMIT that
+ // still needs to be applied once the stage runs (SPARK-57956).
+ val output = Seq(AttributeReference("a", IntegerType)())
+ val logical = UnderCountLeaf(output)
+ val scan = LocalTableScanExec(output, Nil, None)
+ val exchange = BroadcastExchangeExec(
+ HashedRelationBroadcastMode(output, isNullAware = false), scan)
+ val queryStage = LogicalQueryStage(logical, BroadcastQueryStageExec(0,
exchange, exchange))
+
+ assert(!queryStage.isMaterialized)
+ // The under-counted estimate is what computeStats() surfaces...
+ assert(queryStage.stats.rowCount.contains(BigInt(0)))
+ // ...but maxRows must remain the structural bound (2), not the estimate
(0).
+ assert(queryStage.maxRows.contains(2L),
+ "an unmaterialized stage must expose its structural maxRows, not the
row-count estimate")
+
+ // Since the structural bound (2) exceeds the limit (1), the LIMIT must be
retained. With the
+ // estimate wrongly promoted (maxRows = 0), EliminateLimits would instead
drop it.
+ val limited = GlobalLimit(Literal(1), queryStage)
+ assert(EliminateLimits(limited).isInstanceOf[GlobalLimit],
+ "LIMIT must be retained when the child's structural row bound exceeds
the limit")
+ }
+
test("SPARK-48037: Fix SortShuffleWriter lacks shuffle write related metrics
" +
"resulting in potentially inaccurate data") {
withTable("t3") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]