This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 1ba422d5acae [SPARK-57956][4.1][SQL] Fix AQE incorrectly eliminating a
global LIMIT over a non-materialized query stage
1ba422d5acae is described below
commit 1ba422d5acae4e5b3ceddffdcacd9dfe50af79a5
Author: raviston <[email protected]>
AuthorDate: Thu Nov 5 14:12:54 2026 +0800
[SPARK-57956][4.1][SQL] Fix AQE incorrectly eliminating a global LIMIT over
a non-materialized query stage
### What changes were proposed in this pull request?
Backport of #57347 (SPARK-57956) to `branch-4.1`. See also the `branch-4.2`
backport, #60017.
`LogicalQueryStage.maxRows` now uses the stage's `rowCount` only when the
stage is materialized and its statistics are runtime statistics. Otherwise it
falls back to `logicalPlan.maxRows`, so a cost estimate is never used as a hard
upper bound by `EliminateLimits` during AQE re-optimization.
The `LogicalQueryStage` change applies cleanly. `AdaptiveQueryExecSuite`
needed a manual resolution of the import block only, because `master` imports
names used by later tests that are not on `branch-4.1`, and the test needs
`HashedRelationBroadcastMode` added to the `joins` import. The added test is
identical to the one in #57347.
### Why are the changes needed?
SPARK-57956 was merged into `master`, `branch-4.x` and `branch-4.3`, but
not into `branch-4.2` or `branch-4.1`, which still have the old `maxRows`:
```scala
override def maxRows: Option[Long] =
stats.rowCount.map(_.min(Long.MaxValue).toLong)
```
#59434 reports a wrong result on 4.1.3 from the same cause: with AQE on, a
nested `LIMIT ... OFFSET` over a `LEFT JOIN` sometimes returns more rows than
its outer `LIMIT`.
```sql
SELECT x FROM (
SELECT x FROM (
SELECT p.id AS x FROM p LEFT JOIN (
SELECT p.id AS anchor FROM p INNER JOIN k ON p.id = k.a INNER JOIN p
AS q ON k.b = q.id
) o ON p.id = o.anchor
LIMIT 6 OFFSET 2
) LIMIT 3 OFFSET 1
)
```
With the tables from #59434 (RDD-backed, `local[1]`,
`spark.sql.shuffle.partitions=2`), running this query 200 times on `branch-4.1`:
| | AQE on | AQE off |
|---|---|---|
| Before this PR | 3 rows: 182 runs, **5 rows: 18 runs** | 3 rows: 200 runs
|
| After this PR | 3 rows: 200 runs | 3 rows: 200 runs |
### Does this PR introduce _any_ user-facing change?
Yes. It fixes a correctness bug where, with AQE enabled, a query with a
LIMIT over certain plans could return more rows than the LIMIT.
### How was this patch tested?
- The new test `SPARK-57956: unmaterialized query stage exposes structural
maxRows, not the estimate` passes.
- The existing test `SPARK-36424: Support eliminate limits in AQE
Optimizer` still passes.
- The #59434 query above, run 200 times, as shown in the table.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #60018 from Raviston6296/SPARK-57956-4.1.
Lead-authored-by: raviston <[email protected]>
Co-authored-by: Vaibhav Garg <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 68af27f87349ed376c25f2c7bbb162c7f6b8a848)
Signed-off-by: Cheng Pan <[email protected]>
---
.../sql/execution/adaptive/LogicalQueryStage.scala | 16 +++++++-
.../adaptive/AdaptiveQueryExecSuite.scala | 44 ++++++++++++++++++++--
2 files changed, 55 insertions(+), 5 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
index 506f52fd9072..4caa21833d36 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
@@ -71,7 +71,21 @@ case class LogicalQueryStage(
physicalStats.getOrElse(logicalPlan.stats)
}
- override def maxRows: Option[Long] =
stats.rowCount.map(_.min(Long.MaxValue).toLong)
+ override def maxRows: Option[Long] = {
+ // A query stage's `rowCount` is an exact, valid upper bound only when it
comes from the
+ // runtime statistics of a materialized stage. Checking `isMaterialized`
alone is not enough:
+ // `computeStats()` can still fall back to `logicalPlan.stats` (a cost
estimate, with
+ // `isRuntime = false`) when the physical-stage lookup yields no
statistics, and that estimate
+ // can under-count (e.g., returning 0). Treating such an estimate as a
hard upper bound lets
+ // rules such as EliminateLimits wrongly drop a LIMIT and change the
result cardinality
+ // (SPARK-57956). Only trust `rowCount` when the stats are materialized
runtime stats;
+ // otherwise fall back to `logicalPlan.maxRows`, which is always a sound
upper bound.
+ if (isMaterialized && stats.isRuntime) {
+ stats.rowCount.map(_.min(Long.MaxValue).toLong)
+ } else {
+ logicalPlan.maxRows
+ }
+ }
override def isMaterialized: Boolean = physicalPlan.exists {
case s: QueryStageExec => s.isMaterialized
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index fb8fab6a80f8..a85b3963ce1c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -30,9 +30,9 @@ import org.apache.spark.scheduler.{SparkListener,
SparkListenerEvent, SparkListe
import org.apache.spark.shuffle.sort.SortShuffleManager
import org.apache.spark.sql.{DataFrame, Dataset, QueryTest, Row, SparkSession}
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference, Literal}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight,
EliminateLimits}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, GlobalLimit,
LeafNode, LogicalPlan, Statistics}
import org.apache.spark.sql.classic.Strategy
import org.apache.spark.sql.execution._
import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
@@ -41,7 +41,7 @@ import
org.apache.spark.sql.execution.command.DataWritingCommandExec
import org.apache.spark.sql.execution.datasources.noop.NoopDataSource
import org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec
import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec,
ENSURE_REQUIREMENTS, Exchange, REPARTITION_BY_COL, REPARTITION_BY_NUM,
ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike, ShuffleOrigin}
-import org.apache.spark.sql.execution.joins.{BaseJoinExec,
BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, ShuffledHashJoinExec,
ShuffledJoin, SortMergeJoinExec}
+import org.apache.spark.sql.execution.joins.{BaseJoinExec,
BroadcastHashJoinExec, BroadcastNestedLoopJoinExec,
HashedRelationBroadcastMode, ShuffledHashJoinExec, ShuffledJoin,
SortMergeJoinExec}
import org.apache.spark.sql.execution.metric.SQLShuffleReadMetricsReporter
import
org.apache.spark.sql.execution.ui.{SparkListenerSQLAdaptiveExecutionUpdate,
SparkListenerSQLAdaptiveSQLMetricUpdates, SparkListenerSQLExecutionStart}
import org.apache.spark.sql.execution.window.WindowExec
@@ -56,6 +56,16 @@ import org.apache.spark.tags.SlowSQLTest
import org.apache.spark.util.ArrayImplicits._
import org.apache.spark.util.Utils
+/**
+ * A leaf whose cost estimate under-counts its structural row bound. Used by
SPARK-57956 to verify
+ * that an unmaterialized query stage exposes the structural `maxRows` instead
of an estimate.
+ */
+private case class UnderCountLeaf(output: Seq[Attribute]) extends LeafNode {
+ override def maxRows: Option[Long] = Some(2L)
+ override def computeStats(): Statistics =
+ Statistics(sizeInBytes = BigInt(1), rowCount = Some(BigInt(0)))
+}
+
@SlowSQLTest
class AdaptiveQueryExecSuite
extends QueryTest
@@ -2521,6 +2531,32 @@ class AdaptiveQueryExecSuite
}
}
+ test("SPARK-57956: unmaterialized query stage exposes structural maxRows,
not the estimate") {
+ // Build a LogicalQueryStage whose underlying stage is never materialized.
Its computeStats()
+ // falls back to the logical plan's cost estimate - here an under-count of
0 rows - which must
+ // NOT be promoted to a hard maxRows bound. Otherwise EliminateLimits
would drop a LIMIT that
+ // still needs to be applied once the stage runs (SPARK-57956).
+ val output = Seq(AttributeReference("a", IntegerType)())
+ val logical = UnderCountLeaf(output)
+ val scan = LocalTableScanExec(output, Nil, None)
+ val exchange = BroadcastExchangeExec(
+ HashedRelationBroadcastMode(output, isNullAware = false), scan)
+ val queryStage = LogicalQueryStage(logical, BroadcastQueryStageExec(0,
exchange, exchange))
+
+ assert(!queryStage.isMaterialized)
+ // The under-counted estimate is what computeStats() surfaces...
+ assert(queryStage.stats.rowCount.contains(BigInt(0)))
+ // ...but maxRows must remain the structural bound (2), not the estimate
(0).
+ assert(queryStage.maxRows.contains(2L),
+ "an unmaterialized stage must expose its structural maxRows, not the
row-count estimate")
+
+ // Since the structural bound (2) exceeds the limit (1), the LIMIT must be
retained. With the
+ // estimate wrongly promoted (maxRows = 0), EliminateLimits would instead
drop it.
+ val limited = GlobalLimit(Literal(1), queryStage)
+ assert(EliminateLimits(limited).isInstanceOf[GlobalLimit],
+ "LIMIT must be retained when the child's structural row bound exceeds
the limit")
+ }
+
test("SPARK-48037: Fix SortShuffleWriter lacks shuffle write related metrics
" +
"resulting in potentially inaccurate data") {
withTable("t3") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]