This is an automated email from the ASF dual-hosted git repository.

pan3793 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.0 by this push:
     new 1ba422d5acae [SPARK-57956][4.1][SQL] Fix AQE incorrectly eliminating a 
global LIMIT over a non-materialized query stage
1ba422d5acae is described below

commit 1ba422d5acae4e5b3ceddffdcacd9dfe50af79a5
Author: raviston <[email protected]>
AuthorDate: Thu Nov 5 14:12:54 2026 +0800

    [SPARK-57956][4.1][SQL] Fix AQE incorrectly eliminating a global LIMIT over 
a non-materialized query stage
    
    ### What changes were proposed in this pull request?
    
    Backport of #57347 (SPARK-57956) to `branch-4.1`. See also the `branch-4.2` 
backport, #60017.
    
    `LogicalQueryStage.maxRows` now uses the stage's `rowCount` only when the 
stage is materialized and its statistics are runtime statistics. Otherwise it 
falls back to `logicalPlan.maxRows`, so a cost estimate is never used as a hard 
upper bound by `EliminateLimits` during AQE re-optimization.
    
    The `LogicalQueryStage` change applies cleanly. `AdaptiveQueryExecSuite` 
needed a manual resolution of the import block only, because `master` imports 
names used by later tests that are not on `branch-4.1`, and the test needs 
`HashedRelationBroadcastMode` added to the `joins` import. The added test is 
identical to the one in #57347.
    
    ### Why are the changes needed?
    
    SPARK-57956 was merged into `master`, `branch-4.x` and `branch-4.3`, but 
not into `branch-4.2` or `branch-4.1`, which still have the old `maxRows`:
    
    ```scala
    override def maxRows: Option[Long] = 
stats.rowCount.map(_.min(Long.MaxValue).toLong)
    ```
    
    #59434 reports a wrong result on 4.1.3 from the same cause: with AQE on, a 
nested `LIMIT ... OFFSET` over a `LEFT JOIN` sometimes returns more rows than 
its outer `LIMIT`.
    
    ```sql
    SELECT x FROM (
      SELECT x FROM (
        SELECT p.id AS x FROM p LEFT JOIN (
          SELECT p.id AS anchor FROM p INNER JOIN k ON p.id = k.a INNER JOIN p 
AS q ON k.b = q.id
        ) o ON p.id = o.anchor
        LIMIT 6 OFFSET 2
      ) LIMIT 3 OFFSET 1
    )
    ```
    
    With the tables from #59434 (RDD-backed, `local[1]`, 
`spark.sql.shuffle.partitions=2`), running this query 200 times on `branch-4.1`:
    
    | | AQE on | AQE off |
    |---|---|---|
    | Before this PR | 3 rows: 182 runs, **5 rows: 18 runs** | 3 rows: 200 runs 
|
    | After this PR | 3 rows: 200 runs | 3 rows: 200 runs |
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. It fixes a correctness bug where, with AQE enabled, a query with a 
LIMIT over certain plans could return more rows than the LIMIT.
    
    ### How was this patch tested?
    
    - The new test `SPARK-57956: unmaterialized query stage exposes structural 
maxRows, not the estimate` passes.
    - The existing test `SPARK-36424: Support eliminate limits in AQE 
Optimizer` still passes.
    - The #59434 query above, run 200 times, as shown in the table.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #60018 from Raviston6296/SPARK-57956-4.1.
    
    Lead-authored-by: raviston <[email protected]>
    Co-authored-by: Vaibhav Garg <[email protected]>
    Signed-off-by: Cheng Pan <[email protected]>
    (cherry picked from commit 68af27f87349ed376c25f2c7bbb162c7f6b8a848)
    Signed-off-by: Cheng Pan <[email protected]>
---
 .../sql/execution/adaptive/LogicalQueryStage.scala | 16 +++++++-
 .../adaptive/AdaptiveQueryExecSuite.scala          | 44 ++++++++++++++++++++--
 2 files changed, 55 insertions(+), 5 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
index 506f52fd9072..4caa21833d36 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStage.scala
@@ -71,7 +71,21 @@ case class LogicalQueryStage(
     physicalStats.getOrElse(logicalPlan.stats)
   }
 
-  override def maxRows: Option[Long] = 
stats.rowCount.map(_.min(Long.MaxValue).toLong)
+  override def maxRows: Option[Long] = {
+    // A query stage's `rowCount` is an exact, valid upper bound only when it 
comes from the
+    // runtime statistics of a materialized stage. Checking `isMaterialized` 
alone is not enough:
+    // `computeStats()` can still fall back to `logicalPlan.stats` (a cost 
estimate, with
+    // `isRuntime = false`) when the physical-stage lookup yields no 
statistics, and that estimate
+    // can under-count (e.g., returning 0). Treating such an estimate as a 
hard upper bound lets
+    // rules such as EliminateLimits wrongly drop a LIMIT and change the 
result cardinality
+    // (SPARK-57956). Only trust `rowCount` when the stats are materialized 
runtime stats;
+    // otherwise fall back to `logicalPlan.maxRows`, which is always a sound 
upper bound.
+    if (isMaterialized && stats.isRuntime) {
+      stats.rowCount.map(_.min(Long.MaxValue).toLong)
+    } else {
+      logicalPlan.maxRows
+    }
+  }
 
   override def isMaterialized: Boolean = physicalPlan.exists {
     case s: QueryStageExec => s.isMaterialized
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index fb8fab6a80f8..a85b3963ce1c 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -30,9 +30,9 @@ import org.apache.spark.scheduler.{SparkListener, 
SparkListenerEvent, SparkListe
 import org.apache.spark.shuffle.sort.SortShuffleManager
 import org.apache.spark.sql.{DataFrame, Dataset, QueryTest, Row, SparkSession}
 import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference, Literal}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight, 
EliminateLimits}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, GlobalLimit, 
LeafNode, LogicalPlan, Statistics}
 import org.apache.spark.sql.classic.Strategy
 import org.apache.spark.sql.execution._
 import org.apache.spark.sql.execution.aggregate.BaseAggregateExec
@@ -41,7 +41,7 @@ import 
org.apache.spark.sql.execution.command.DataWritingCommandExec
 import org.apache.spark.sql.execution.datasources.noop.NoopDataSource
 import org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec
 import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, 
ENSURE_REQUIREMENTS, Exchange, REPARTITION_BY_COL, REPARTITION_BY_NUM, 
ReusedExchangeExec, ShuffleExchangeExec, ShuffleExchangeLike, ShuffleOrigin}
-import org.apache.spark.sql.execution.joins.{BaseJoinExec, 
BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, ShuffledHashJoinExec, 
ShuffledJoin, SortMergeJoinExec}
+import org.apache.spark.sql.execution.joins.{BaseJoinExec, 
BroadcastHashJoinExec, BroadcastNestedLoopJoinExec, 
HashedRelationBroadcastMode, ShuffledHashJoinExec, ShuffledJoin, 
SortMergeJoinExec}
 import org.apache.spark.sql.execution.metric.SQLShuffleReadMetricsReporter
 import 
org.apache.spark.sql.execution.ui.{SparkListenerSQLAdaptiveExecutionUpdate, 
SparkListenerSQLAdaptiveSQLMetricUpdates, SparkListenerSQLExecutionStart}
 import org.apache.spark.sql.execution.window.WindowExec
@@ -56,6 +56,16 @@ import org.apache.spark.tags.SlowSQLTest
 import org.apache.spark.util.ArrayImplicits._
 import org.apache.spark.util.Utils
 
+/**
+ * A leaf whose cost estimate under-counts its structural row bound. Used by 
SPARK-57956 to verify
+ * that an unmaterialized query stage exposes the structural `maxRows` instead 
of an estimate.
+ */
+private case class UnderCountLeaf(output: Seq[Attribute]) extends LeafNode {
+  override def maxRows: Option[Long] = Some(2L)
+  override def computeStats(): Statistics =
+    Statistics(sizeInBytes = BigInt(1), rowCount = Some(BigInt(0)))
+}
+
 @SlowSQLTest
 class AdaptiveQueryExecSuite
   extends QueryTest
@@ -2521,6 +2531,32 @@ class AdaptiveQueryExecSuite
     }
   }
 
+  test("SPARK-57956: unmaterialized query stage exposes structural maxRows, 
not the estimate") {
+    // Build a LogicalQueryStage whose underlying stage is never materialized. 
Its computeStats()
+    // falls back to the logical plan's cost estimate - here an under-count of 
0 rows - which must
+    // NOT be promoted to a hard maxRows bound. Otherwise EliminateLimits 
would drop a LIMIT that
+    // still needs to be applied once the stage runs (SPARK-57956).
+    val output = Seq(AttributeReference("a", IntegerType)())
+    val logical = UnderCountLeaf(output)
+    val scan = LocalTableScanExec(output, Nil, None)
+    val exchange = BroadcastExchangeExec(
+      HashedRelationBroadcastMode(output, isNullAware = false), scan)
+    val queryStage = LogicalQueryStage(logical, BroadcastQueryStageExec(0, 
exchange, exchange))
+
+    assert(!queryStage.isMaterialized)
+    // The under-counted estimate is what computeStats() surfaces...
+    assert(queryStage.stats.rowCount.contains(BigInt(0)))
+    // ...but maxRows must remain the structural bound (2), not the estimate 
(0).
+    assert(queryStage.maxRows.contains(2L),
+      "an unmaterialized stage must expose its structural maxRows, not the 
row-count estimate")
+
+    // Since the structural bound (2) exceeds the limit (1), the LIMIT must be 
retained. With the
+    // estimate wrongly promoted (maxRows = 0), EliminateLimits would instead 
drop it.
+    val limited = GlobalLimit(Literal(1), queryStage)
+    assert(EliminateLimits(limited).isInstanceOf[GlobalLimit],
+      "LIMIT must be retained when the child's structural row bound exceeds 
the limit")
+  }
+
   test("SPARK-48037: Fix SortShuffleWriter lacks shuffle write related metrics 
" +
     "resulting in potentially inaccurate data") {
     withTable("t3") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to