This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new f750be2f6fc4 [SPARK-61088][SQL] Fix wrong results by stale 
partitioning and constraints when a checkpointed Dataset is self-joined
f750be2f6fc4 is described below

commit f750be2f6fc429870fff2a6fa9b0ce85e0c0c91f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 7 12:09:40 2026 -0800

    [SPARK-61088][SQL] Fix wrong results by stale partitioning and constraints 
when a checkpointed Dataset is self-joined
    
    ### What changes were proposed in this pull request?
    
    This PR fixes a correctness issue: a self-join of a checkpointed Dataset 
can return wrong results.
    
    `LogicalRDD.newInstance` re-issues the expression IDs of `output` and 
rewrites `outputPartitioning`, `outputOrdering`, `originStats` and 
`originConstraints` through a map from the old output attributes to the new 
ones. The map was built with `Utils.toMap`, so a lookup uses 
`AttributeReference.equals`, which also compares the qualifier and the 
nullability. However, `LogicalRDD.fromDataset` takes `output` from the analyzed 
plan, the partitioning and the ordering from the executed plan,  [...]
    
    This PR builds the map with `AttributeMap`, which looks up attributes by 
expression ID, like `InMemoryRelation.withOutput` already does. For `output`, 
it keeps each attribute and swaps only its expression ID, like 
`CTERelationRef.newInstance` does since SPARK-54718, so that the output columns 
sharing an expression ID keep their own names.
    
    ### Why are the changes needed?
    
    When a checkpointed Dataset is joined with itself, `DeduplicateRelations` 
calls `LogicalRDD.newInstance` on one copy. The renewed copy then reports:
    
    1. A partitioning over the column of the other copy. When a broadcast hash 
join streams the renewed copy, it passes this partitioning up, and an aggregate 
grouped by the column of the other copy skips its shuffle. The aggregate 
returns duplicated groups with split counts. This needs AQE to be disabled when 
the Dataset is checkpointed, because `AdaptiveSparkPlanExec` reports 
`UnknownPartitioning`.
    2. Constraints over the column of the other copy. For a filter over a 
`RIGHT OUTER` join whose right side is the renewed copy, `EliminateOuterJoin` 
reads such a constraint as a null-filtering predicate on the left side and 
turns the join into an `INNER` join, so the null-extended rows are lost. This 
happens with the default configurations.
    3. An ordering over the column of the other copy, which can make an 
operator skip its `Sort`, e.g., a window over a self-join of a single-partition 
checkpoint.
    
    ```scala
    // Issue 1: It should return 10, but returns 50.
    spark.conf.set("spark.sql.adaptive.enabled", "false")
    val cp1 = spark.range(10).repartition($"id").as("s").localCheckpoint()
    broadcast(cp1.as("x")).join(cp1.as("y"), $"x.id" % 2 === $"y.id" % 
2).groupBy($"x.id").count().count()
    
    // Issue 2: It should return 4, but returns 2.
    spark.conf.unset("spark.sql.adaptive.enabled")
    val cp2 = spark.range(10).as("t").where($"t.id" > 
5).as("s").localCheckpoint()
    cp2.as("x").join(cp2.as("y"), $"x.id" === $"y.id" + 2, 
"right_outer").where($"y.id" > 0).count()
    ```
    
    In both cases, the attributes of the partitioning or the constraints have a 
different qualifier (none or `t`) from the output attribute (`s`). A 
nullability difference triggers the same issue, e.g., when `EliminateOuterJoin` 
turns a `FULL OUTER` join into a `LEFT OUTER` join, the partitioning is over 
the non-nullable join key while the analyzed output is nullable.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, this fixes wrong results. The partitioning and ordering issues exist 
since Apache Spark 2.1.0 
([v2.1.0](https://github.com/apache/spark/releases/tag/v2.1.0), 2016-12-15, 
SPARK-17972) and the constraints issue exists since Apache Spark 3.4.0 
([v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0), 2023-04-07, 
SPARK-39834).
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case, which fails without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60197 from dongjoon-hyun/SPARK-61088.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    
    (cherry picked from commit 2af0404f3eeea8e1d09fa428c7363c388655c184)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../apache/spark/sql/execution/ExistingRDD.scala    |  8 +++++---
 .../scala/org/apache/spark/sql/DatasetSuite.scala   | 21 +++++++++++++++++++++
 2 files changed, 26 insertions(+), 3 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
index 06085497de19..d28ad0069b19 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
@@ -30,7 +30,6 @@ import org.apache.spark.sql.catalyst.util.truncatedString
 import org.apache.spark.sql.classic.{Dataset, SparkSession}
 import org.apache.spark.sql.connector.read.streaming.SparkDataStream
 import org.apache.spark.sql.execution.metric.SQLMetrics
-import org.apache.spark.util.collection.Utils
 
 object ExternalRDD {
 
@@ -116,7 +115,9 @@ case class LogicalRDD(
     session :: originStats :: originConstraints :: Nil
 
   override def newInstance(): LogicalRDD.this.type = {
-    val rewrite = Utils.toMap(output, output.map(_.newInstance()))
+    // Keyed by exprId, because the attributes in the partitioning, the 
ordering, the stats and
+    // the constraints can differ from `output` in qualifier or nullability.
+    val rewrite = AttributeMap(output.zip(output.map(_.newInstance())))
 
     val rewrittenPartitioning = outputPartitioning match {
       case p: Expression =>
@@ -135,7 +136,8 @@ case class LogicalRDD(
     val rewrittenConstraints = originConstraints.map(rewriteConstraints(_, 
rewrite))
 
     LogicalRDD(
-      output.map(rewrite),
+      // Swap only the exprId, so that the columns sharing an exprId keep 
their own names.
+      output.map(a => a.withExprId(rewrite(a).exprId)),
       rdd,
       rewrittenPartitioning,
       rewrittenOrdering,
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
index 0e0f454606af..11de5a364989 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
@@ -1852,6 +1852,27 @@ class DatasetSuite extends QueryTest
     }
   }
 
+  test("SPARK-61088: self-join of a checkpointed Dataset should not use the 
partitioning, " +
+      "ordering and constraints of the other copy") {
+    // The partitioning, the ordering and the constraints refer to `t.id`, 
while the output is
+    // `s.id`. With AQE, the checkpoint would not keep the partitioning and 
the ordering.
+    val cp = withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") {
+      spark.range(16).as("t").where($"t.id" > 5).repartition($"t.id")
+        .sortWithinPartitions($"t.id").as("s").localCheckpoint()
+    }
+    val renewed = cp.logicalPlan.asInstanceOf[LogicalRDD].newInstance()
+    assert(renewed.outputOrdering.map(_.references) === Seq(renewed.outputSet))
+    val (x, y) = (cp.as("x"), cp.as("y"))
+    val agg = broadcast(x).join(y, $"x.id" % 2 === $"y.id" % 
2).groupBy($"x.id").count()
+    checkAnswer(agg, (6L until 16L).map(Row(_, 5L)))
+    val outer = x.join(y, $"x.id" === $"y.id" + 1, 
"right_outer").where($"y.id" > 0)
+    checkAnswer(outer, (6L until 15L).map(i => Row(i + 1, i)) :+ Row(null, 
15L))
+
+    // The output columns sharing an exprId keep their names.
+    val cp2 = spark.range(3).select($"id", $"ID").localCheckpoint()
+    assert(cp2.join(cp2).columns === Array("id", "ID", "id", "ID"))
+  }
+
   test("Dataset().localCheckpoint() lazy with StorageLevel") {
     val df = spark.range(10).repartition($"id" % 2)
     val checkpointedDf = df.localCheckpoint(eager = false, 
StorageLevel.DISK_ONLY)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to