mbyne opened a new pull request, #59557:
URL: https://github.com/apache/spark/pull/59557
### What changes were proposed in this pull request?
`LogicalRDD.newInstance()` and the `LogicalRDD.rewriteStatistics` /
`LogicalRDD.rewriteConstraints` helpers rewrite attribute references through a
plain `Map[Attribute, Attribute]`. `Attribute` equality includes the name, so a
reference that differs only cosmetically from the output attribute (for example
`Foo` vs `foo` under case-insensitive resolution) isn't found in the map, and
the old reference is left as-is.
This PR does the lookups by `exprId` instead, using `AttributeMap`:
- `newInstance()` builds the rewrite map as
`AttributeMap(output.zip(newOutput))` and uses `newOutput` directly as the new
output.
- `rewriteStatistics` and `rewriteConstraints` wrap the incoming
`colRewrite` in an `AttributeMap` before looking anything up.
### Why are the changes needed?
When a `LogicalRDD` is created from a checkpoint, its constraints are copied
from the optimized plan. If those constraints came from a case-insensitive
reference (for example `df.where("Foo == 0")` on a column named `foo`), the
rewrite misses them. They then point at the old exprIds, which no longer exist
in the plan. The next checkpoint of a plan built on top of that one then fails:
```scala
spark.conf.set("spark.sql.caseSensitive", "false")
val df1 = spark.range(1).toDF("foo").localCheckpoint()
val df2 = df1.where("Foo == 0").localCheckpoint()
df1.unionByName(df2).localCheckpoint()
// java.util.NoSuchElementException: key not found: Foo#1L
```
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix: checkpointing a Dataset built from earlier checkpoints that
used case-insensitive column references no longer fails. This affects released
versions (reported against 3.5.6 and 4.0.0), so it would be good to backport
the fix to the maintenance branches, including `branch-3.5`.
### How was this patch tested?
Added a test to `DatasetSuite`: `SPARK-52878: checkpoint of union of
checkpoints with case-insensitive references`.
```
build/sbt 'sql/testOnly *DatasetSuite -- -z "SPARK-52878"'
```
Without the fix in `ExistingRDD.scala`, the test fails with
`java.util.NoSuchElementException: key not found: Foo#1L`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
