vb-dbrks opened a new pull request, #59259: URL: https://github.com/apache/spark/pull/59259
### What changes were proposed in this pull request? Use `AttributeMap` once per invocation of `LogicalRDD.rewriteConstraints` so constraint references are rewritten by expression ID even when their names or qualifiers differ. Preserve the existing fallback for unmapped attributes and the checks that associate source and destination outputs. Add regression coverage beside the existing LogicalRDD constraint tests in `DataFrameSuite`. ### Why are the changes needed? [SPARK-52878](https://issues.apache.org/jira/browse/SPARK-52878) reports a valid checkpoint/union pipeline failing with `NoSuchElementException`. With case-insensitive resolution and a temporary checkpoint directory: ```scala val first = spark.range(1).toDF("foo").checkpoint(eager) val second = first.where("Foo = 0").checkpoint(eager) first.unionByName(second).checkpoint(eager).collect() ``` The generic map compares complete attributes and can miss `Foo` even though it has the same expression ID as `foo`. A LogicalRDD can consequently retain stale constraint references after its output IDs change, causing Union's constraint lookup to fail. Rewriting by expression ID removes the stale references at their source. The affected operation was introduced by [SPARK-39834 / #37248](https://github.com/apache/spark/pull/37248) and extracted into the current helper by [SPARK-41539 / #39082](https://github.com/apache/spark/pull/39082). This is source-history evidence; the first release exhibiting this exact reproduction has not been bisected. ### Does this PR introduce _any_ user-facing change? Yes. The previously failing valid query succeeds and returns two zero-valued rows with either eager or lazy reliable checkpointing. The issue reports affected releases 3.5.6 and 4.0.0; this patch's reproduction was validated on the current master baseline. Strict case-sensitive resolution continues to reject `Foo` when only `foo` exists. There are no new public APIs, configuration settings, dependencies or persistence formats, and no migration action is required. ### How was this patch tested? Added tests for eager/lazy reliable checkpointing, expression-ID rewriting with differing names and qualifiers, `LogicalRDD.newInstance()`, same-name/different-ID and unmapped-attribute controls, and strict case-sensitive resolution. On fresh baseline `43ccc6baa6e55b250faddae0e0a25a431337309c`, the four bug regressions fail and the strict-resolution control passes. The checkpoint failures are `NoSuchElementException`; the two direct identity regressions retain the old expression IDs. Local validation used JDK 21 and the repository SBT launcher: ```sh ./build/sbt "sql/testOnly org.apache.spark.sql.DataFrameSuite -- -z SPARK-52878" ./build/sbt "sql/testOnly org.apache.spark.sql.DataFrameSuite org.apache.spark.sql.DatasetSuite org.apache.spark.sql.StatisticsCollectionSuite" ./build/sbt "sql/scalastyle" "sql/Test/scalastyle" ``` The five focused tests pass after the fix. The three complete suites pass all 448 executed tests, with one pre-existing ignored test (SPARK-19372). These runs include the existing SPARK-39834 constraint/statistics and SPARK-46794 subquery-exclusion coverage. Both SQL production and test scalastyle checks pass with zero errors or warnings. Initial launcher/dependency downloads failed before tests ran. Validation subsequently used the exact cached SBT launcher and an existing Maven mirror; all Spark code was freshly compiled from the stated baseline in a new worktree. No compiled Spark artifacts from other branches were reused. No fork CI success is claimed. No performance benchmark is included because this is a correctness fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (bundled CLI version 0.153.4). Codex assisted with investigation, patch and regression-test authoring, local validation, and preparation of this description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
