andygrove opened a new pull request, #5480: URL: https://github.com/apache/datafusion-comet/pull/5480
## Which issue does this PR close? Backports five bug fixes from `main` to `branch-1.0` for the 1.0.1 patch release. Closes #5334 on `branch-1.0`. ## Rationale for this change `branch-1.0` diverged from `main` at #5240, and 79 commits have landed on `main` since. Four fixes were already backported in #5321. Of the remaining `fix:` commits, these five address correctness bugs or outright job failures that are present in 1.0.0, and none of them depend on post-1.0 features. | Backported | Fixes | Impact | | --- | --- | --- | | #5185 | #5334 (priority:critical) | Boolean-to-decimal cast produces an invalid `Decimal128` when `10^scale` does not fit the target precision. | | #5376 | #5270 (priority:critical) | `arrays_overlap` compares raw IEEE 754 bit patterns on flat float arrays, so Comet silently returns a different answer than Spark when a `NaN` is present. | | #5364 | Part of #5149 (priority:critical) | `to_time` / `try_to_time` do not match Spark's whitespace trimming. | | #5392 | #5391 | Native shuffle fails with a 2GB task serialization OOM on jobs with very many partitions. | | #5443 | #5208 | Native `make_date` rejects years that Spark accepts. | #5185 is titled `feat:` on `main` because it removes the native cast rather than repairing it, but it is the fix for a critical correctness bug and belongs in a patch release. ## What changes are included in this PR? Five clean cherry-picks, in `main` order, with original authorship and `(cherry picked from commit ...)` trailers preserved. No changes were needed to adapt any of them to `branch-1.0`. ### Deliberately excluded - **#5461** (release native shuffle reservation after spill failure) cherry-picks cleanly but does **not compile** on `branch-1.0`: its regression test references `spill_accounted_input_buffers`, `repeated_spill_buffer_bytes`, and `metrics.memory_spilled_bytes`, all introduced by #5370. Backporting it would mean dragging #5370 along or rewriting the test. - **#5410** is not applicable. It narrows a predicate introduced by #5298 (Iceberg V2 write fall-back detection), which is not on `branch-1.0`. - **#5234, #5225, #5045, #5359** apply cleanly but are labelled `enhancement` upstream, not bug fixes. - **#5370, #5445, #5463, #5408, #5398, #5464** correct Spark UI metrics rather than query results. A patch release is not the place for observability behaviour changes with no correctness gain, and #5445 conflicts across five files. - **#5369** is a real bug, but `spark.comet.parquet.write.enabled` defaults to `false` on `branch-1.0` and is documented as highly experimental and not for production, so no 1.0.0 user is exposed. ### Still open for 1.0.1 Two milestoned correctness issues have no fix on `main` yet, so there is nothing to backport for them: - #5366 (priority:critical) — #5364 fixes only the whitespace half of `to_time`. The native parser still rejects `'T12'` and `'12:30:45.'`, which Spark accepts. - #4412 (priority:high) — `CometHashAggregateExec` does not participate in `AQEPropagateEmptyRelation`. ## How are these changes tested? Each fix carries its own regression tests, which came across with the cherry-picks. Verified locally on this branch: - `cargo check --workspace --all-targets` — clean - `cargo test -p datafusion-comet-spark-expr` — 610 passed - `cargo test -p datafusion-comet-shuffle` — 31 passed - `./mvnw test-compile` — BUILD SUCCESS - `./mvnw spotless:check scalastyle:check` — clean for the parent, `comet-common`, and `comet-spark` modules - `CometSqlFileTestSuite` — 436 passed, covering all six new/changed `.sql` fixtures - `CometCastSuite` — 168 passed, 8 ignored - `CometArrayExpressionSuite` — 46 passed - `CometNativeShuffleSuite` — 28 passed - `CometNativeShuffleInputRDDSuite` — 1 passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
