snmvaughan opened a new pull request, #7263: URL: https://github.com/apache/datafusion-comet/pull/7263
## Which issue does this PR close? Backport of #7171 (squash commit 1dec4c058) to `branch-1.2`, for #7170. ## Rationale for this change Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates, change two results that Comet reproduces natively: - When `stop - start` overflows a long but the sequence is short, `sequence` returns its elements, where earlier releases raise `INTERNAL_ERROR` (SPARK-58821). - A string zero with a large exponent, such as `'0e39'`, casts to 0 instead of NULL, or `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI (SPARK-60118). Without this change, Comet 1.2 on those releases raises the internal error and returns NULL where Spark doesn't. See #7170 and #7171. ## What changes are included in this PR? A cherry-pick of #7171 with `-x`. The only conflicts were the import lists in `CometCast.scala` and `CometNativeCastSuite.scala`, where `main` also imports names from later PRs. The result keeps `branch-1.2`'s imports plus the ones #7171 adds. Everything else is identical to the source commit: - `SparkFixes`, which holds the version checks (4.0.5+, 4.1.4+, 4.2.1+ and 4.3+). - `spark_sequence_recomputed`, a native `sequence` variant that `CometSequence.nativeFunction` picks where SPARK-58821 is present. - The `decimal_zero_any_exponent` Cast proto field, which `CometCast` sets from `SparkFixes.castsZeroWithAnyExponentToDecimal`. - The tests, `SparkFixesSuite` in both PR build workflows, and the docs. Backports to other branches: `branch-1.1`, in a separate PR, adapted to that branch's `sequence` kernel. ## How are these changes tested? The tests come with the cherry-pick. On this branch, all 1289 `datafusion-comet-spark-expr` unit tests pass, `cargo fmt --check`, `cargo clippy --all-targets --workspace -- -D warnings` and spotless are clean, and the Scala code compiles, with `SparkFixesSuite` passing. I haven't run the other Scala suites locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
