snmvaughan opened a new issue, #7170: URL: https://github.com/apache/datafusion-comet/issues/7170
### Describe the bug Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates (4.0.5-rc3, 4.1.4-rc4 and 4.2.1-rc3), include two fixes that change results Comet reproduces natively. Comet still returns the old results: - [SPARK-58821](https://issues.apache.org/jira/browse/SPARK-58821) (apache/spark#58047): when `stop - start` overflows a long but the sequence is short, `sequence` returns its elements. Earlier releases raise `INTERNAL_ERROR("Unreachable code reached.")`, and Comet still does. - [SPARK-60118](https://issues.apache.org/jira/browse/SPARK-60118) (apache/spark#59334): casting a string to decimal no longer applies the integral-digit limit to zero, so a zero with a large exponent, such as `'0e79'`, casts to 0. Earlier releases return NULL (or raise `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI), and Comet still does. ### Steps to reproduce On a Spark build with these fixes (branch-4.0, branch-4.1, branch-4.2 or the release candidates above), with Comet enabled: ```sql CREATE TABLE seq_t (start BIGINT, stop BIGINT, step BIGINT) USING parquet; INSERT INTO seq_t VALUES (-9223372036854775808L, 9223372036854775807L, 9223372036854775807L); SELECT sequence(start, stop, step) FROM seq_t; -- Spark: [-9223372036854775808, -1, 9223372036854775806] -- Comet: fails with INTERNAL_ERROR ("Unreachable code reached.") CREATE TABLE dec_t (v STRING) USING parquet; INSERT INTO dec_t VALUES ('0e79'); SET spark.sql.ansi.enabled=false; SELECT CAST(v AS DECIMAL(10, 2)) FROM dec_t; -- Spark: 0.00 -- Comet: NULL ``` The values come from tables so that the optimizer doesn't fold the expressions before Comet sees them. With ANSI enabled, the cast returns 0.00 in Spark and raises `NUMERIC_OUT_OF_SUPPORTED_RANGE` in Comet. ### Expected behavior Comet matches Spark on each version: the new results on 4.0.5+, 4.1.4+ and 4.2.1+, and the old results on earlier releases. ### Additional context Neither fix leaves anything in the plan for Comet to detect, so Comet has to choose the behavior by Spark version, as it already does for SPARK-59602 in `ArraySetSupport.normalizesFloats`. Against builds with these fixes, the `expect_error(Unreachable code reached)` case in `sequence.sql` fails, and so do the `CometNativeCastSuite` string-to-decimal tests whose inputs include such zeros (`check 0 mantissa` and the generated-value tests). ### Willingness to contribute I can contribute a fix for this bug independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
