akupchinskiy opened a new pull request, #7552: URL: https://github.com/apache/datafusion-comet/pull/7552
## Which issue does this PR close? Closes #7551. ## Rationale for this change `arr[i]` fails in Comet when `i` is not an INT, while Spark accepts any integer type there (for example `sort_array(collect_list(x))[floor(count(*) / 2)]`, since `floor` returns a BIGINT). ## What changes are included in this PR? If the index is not an INT, `CometGetArrayItem` casts it to INT before passing it to `ListExtract`. The cast is the legacy one even in ANSI mode, because Spark's conversion (`intValue()`) never fails: it keeps the low 32 bits of a BIGINT and widens TINYINT and SMALLINT, and the legacy cast does the same. INT indexes are unchanged. The bounds check still runs on the converted index, so in ANSI mode Comet reports the same out-of-range index as Spark. ## How are these changes tested? New cases in the `get_array_item.sql` and `get_array_item_ansi.sql` SQL file tests, compared with Spark: - TINYINT, SMALLINT and BIGINT indexes read from Parquet, with NULL indexes and NULL arrays - BIGINT values whose low 32 bits are a valid index (`4294967296` → 0, the BIGINT minimum → 0) or an invalid one (`2147483648`, `4294967295`, the BIGINT maximum → -1), the invalid ones giving NULL with ANSI off - with ANSI on, the `INVALID_ARRAY_INDEX` error reports index -1 for `4294967295`, like Spark - the median query above Both files fail without the change. With the change on top of main (891008e), all `expressions/array` SQL file tests and `CometArrayExpressionSuite` pass with the Spark 4.1 profile, and `spotless:check` is clean. Generated-by: Claude Code (Claude Opus 5.5) I reviewed the change and ran the tests myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
