akupchinskiy opened a new issue, #7551:
URL: https://github.com/apache/datafusion-comet/issues/7551
### Describe the bug
Reading an array element with a non-INT index fails in Comet. Spark accepts
any integer type as the index (TINYINT, SMALLINT, BIGINT) and converts it to an
int, but Comet errors out:
```
org.apache.comet.CometNativeException: could not cast array of type Int64 to
arrow_array::array::primitive_array::PrimitiveArray<arrow_array::types::Int32Type>.
```
I ran into it with a simple median, where the index is a BIGINT without any
explicit cast (`floor` of a DOUBLE returns BIGINT):
```sql
SELECT endpoint,
sort_array(collect_list(latency_ms))[floor(count(*) / 2)] AS median_ms
FROM requests
GROUP BY endpoint
```
Plain Spark 4.1.4 returns the medians, Comet 1.2.0 fails. It still happens
on main (891008e).
### Steps to reproduce
```sql
CREATE TABLE t(arr array<int>, i bigint) USING parquet;
INSERT INTO t VALUES (array(10, 20, 30), 1);
SELECT arr[i] FROM t;
```
### Expected behavior
The same result as Spark. Spark keeps the low 32 bits of a BIGINT index, so
even `arr[4294967296]` is `arr[0]` there.
### Additional context
`CometGetArrayItem` passes the index to `ListExtract` as is, and
`ListExtract` only reads Int32. Casting a non-INT index to INT in the serde
fixes it; I'll open a PR.
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
