akupchinskiy commented on issue #16550:
URL: https://github.com/apache/datafusion/issues/16550#issuecomment-6636031007
I reproduced this on `main` (238ae32) and on 55.0.0 with a dictionary column
that isn't a
partition column and has only three distinct values, so it looks like a
general problem with
dictionary columns in `first_value`/`last_value`. `DISTINCT ON` fails too,
because it's rewritten
to `first_value`.
```sql
-- 40,000 groups; the dictionary column has only 3 distinct values
COPY (SELECT value AS id, value % 40000 AS k,
arrow_cast(CASE value % 3 WHEN 0 THEN 'a' WHEN 1 THEN 'b' ELSE
'c' END,
'Dictionary(Int16, Utf8)') AS d
FROM generate_series(0, 79999)) TO 'm.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'm.parquet';
SET datafusion.execution.target_partitions = 1;
SELECT count(DISTINCT d) FROM (SELECT k, min(d) AS d FROM t GROUP BY k);
-- ok
SELECT count(DISTINCT d) FROM (SELECT k, first_value(d ORDER BY id) AS d
FROM t GROUP BY k); -- error
SELECT count(DISTINCT d) FROM (SELECT k, last_value(d ORDER BY id) AS d FROM
t GROUP BY k); -- error
SELECT count(DISTINCT d) FROM (SELECT DISTINCT ON (k) k, d FROM t ORDER BY
k, id); -- error
```
```
Internal error: Can not create index of type Int16 from value 32768.
```
With 32,768 groups it passes. With 10 partitions it passes as well in this
repro, because each
aggregation instance then holds fewer groups.
As far as I can tell, the path is the following:
- Dictionary types aren't supported by `FirstLastGroupsAccumulator`, so the
aggregate runs through
`GroupsAccumulatorAdapter` with a
`FirstValueAccumulator`/`LastValueAccumulator` per group.
- The resulting dictionary scalars, from `evaluate()` and also from
`state()` for partial
aggregation, go through `ScalarValue::iter_to_array`.
- For dictionaries that collects the values one per row and calls
`dict_from_values`, which gives
row `i` the key `i`.
Repeated values get their own keys, so the non-null row that needs key
32,768 fails for `Int16`;
for `UInt16` in your case it's key 65,536. `min`/`max` coerce dictionary
inputs to the value type,
which is why they work.
A fix probably belongs in how `iter_to_array` builds dictionaries:
deduplicate the values, for
example by casting the values array to the dictionary type, which Arrow does
for `Utf8`. That
changes the dictionary layout for other callers, though, and not every value
type casts that way,
so it needs a look at the other users. The workaround from the description,
`first_value(CAST(col AS VARCHAR))`, works here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
