ViderStar opened a new pull request, #55253:
URL: https://github.com/apache/arrow/pull/55253
### Rationale for this change
`unique` and `value_counts` on a chunked dictionary array whose chunks have
different dictionaries unify the dictionaries and transpose the indices, but
the output keeps the input index type. When the unified dictionary has more
entries than that index type can address, the transposed indices wrap around
and the result is an invalid array (`ValidateFull()` fails). The example from
the issue still reproduces on pyarrow 26.0.0: two `int8`-indexed chunks with 64
and 65 distinct values give indices `[63, -128]`.
### What changes are included in this PR?
`DictionaryHashKernel` now builds the unified dictionary with
`DictionaryUnifier::GetResultWithIndexType()` using the input index type,
instead of `GetResult()`, which picks a possibly wider index type that the
output never uses. If the unified dictionary does not fit, the kernel returns
the same `Invalid` error that the unifier already produces in other places
("The unified dictionary requires a larger index type").
Widening the output index type would change the declared output type of
`unique` (`FirstType`), so I went with an error here.
### Are these changes tested?
Yes. Added `TestHashKernel.DictionaryUniqueAndValueCountsIndexOverflow`,
which checks a case that just fits in `int8` (127 unified entries, result
validates) and one that does not (129 entries, `Invalid`). The test fails
without the fix. `arrow-compute-vector-test` passes locally (macOS arm64,
Debug).
### Are there any user-facing changes?
`unique` / `value_counts` now return an error instead of an invalid
dictionary array in this case.
**This PR contains a "Critical Fix".** It fixes a bug that produced invalid
data (out-of-range dictionary indices).
### Was AI used for this PR?
In accordance to the [AI generation
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [ ] Human
- [x] AI
**Reviewed before submission by:**
- [ ] Human
- [x] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
