AnuragRaut08 opened a new pull request, #55611: URL: https://github.com/apache/arrow/pull/55611
### Rationale for this change Float16 dictionary unification and `count_distinct` can produce incorrect results because raw `uint16_t` storage values are interpreted as numeric values instead of Float16 bit patterns. For example, the Float16 bit pattern `0x3C00`, representing `1.0`, can be interpreted as `15360.0` when passed through the converting constructor. This causes dictionary values to be corrupted during dictionary unification and distinct Float16 values to be incorrectly merged by `count_distinct`. ### What changes are included in this PR? - Use `Float16::FromBits()` when constructing memo table keys from raw Float16 values in `DictionaryUnifierImpl::Unify`. - Fix Float16 handling in the array path of `CountDistinctImpl::Consume`. - Fix Float16 scalar unboxing in `UnboxScalar<HalfFloatType>::Unbox` to preserve the original bit representation. ### Are these changes tested? Added regression tests: - `TestDictionaryUnifier.HalfFloat`: verifies unified dictionary values and transpose maps. - `TestCountDistinctKernel.HalfFloat`: covers distinct Float16 values, duplicates, nulls, chunked arrays, and scalar input. Both regression tests were verified against the original implementation, where they reproduced the incorrect behavior. Validation performed: - `arrow-array-test` - `arrow-compute-aggregate-test` - `clang-format --dry-run --Werror` - `git diff --check` ### Are there any user-facing changes? Fixes incorrect Float16 values returned by dictionary unification and incorrect distinct counts for Float16 arrays. **This PR includes breaking changes to public APIs.** No. **This PR contains a "Critical Fix".** Yes. This fixes incorrect data produced by Float16 dictionary unification and `count_distinct`. ### Was AI used for this PR? **PR code and description written by:** - [x] Human - [ ] AI **Reviewed before submission by:** - [x] Human - [ ] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
