AnuragRaut08 opened a new pull request, #55611:
URL: https://github.com/apache/arrow/pull/55611

   ### Rationale for this change
   
   Float16 dictionary unification and `count_distinct` can produce incorrect 
results because raw `uint16_t` storage values are interpreted as numeric values 
instead of Float16 bit patterns.
   
   For example, the Float16 bit pattern `0x3C00`, representing `1.0`, can be 
interpreted as `15360.0` when passed through the converting constructor.
   
   This causes dictionary values to be corrupted during dictionary unification 
and distinct Float16 values to be incorrectly merged by `count_distinct`.
   
   ### What changes are included in this PR?
   
   - Use `Float16::FromBits()` when constructing memo table keys from raw 
Float16 values in `DictionaryUnifierImpl::Unify`.
   - Fix Float16 handling in the array path of `CountDistinctImpl::Consume`.
   - Fix Float16 scalar unboxing in `UnboxScalar<HalfFloatType>::Unbox` to 
preserve the original bit representation.
   
   ### Are these changes tested?
   
   Added regression tests:
   
   - `TestDictionaryUnifier.HalfFloat`: verifies unified dictionary values and 
transpose maps.
   - `TestCountDistinctKernel.HalfFloat`: covers distinct Float16 values, 
duplicates, nulls, chunked arrays, and scalar input.
   
   Both regression tests were verified against the original implementation, 
where they reproduced the incorrect behavior.
   
   Validation performed:
   
   - `arrow-array-test`
   - `arrow-compute-aggregate-test`
   - `clang-format --dry-run --Werror`
   - `git diff --check`
   
   ### Are there any user-facing changes?
   
   Fixes incorrect Float16 values returned by dictionary unification and 
incorrect distinct counts for Float16 arrays.
   
   **This PR includes breaking changes to public APIs.**
   
   No.
   
   **This PR contains a "Critical Fix".**
   
   Yes. This fixes incorrect data produced by Float16 dictionary unification 
and `count_distinct`.
   
   ### Was AI used for this PR?
   
   **PR code and description written by:**
   
   - [x] Human
   - [ ] AI
   
   **Reviewed before submission by:**
   
   - [x] Human
   - [ ] AI
   - [ ] Not reviewed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to