AnuragRaut08 opened a new pull request, #55610:
URL: https://github.com/apache/arrow/pull/55610

   ### Rationale for this change
   
   Float16 dictionary unification and `count_distinct` produce incorrect 
results because raw `uint16_t` storage values are interpreted as numeric values 
instead of Float16 bit patterns.
   
   For example, the Float16 bit pattern `0x3C00`, representing `1.0`, is 
interpreted as `15360.0` when passed through the converting constructor.
   
   This causes dictionary values to be corrupted during dictionary unification 
and distinct Float16 values to be incorrectly merged by `count_distinct`.
   
   ### What changes are included in this PR?
   
   - Use `Float16::FromBits()` when constructing memo table keys from raw 
Float16 values in `DictionaryUnifierImpl::Unify`.
   - Fix Float16 handling in the array path of `CountDistinctImpl::Consume`.
   - Fix Float16 scalar unboxing to preserve the original bit representation in 
`UnboxScalar<HalfFloatType>::Unbox`.
   
   ### Are these changes tested?
   
   Added regression tests:
   
   - `TestDictionaryUnifier.HalfFloat`: verifies unified dictionary values and 
transpose maps.
   - `TestCountDistinctKernel.HalfFloat`: covers distinct Float16 values, 
duplicates, nulls, chunked arrays, and scalar input.
   
   Verified that both new regression tests fail with the original 
implementation and pass with the fix.
   
   Validation performed:
   
   - `arrow-array-test`: 346 tests passed (2 disabled).
   - `arrow-compute-aggregate-test`: 346 tests passed.
   - `clang-format --dry-run --Werror`: passed.
   - `git diff --check`: passed.
   
   ### Are there any user-facing changes?
   
   Fixes incorrect Float16 values returned by dictionary unification and 
incorrect distinct counts for Float16 arrays.
   
   ### Critical Fix
   
   This PR fixes a bug that produces incorrect data. Float16 dictionary values 
could be corrupted during unification, and `count_distinct` could undercount 
distinct Float16 values.
   
   ### Was AI used for this PR?
   
   **PR code and description written by:**
   
   - [ ] Human
   - [x] AI
   
   **Reviewed before submission by:**
   
   - [ ] Human
   - [x] AI
   - [ ] Not reviewed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to