LuciferYang opened a new issue, #11109:
URL: https://github.com/apache/paimon/issues/11109

   **Search before asking**
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   **Paimon version**
   master @ eb0c10562e
   
   **Compute Engine**
   Engine-agnostic Java code generation.
   
   **Minimal reproduce step**
   Generate a `RecordEqualiser` for `ARRAY<MAP<INT, INT>>` and compare `[{1:10, 
2:20}]` with `[{2:20, 1:10}]`. Two `GenericArray` values compare equal, because 
the map comparison matches entries by key. Two `BinaryArray` values compare 
unequal, because `generateArrayComparison` compares them by bytes when the 
element type has no FLOAT/DOUBLE. The `BinaryRow` fast path in 
`EqualiserCodeGenerator` does the same for a top-level `MAP` or `MULTISET` 
field.
   
   A user-visible case: a `collect` field with `distinct=true` and 
`count-limit=2` on `ARRAY<ARRAY<MAP<INT, INT>>>`, written with rows `[{1:10, 
2:20}]`, `[{2:20, 1:10}]`, `[{3:30}]` for one key. The write buffer passes 
binary arrays to `FieldCollectAgg`, so the first two count as different 
elements and the result is the first two rows. Merging the same rows as generic 
arrays keeps the first and the third.
   
   **What doesn't meet your expectations?**
   The result of the generated equaliser should not depend on whether a value 
is binary or generic. Maps should match by key in every representation, as they 
already do for top-level `MAP` fields read from columnar or generic rows.
   
   **Anything else?**
   #8534 disabled the same byte shortcuts for FLOAT/DOUBLE for the same reason. 
`count-limit` for `collect` (#10540) is not in a release yet. This was found 
while reviewing apache/paimon-rust#1352, which ports `count-limit` and has to 
pick one of the two behaviors.
   
   **Are you willing to submit a PR?**
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to