LuciferYang opened a new pull request, #1352:
URL: https://github.com/apache/paimon-rust/pull/1352

   ### Purpose
   
   Linked issue: close #1351
   
   `collect` ignored `fields.<col>.count-limit`, merged `aggReversed` in a 
different order from Java master, and checked `distinct` membership with a 
linear scan. This ports the merge logic of Java `FieldCollectAgg` as of 
apache/paimon#10607.
   
   ### Brief change log
   
   - The merge is a port of the private `agg(accumulator, 
accumulatorHasDuplicate, input, inputHasDuplicate)` of `FieldCollectAgg`. `agg` 
merges `(acc, false, input, true)` and `aggReversed` merges `(input, true, acc, 
false)`. A negative limit collects both operands backwards, input first, and 
reverses the result. For element types where Java uses the equaliser, 
`checkDuplicateUntil` decides how far back duplicates are checked.
   - `fields.<col>.count-limit` is read as a 32-bit integer and defaults to 
`i32::MAX`. A DELETE value is still stored as it is; the next merge applies the 
limit and the deduplication, like Java.
   - The ignore-retract wrapper passes `agg_reversed` to the wrapped function, 
as `FieldIgnoreRetractAgg` does since apache/paimon#10540. 
`agg_reversed_via_agg` had no other user and is removed.
   - Whether an element type uses the equaliser follows Java `needsEqualiser`, 
based on the Paimon type: ARRAY, VECTOR, MULTISET, MAP, ROW, BINARY and 
VARBINARY. Before, the choice followed the Arrow type, which put VARIANT and 
BLOB on the equaliser path and VECTOR on the set path.
   - With `distinct=true`, element types that Java collects into a 
`LinkedHashSet` are compared by their arrow-row encoded key in a hash set.
   
   Writing 20,000 single-element rows for one key with `distinct=true` (release 
build, in-memory FileIO): 8.7 s before, 7 ms after. I also replayed about 210k 
random `agg` / `aggReversed` / DELETE / retract steps whose results came from 
Java master's `FieldCollectAgg`, covering INT, ROW<INT> and BYTES elements, 
`distinct` on and off, limits from -5 to 5 plus the default, with and without 
ignore-retract. Rust matched every step. The same replay finds differences on 
main.
   
   ### Tests
   
   - `count_limit_matches_java_for_scalar_and_row_elements`: the cases of Java 
`testFieldCollectAggWithCountLimit`, for one row, two rows forward, and two 
rows reversed.
   - `agg_reversed_puts_older_input_first` and 
`ignore_retract_reversed_puts_older_input_first`: the older input comes first, 
with and without the ignore-retract wrapper.
   - `ignore_retract_reversed_deduplicates_older_binary_input`: with the 
wrapper delegating, an older binary input's own duplicates are removed (the old 
test expected them to be kept).
   - `agg_reversed_keeps_duplicates_of_a_binary_delete_value`: with the 
equaliser, a DELETE value is checked only against elements collected before it, 
so its own duplicates stay.
   - `delete_value_is_capped_by_the_next_merge`: a DELETE value keeps all its 
elements, and the next merge keeps the first or last N.
   - `delete_value_duplicates_follow_the_java_element_family`: after a DELETE 
value with a repeated element, VECTOR keeps both copies and VARIANT and BLOB 
keep one.
   - `count_limit_option_is_read_and_validated`: the option is read from the 
table options, and non-integer or out-of-range values are rejected.
   - `distinct_keys_follow_arrow_equality`: nulls, retract followed by agg, NaN 
and -0.0 behave as with Arrow `==`.
   
   ### API and Format
   
   No API or storage format change. Tables that set `count-limit` on a 
`collect` column now get capped arrays when Rust merges them.
   
   ### Documentation
   
   None.
   
   Generated-by: Claude Opus
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to