LuciferYang opened a new issue, #1351: URL: https://github.com/apache/paimon-rust/issues/1351
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Please describe the bug 🐞 The `collect` aggregator differs from Java `FieldCollectAgg` on current master in three ways. 1. `fields.<col>.count-limit` (apache/paimon#10540) is accepted as a table option but ignored. A table with `count-limit=3` keeps every element when Rust merges or compacts it, while Java keeps the first 3 (or the last 3 for `-3`). 2. Java `aggReversed(acc, input)` is `agg(input, true, acc, false)` since apache/paimon#10607: the older input goes first. Rust appends it after the accumulator, so a partial-update table with a sequence group gets a different ARRAY order from Rust and Java for the same rows. Java's `FieldIgnoreRetractAgg` also delegates `aggReversed` to the wrapped function now, and Rust still uses the default `agg(input, acc)` path there. 3. With `distinct=true`, Rust checks each element against every collected element with Arrow `==`. Java uses a `LinkedHashSet` for non-binary scalar elements. Writing 20,000 single-element rows for one key with `distinct=true` takes 8.7 s on main (release build, in-memory FileIO). ### Solution Port the private `agg(accumulator, accumulatorHasDuplicate, input, inputHasDuplicate)` of `FieldCollectAgg`, including `checkDuplicateUntil` and the reversed collection for negative limits, read `count-limit` like Java's `intType` option, delegate `aggReversed` in the ignore-retract wrapper, and look up distinct elements that Java puts in a `LinkedHashSet` by their arrow-row encoded key. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
