JeonDaehong opened a new pull request, #1420: URL: https://github.com/apache/paimon-rust/pull/1420
### Purpose Linked issue: part of #1336 Reading a partial-update table builds a one-row `RecordBatch` for every merged key. `merge_unreduced` slices each selected column, adds one-row metadata arrays, and the reader then interleaves those batches. On a table with 2M keys written three times, a full read takes about 1.7 s, against about 0.5 s for deduplicate. ### Brief change log - Partial-update without field aggregators implements a new `MergeFunction::merge_picked`. It returns a live merged row as the input row each output column comes from (`PickedRow`). The sort-merge reader gathers these together with the rest of its output, in one `interleave` per column. NULL columns and `_SEQUENCE_NUMBER` / `_VALUE_KIND` come from one extra array per output batch. - Deletes, omitted keys, field aggregators and lone versions still take the existing `merge` path. So do other merge functions (including the lookup and audit-log wrappers) and the write buffer. PK table, 2M keys written 3 times, full read, in-memory FileIO, release build. I measured in one binary with `merge_picked` switched off by an env var, over 3 alternating rounds: - every round writes both value columns: 1698–1797 → 818–863 ms - rounds write different columns (`v` only, then `s` only): 1657–1729 → 775–837 ms ### Tests - `test_partial_update_picked_rows_match_materialized_rows`: three overlapping runs with NULL columns, a lone key and batch size 2. It reads them once through `merge_picked` and once through a wrapper that only has `merge`, and checks that the batches are equal and that `_SEQUENCE_NUMBER` is the latest version's. - The existing partial-update sort-merge tests (sequence groups, ignore-delete, delete baseline) now also go through the picked path. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). In my environment, two `vector_search` pickle tests in `bindings/python` `make test-rust` fail the same way on main. ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
