JeonDaehong opened a new pull request, #1420:
URL: https://github.com/apache/paimon-rust/pull/1420

   ### Purpose
   
   Linked issue: part of #1336
   
   Reading a partial-update table builds a one-row `RecordBatch` for every 
merged key. `merge_unreduced` slices each selected column, adds one-row 
metadata arrays, and the reader then interleaves those batches. On a table with 
2M keys written three times, a full read takes about 1.7 s, against about 0.5 s 
for deduplicate.
   
   ### Brief change log
   
   - Partial-update without field aggregators implements a new 
`MergeFunction::merge_picked`. It returns a live merged row as the input row 
each output column comes from (`PickedRow`). The sort-merge reader gathers 
these together with the rest of its output, in one `interleave` per column. 
NULL columns and `_SEQUENCE_NUMBER` / `_VALUE_KIND` come from one extra array 
per output batch.
   - Deletes, omitted keys, field aggregators and lone versions still take the 
existing `merge` path. So do other merge functions (including the lookup and 
audit-log wrappers) and the write buffer.
   
   PK table, 2M keys written 3 times, full read, in-memory FileIO, release 
build. I measured in one binary with `merge_picked` switched off by an env var, 
over 3 alternating rounds:
   - every round writes both value columns: 1698–1797 → 818–863 ms
   - rounds write different columns (`v` only, then `s` only): 1657–1729 → 
775–837 ms
   
   ### Tests
   
   - `test_partial_update_picked_rows_match_materialized_rows`: three 
overlapping runs with NULL columns, a lone key and batch size 2. It reads them 
once through `merge_picked` and once through a wrapper that only has `merge`, 
and checks that the batches are equal and that `_SEQUENCE_NUMBER` is the latest 
version's.
   - The existing partial-update sort-merge tests (sequence groups, 
ignore-delete, delete baseline) now also go through the picked path.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go). In my environment, two `vector_search` 
pickle tests in `bindings/python` `make test-rust` fail the same way on main.
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to