XiaoHongbo-Hope opened a new pull request, #9259: URL: https://github.com/apache/paimon/pull/9259
### Purpose Follow-up to #9253 (please review that one first; this PR contains its commits until it merges — the own diff is the last commit, +178 lines). #9253 inserts missing keys through a per-row byte-level rebuild (~20k rows/s). This PR adds `_root_insert_splice`, a vectorized fast path for the common case of inserting one key into every row's root OBJECT (e.g. `$.processed`): - appending a field only touches four spots of the object encoding (size, one id-table entry, one offset-table entry + sentinel, payload at the end of the data region), so each row is emitted as a join of byte slices instead of a rebuild; - eligibility is verified per plan group with numpy in one pass (uniform header/field count/id table, offsets in bounds, size/id/offset widths still sufficient); the group's first row is deep-validated; - rows failing any per-row check (e.g. crossing an offset-width boundary) fall back to the existing rebuild inside the same group; nested parents, multi-key inserts, and non-fixed replacements keep using the rebuild path; - combined with the existing in-place patch, "replace N doubles + insert one key" stays fully vectorized. Measured locally (50k rows, Python 3.11 / pyarrow 19): | scenario | without this PR | with this PR | per-row GenericVariant baseline | |---|---|---|---| | insert BOOL mark | 19.6k rows/s | 164k rows/s | 7.5k rows/s | | insert STRING mark | ~20k rows/s | 201k rows/s | 7.7k rows/s | | negate 4 DOUBLE + insert BOOL | 6.9k rows/s | 153k rows/s | 7.8k rows/s | ### Tests - re-enables the strict fast-path assertion (`_apply_edits` must not be called for uniform root inserts); - offset-width boundary test: rows crossing the 1-byte offset limit are rebuilt within the same plan group while the rest are spliced; - all `variant_set` semantics tests from #9253 pass unchanged on Python 3.6 / pyarrow 6.0.1 and Python 3.11 / pyarrow 19.0.1; output verified to round-trip through the Java `GenericVariant` reader (including binary search for 32+ field objects); flake8 clean. ### API and Format No API or format change; pure performance optimization. ### Documentation None needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
