XiaoHongbo-Hope opened a new pull request, #9253: URL: https://github.com/apache/paimon/pull/9253
### Purpose Follow-up to #9147. `variant_replace` cannot insert a missing key, so adding an idempotency mark to a VARIANT OBJECT still required per-row full `GenericVariant` decode/re-encode (~7.8k rows/s vs ~630k rows/s for vectorized replace). This PR adds a public `variant_set(column, path, value)` API to `pypaimon.data`: - an existing path is replaced exactly like `variant_replace` (same exact-type checks, same vectorized fast path, no regression); - a missing final key is inserted when its parent path exists and is an OBJECT, reusing an existing metadata key id or appending a new key to the metadata dictionary; - supports single path or path mappings, `pa.Scalar` broadcast and per-row `pa.Array` / `pa.ChunkedArray` values, `binary`/`large_binary` layouts, sliced arrays, and preserves chunk boundaries; - SQL NULL rows stay SQL NULL and Arrow NULL values are encoded as VARIANT NULL; - missing intermediate paths, non-OBJECT parents, missing array indices, and overlapping/duplicate paths raise `ValueError`; it never creates intermediate objects or touches arrays. Implementation highlights: - group-level planning caches metadata parsing and rebuild plans per identical metadata/layout (`_metadata_with_keys`, `_cached_metadata_key_ids`); - a vectorized splice fast path inserts one field into uniform root objects without per-row decoding (id table, offset table, size, and metadata field id are updated in place; field tables stay name-sorted so the Java `GenericVariant#getFieldByKey` binary search keeps working); - non-uniform rows fall back to a single-pass byte-level rebuild (`_apply_edits`), never to full `GenericVariant` decode/encode; - the group planner now also proves group-wide key absence, which keeps replace/get vectorized for that case too. Benchmark (`python -m pypaimon.benchmark.variant_set_bench`, 50k rows, no CI wall-clock assertions): | scenario | variant_set | per-row GenericVariant baseline | |---|---|---| | negate 4 existing DOUBLE paths | 631k rows/s (replace: 632k) | 7.8k rows/s | | insert BOOL mark | 164k rows/s | 7.5k rows/s | | insert STRING mark | 201k rows/s | 7.7k rows/s | | negate 4 + insert BOOL | 153k rows/s | 7.8k rows/s | ### Tests - `pypaimon/tests/variant_set_test.py` (39 cases): replace/insert parity with `variant_replace`, metadata with/without the inserted key, mixed metadata/field-order/presence within one chunk, multi-path partial replace + insert, Scalar/Array/ChunkedArray, SQL NULL vs VARIANT NULL vs missing path, sliced/large_binary/multi-chunk inputs, offset-width boundary fallback, low-memory 32-bit offset overflow guard, input immutability, mocks asserting no full `GenericVariant` decode/encode and no per-row planning on uniform data, and Java interop golden bytes generated by `GenericVariantBuilder`. - Java compatibility verified both ways: Java `GenericVariant#toJson`/`getFieldByKey` (including the binary-search path for 32+ field objects) reads Python `variant_set` output; Python reads Java-built bytes (kept as a golden fixture). - All existing VARIANT tests pass on Python 3.6 / pyarrow 6.0.1 and Python 3.11 / pyarrow 19.0.1; flake8 clean. ### API and Format Adds public `pypaimon.data.variant_set`; no storage format change (output stays spec-conformant Variant binary readable by Java Paimon). ### Documentation `docs/docs/pypaimon/python-api.mdx` documents the upsert semantics, supported inputs, and limitations (no intermediate-object creation, no array insertion, no removal). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
