JingsongLi opened a new pull request, #10382: URL: https://github.com/apache/paimon/pull/10382
### Purpose Enable Native MAP shared-shredding writes and updates using the Rust core implementation in https://github.com/apache/paimon-rust/pull/1027. **Merge dependency:** merge apache/paimon-rust#1027 first, then rerun Native CI. This PR is a draft until that dependency reaches Rust main. CI continues to install upstream Rust main; it is not pinned to a fork or branch. - Remove the unconditional Native fallback for configured MAP shared-shredding layouts. Rust validates the schema/options and handles column placement, adaptive widths and Parquet metadata. - Separate the common Parquet row-ID update constraints from the Python streaming writer's narrower layout support. This enables Native MAP row-ID, predicate/callable updates and upserts without bypassing the Python writer's MAP converter. - Document Native MAP support. Keep Python implementation-specific buffering/row-group tests explicitly on the Python writer; add dedicated Native integration coverage. ### Tests Against a locally rebuilt native package from Rust PR #1027 (`999c478a3af54880e3996c434c8094617917b042`): - 50 new cases cover append, PK and data-evolution tables; batch/stream writes; plain/sequential/LRU physical mappings; adaptive rolling and partition isolation; none/lz4/zstd; duplicate keys with Java's last-value semantics; nested and sliced Arrow values; NULL/empty maps; PK merge engines and input changelogs; abort cleanup; row-ID, predicate/callable updates and upserts. - Verify physical Parquet MAP metadata and both Python/Native full reads and literal-key projections. Native cases reject fallback to the Python writer/updater/reader. - Related regression suites with all five Native CI flags enabled (plan/read/write/update/commit): **828 passed, 3 skipped, 78 subtests passed**. Includes native writes/updates/Blob writes, Arrow schema tests, shared-shredding readers/writers, and MAP-key projections. - `python -m flake8 --config=dev/cfg.ini` on all changed Python files: passed. No Python implementation of the MAP placement policies or adaptation is added here; the Native path uses Rust core. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
