JingsongLi opened a new pull request, #10404: URL: https://github.com/apache/paimon/pull/10404
### Purpose Enable Native Data Evolution batch/stream writes, reads and updates with ROW sidecars. **Depends on https://github.com/apache/paimon-rust/pull/1047. Merge the Rust PR first: Native CI continues building Rust `main`, with no branch pin or compatibility fallback.** - Remove the sidecar-specific Native write/update/read fallback gates and pass the parsed sidecar-generation option into Rust. - Keep ROW BLOB payloads distinct from Parquet BLOB references through Python's file-reader normalization, batch coalescing, Data Evolution column merge and projection. Skip reference/view resolution for decoded payloads, and remove the private provenance marker before exposing output. - Retain primary-file reads when `blob-as-descriptor=true` projects BLOBs, so sparse and full reads return the original references. - Test Native batch/stream writes, direct/incremental updates, deletion vectors, schema evolution, external paths, logical MAP/VARIANT ROW layouts, dedicated BLOB exclusions, and Python/Rust read interoperability. BLOB sidecars are produced by Rust and consumed by both readers. - Update the iterable-failure regression to retain prepared files after ownership handoff, while verifying the committed snapshot stays unchanged. - Limit the LeRobot Python-version mock to the tested module; modifying global `sys.version_info` interferes with pytest subtests on Python 3.13. ### Tests With a local binding built from Rust PR #1047 and all Native test lanes enabled: - Final affected-area regression: **1174 passed, 31 skipped, 88 subtests passed**. Native counters: plans 1195, reads 1387, writes 1111, commits 161; all update modes exercised. - New sidecar and batch-reader regressions: **37 passed, 4 subtests passed**. - Existing `py36` regression directory: **79 passed** on the local Python 3.13 runtime (not a separate Python 3.6 interpreter). - Repository Flake8 configuration and `git diff --check`: passed. An earlier broad Native CI scope run completed with 8520 passing tests. Its failures were missing local Lance dependencies and the global version mock above; dependencies were installed, the mock was fixed, and the failed cases were rerun successfully. The final affected-area run uses the completed implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
