JingsongLi opened a new pull request, #10383: URL: https://github.com/apache/paimon/pull/10383
### Purpose Enable native REST publication for supported Data Evolution commits and custom manifest targets, with Java-compatible metadata semantics. Depends on https://github.com/apache/paimon-rust/pull/1029. This PR is draft until that core fix is merged. Native CI continues building paimon-rust from `main` and will need a rerun after the dependency lands. - Route DE appends, overwrites, deletion-vector updates, and updates with row-ID rebase disabled through the existing Rust core committer. - Keep PyPaimon's automatic staged-file rebase recovery for preassigned row-ID updates when enabled. - Normalize the partition grouping option before constructing a native table. - Align Python's sequence sentinel handling with Java `RowTrackingCommitUtils`, preserving the minimum when only the maximum is pending. - Align materialized manifest rolling with Java's 1000-record size-check cadence and fixed Avro sync interval, allowing native publication with custom manifest targets. The Rust dependency also fixes Blob range validation before publication and adds Java's `IGNORE` global-index commit policy. `IGNORE` retains existing index metadata; incremental index refresh is a separate capability. ### Tests New REST integration coverage verifies actual native publication without Python commit fallback: - Python/native writers, both readers, partition grouping and row IDs. - Sidecar publication and row-range pruning. - Partial updates, concurrent same-column rejection and disjoint-column updates. - Overwrite sequence preservation, deletion vectors, stream abort, and subsequent row-ID allocation. - Independently rolled Blob columns and external file paths. - Rejection of a real stale Blob file spanning two adjacent normal ranges, leaving a readable snapshot unchanged. - All three Java global-index update policies. - Custom manifest targets and Python rebase fallback. Validation with a locally rebuilt wheel from the Rust companion: - Five native switches enabled; selected commit/manifest/Data Evolution/native write/update/Blob suites: **929 passed, 3 skipped, 73 subtests passed**. The three additional Vortex tests could not run because this local environment does not have the optional `vortex` module; their failures are unrelated to these changes. - Changed Python files pass the repository's Flake8 configuration. - Sequence and rolling regressions were confirmed failing before their fixes; the real stale-Blob and `IGNORE` fixtures also failed with the previous Rust wheel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
