XiaoHongbo-Hope opened a new pull request, #10947: URL: https://github.com/apache/paimon/pull/10947
Sparse row-ID updates currently decode and re-encode the complete ordinary Parquet file group. This adds an experimental, opt-in path that rewrites affected row groups and copies untouched compressed column chunks into a complete replacement file. The Python gate requires a single latest physical file to provide exactly the updated columns. Unsupported layouts and helper failures fall back to the existing writer. The Rust helper retains column statistics, page/offset indexes, bloom filters and schema metadata. Existing row-ID conflict checks and commit handling remain in place. Validation: - Rebased onto master `bf6c81cbc8`; 35 targeted Python tests passed with the real helper. - Helper test passed for values, nested fields, field IDs and page indexes. - Java reader smoke test passed on a 30,000-row table produced by the original exact-version prototype. Java/VVR compaction remains unverified. - One local synthetic 300,000-row / 10-column comparison, updating 84 rows in two row groups: preparation 1.866 s → 0.309 s. This excludes remote IO and conflict retries; it is not an end-to-end production result. - `git diff --check` passed. Draft limitations: - Requires an explicitly installed standalone Rust helper plus `data-evolution.row-group-reuse.enabled=true`; helper-dependent tests skip when the executable is absent. CI packaging/integration is pending. - Downloads the full source and writes a full replacement; remote byte volume is not reduced. - Conservative gates reject metadata-statistics-enabled tables, schema changes, multi-source columns, DV/sidecars/special formats, oversized files/row groups and insufficient staging space. - Before merge: complete failure-injection and index compatibility coverage, Java/VVR compaction verification, remote IO benchmarking and helper distribution design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
