XiaoHongbo-Hope opened a new pull request, #10947:
URL: https://github.com/apache/paimon/pull/10947

   Sparse row-ID updates currently decode and re-encode the complete ordinary 
Parquet file group. This adds an experimental, opt-in path that rewrites 
affected row groups and copies untouched compressed column chunks into a 
complete replacement file.
   
   The Python gate requires a single latest physical file to provide exactly 
the updated columns. Unsupported layouts and helper failures fall back to the 
existing writer. The Rust helper retains column statistics, page/offset 
indexes, bloom filters and schema metadata. Existing row-ID conflict checks and 
commit handling remain in place.
   
   Validation:
   - Rebased onto master `bf6c81cbc8`; 35 targeted Python tests passed with the 
real helper.
   - Helper test passed for values, nested fields, field IDs and page indexes.
   - Java reader smoke test passed on a 30,000-row table produced by the 
original exact-version prototype. Java/VVR compaction remains unverified.
   - One local synthetic 300,000-row / 10-column comparison, updating 84 rows 
in two row groups: preparation 1.866 s → 0.309 s. This excludes remote IO and 
conflict retries; it is not an end-to-end production result.
   - `git diff --check` passed.
   
   Draft limitations:
   - Requires an explicitly installed standalone Rust helper plus 
`data-evolution.row-group-reuse.enabled=true`; helper-dependent tests skip when 
the executable is absent. CI packaging/integration is pending.
   - Downloads the full source and writes a full replacement; remote byte 
volume is not reduced.
   - Conservative gates reject metadata-statistics-enabled tables, schema 
changes, multi-source columns, DV/sidecars/special formats, oversized files/row 
groups and insufficient staging space.
   - Before merge: complete failure-injection and index compatibility coverage, 
Java/VVR compaction verification, remote IO benchmarking and helper 
distribution design.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to