XiaoHongbo-Hope opened a new pull request, #9840: URL: https://github.com/apache/paimon/pull/9840
### Purpose Sparse column updates on a large data-evolution file group materialize the complete original columns and retain the merged output until commit. Disabling file rolling also makes the ordinary writer buffer the whole output. Updating a few rows can therefore exhaust worker memory. Stream ordinary Parquet overlays through read, merge and write batches while preserving one output file and its original row-id range. Accumulate file statistics without retaining the payload, and close the reader and delete incomplete output on failure. Reuse the existing chunk-aware merge for nested types and Arrow offset handling. Scope: non-PK Parquet tables without BLOB fields, dedicated vector format, configured variant shredding, row sidecars or input changelogs. Those specialized paths keep their existing implementation. Input updates, reader/Parquet buffers and file metadata still consume memory; this does not impose a hard per-task memory limit. ### Validation - New end-to-end regression fails on master at the whole-group `to_arrow()` call. Batch and stream modes verify read/write interleaving, nested values, nulls, unsorted updates, two successive commits, complete readback, row-id coverage and file statistics. Injected failure after the first output batch verifies reader closure and removal of partial output. - Update, upsert, chunked-update and BLOB suites: 294 passed with Python 3.9 / PyArrow 19.0.1. Final new regression also passes with Python 3.6 / PyArrow 6.0.1. - Synthetic single-file comparison: 524,288 rows with 1 KiB strings, 8,192-row Parquet groups, 50 sparse updates. Separate equivalent fixtures and update processes: peak RSS 839 MiB on master versus 492 MiB with this change; both write one 524,288-row overlay. Large source row groups can still dominate decoder memory. - Flake8 and `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
