yihua opened a new issue, #929: URL: https://github.com/apache/hudi-rs/issues/929
### Feature Description This is a pure performance optimization. It does not change the files a write produces, the commit metadata, or any read result, and the current write path is correct as it is. The write path runs one task per file (`run_write_tasks` in `crates/core/src/write/mod.rs`), up to `hoodie.write.task.parallelism` at a time, 2x the available cores by default. That leaves two gaps. **Memory is bounded by task count, not bytes.** A copy-on-write merge task holds the file group's current rows, the incoming rows, the combined and merged batches, and the encoded parquet bytes until the put finishes. With merges now running concurrently (#884), peak memory scales with cores times file group size: a 1M-row copy-on-write upsert peaks at about 3.0 GiB, against about 1.9 GiB when merges ran one at a time. On a many-core machine rewriting large base files, the default parallelism can exceed available memory even though each task on its own fits. A byte budget would bound this directly: each task reserves its estimated footprint (for a merge, roughly the file slice size plus its share of the incoming batch) from a semaphore before it reads, and releases it after the put. The budget could default to a fraction of available memory, be configurable, and always admit at least one task so a single large file group still makes progress. **Parallelism stops at the file.** Each file is encoded on one thread, so a write that touches few file groups, or a single large one, leaves most cores idle while one thread encodes every column of every row group. Two finer units would help: - Per input partition: the write APIs take batches that are already in memory. Accepting a partitioned stream of batches (for example a DataFusion plan's output partitions) would let each partition be prepared and written as it arrives, overlapping input production with encoding and keeping memory bounded by what is in flight rather than by the whole input. - Per column within a row group: the parquet crate can encode the columns of a row group independently (`ArrowRowGroupWriterFactory` / `ArrowColumnWriter`) and assemble the chunks afterwards, so one file's encode could use several cores. The column tasks belong off the async workers, as the merge and encode are today. Both should keep today's bound on total concurrency, and with the byte budget above, a finer split should not raise peak memory. ### Additional context Measured with the upsert benchmark from #880 (`uuid` scenario, 1M rows, `hudi-cow` leg) on a 14-core machine; the timings from that run were taken under load and are not quoted here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
