yihua opened a new issue, #883: URL: https://github.com/apache/hudi-rs/issues/883
### Feature Description `run_write_tasks` drives a write's per-file tasks with `buffer_unordered` on the task that awaits it. The futures overlap their I/O, but their CPU work (reading the base file, merging, building the output batches) runs on that one task, one file group after another; only the Parquet encode inside `spawn_blocking` reaches the runtime's other threads. A copy-on-write upsert that touches several file groups therefore merges them serially, with the calling thread busy for nearly the whole upsert while the workers sit mostly idle. Proposed: spawn each task onto the runtime inside the existing bounded stream, keeping the concurrency bound and input-order results and aborting the tasks if the write is dropped, and run the copy-on-write merge together with the encode on the blocking pool so the long synchronous section does not stall the async workers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
