yihua opened a new pull request, #884:
URL: https://github.com/apache/hudi-rs/pull/884
## Description
closes #883
`run_write_tasks` now spawns each per-file task onto the runtime inside the
same bounded stream, so file groups are read and merged in parallel instead of
one after another on the awaiting task. The concurrency bound and input-order
results are unchanged, and dropping the write aborts the spawned tasks. The
copy-on-write merge now runs together with the encode inside the existing
`spawn_blocking`, so its long synchronous section does not stall the async
workers; this is where most of the gain comes from, since the other call sites
already encode on the blocking pool. A panic in a task now surfaces as a write
error instead of unwinding past the caller's cleanup. Writes must be awaited on
a tokio runtime, which every caller already does.
On a 1M-row copy-on-write table with 10,000 updates per commit, the median
upsert went from about 1.19 s to about 0.59 s (local SSD, same final table by
digest). Peak memory can rise with concurrent merges and stays bounded by
`hoodie.write.task.parallelism`.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Unit tests check that tasks run off the caller's thread, that dropping the
write aborts them, and that a task panic becomes an error.
`test_cow_upsert_merge_runs_off_the_async_worker` runs an upsert on a
single-worker runtime and counts allocations made on the worker thread; it
fails with the merge left on the async worker and does not depend on timing.
The existing write suites pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
