yihua opened a new pull request, #898:
URL: https://github.com/apache/hudi-rs/pull/898
## Description
closes #897
`update_record_index` encoded each record-level index shard (sort, HFile,
log block framing) sequentially on the writing task. Each shard is now built in
`spawn_blocking`, at most one per core at a time through an ordered `buffered`
stream, and written as soon as it is ready. Shards are still written in shard
order with the same file names, contents and write stats, and only the shards
in flight hold their encoded bytes, so memory stays bounded.
A 10M-row bulk append with the record index went from about 16 s to about 12
s locally.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Behavior-preserving; the record index write, read-back and compaction suites
pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
