fhan688 commented on issue #20014:
URL: https://github.com/apache/hudi/issues/20014#issuecomment-5791070288
Thanks @danny0405 and @cshuo. I agree that this should be framed as an
optional Hudi storage-cost optimization rather than a missing part of the
conventional LSM lifecycle or an RFC-103 correctness requirement.
I will revise the issue to remove the target-sized-file implication and
treat lower write amplification / higher throughput as hypotheses to be
measured. The comparison baseline will be the current direct-flush path with
the same writer-memory budget and the same log/minor-compaction configuration.
I will also extend the design in three areas before opening the
integration PR:
1. cover both inactive-victim reclaim and the single-hot-bucket path
after an active buffer write fails;
2. avoid rematerializing an externally spilled batch in
`FlinkWriteHelper.deduplicateRecords()` when pre-combine is enabled, either
through streaming key-group reduction or an explicit initial limitation;
3. define failure/recovery semantics, task-level disk limits, bounded
merge fan-in, working-memory reservation, cleanup and checkpoint-latency
metrics.
Before production wiring, I will publish spill-on/off results on the same
master baseline for many buckets, one hot bucket and different checkpoint
intervals, including file counts/sizes, throughput, checkpoint p95/p99, spill
I/O, remote bytes including compaction, compaction backlog and read latency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
