yihua commented on code in PR #689:
URL: https://github.com/apache/hudi-rs/pull/689#discussion_r3910875891
##########
crates/core/src/file_group/reader_v2/engine.rs:
##########
@@ -133,9 +134,25 @@ pub struct HoodieFileGroupReader {
// visible to (a) the base parquet read here, and (b) the parquet log
// block decoder in `file_group::log_file::content::Decoder`. The gate
// (CoW || mor_pk_safe) lives at the use sites; this file's gate is at
- // `make_base_file_source` below.
+ // `base_file_source` below.
}
+/// Rows per base batch handed to the merge, and therefore per merged chunk.
+///
+/// Load-bearing rather than cosmetic: merging a chunk is synchronous work on
the
+/// task that polls the stream, and its cost is linear in the chunk's rows. On
+/// this machine, one merge of a 1024-row chunk against a 50k-key log map takes
+/// 0.4-1.1 ms, and 5.7-6.1 ms once the merge map has spilled to disk; at 8192
+/// rows those become 2.8 ms and ~40 ms. So the chunk size is what bounds how
long
+/// a single poll occupies its executor, and it is set here rather than
inherited.
+///
+/// 1024 is what `parquet` already defaults to, so this pins today's behaviour
+/// instead of changing it. Pinned because the bound is silent if it moves: a
+/// larger default upstream would multiply the blocking above with nothing
+/// failing. Measured by `spilled_merge_blocking_duration` (ignored; run with
+/// `--release --ignored --nocapture`).
+const MERGE_CHUNK_ROWS: usize = 1024;
Review Comment:
non-blocking: The chunk bound pinned here is 1024 for merged base chunks,
but the log-only drain still emits 4096-row chunks per poll
(`DEFAULT_BATCH_SIZE` in `drain_log_only_inserts`), and a spilled map's drain
does its RocksDB pulls on the same polling task the bound exists to protect.
Was leaving the drain at 4x the merge bound deliberate — e.g. because the drain
has no per-row map probe — or should it follow `MERGE_CHUNK_ROWS` too? Either
answer seems fine, but the asymmetry is the one occupancy decision the PR
doesn't record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
