leaves12138 commented on code in PR #896:
URL: https://github.com/apache/paimon-rust/pull/896#discussion_r4060116425
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -231,6 +242,19 @@ impl DataEvolutionReader {
self
}
+ pub(crate) fn with_limit(mut self, limit: Option<usize>) -> Self {
+ self.limit = limit;
+ self
+ }
+
+ fn effective_batch_size(&self) -> Option<usize> {
+ match (self.batch_size, self.limit) {
+ (Some(size), Some(limit)) if limit > 0 => Some(size.min(limit)),
+ (None, Some(limit)) if limit > 0 => Some(limit),
Review Comment:
[P2] Bound physical managed-BLOB reads by the remaining quota, not just the
initial LIMIT.
This batch size stays constant as `remaining` decreases. Managed `.blob`
payloads are decoded in the source stream, before `finish_wide_batch` applies
`take_limited_batch`, so the final partial batch can still fetch rows outside
the requested output.
I reproduced both cases with the companion Python PR, native
planning/reading enabled, no predicate, and an `id` + managed `payload` table:
1. Write five rows, set `read.batch-size=2`, and read with `LIMIT 3`. The
second batch decodes payloads for rows 3 **and 4**, although only row 3 is
needed.
2. With the default batch size, commit two rows and then three rows. `LIMIT
3` consumes the first split's two rows, but the second split still decodes all
three payloads instead of just one.
Changing only row 4's payload bytes while preserving the BLOB index/file
length makes both native reads fail with `Blob entry CRC32 mismatch`; the
Python deferred reader returns rows 1-3 successfully. This is observable extra
I/O and can turn an otherwise successful limited read into an error.
Could we cap the physical BLOB selection/read request by the remaining
output quota for this no-post-filter path, or defer managed-payload resolution
until after limiting? Please cover both a partial final batch and a transition
between files/splits; the current `LIMIT 1` managed-BLOB test does not exercise
either boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
