SteNicholas opened a new issue, #1444: URL: https://github.com/apache/paimon-rust/issues/1444
### Search before asking - [x] I searched existing issues and open pull requests. No issue or active PR covering this specific scope was found on November 18, 2026. ### Motivation [AvroFormatReader::read_batch_stream](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/arrow/format/avro.rs#L45-L112) currently: 1. Reads `0..file_size` into memory. 2. Deserializes every record into `Vec<Value>`. 3. Applies positional selection with a file-sized mask. 4. Only then returns a stream that converts chunks into Arrow batches. Consequently, the first batch waits for full-file I/O and decoding, peak memory includes the entire encoded file and decoded records, and a caller consuming only a small prefix cannot avoid the initial full-file work. Java's [AvroBulkFormat](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-format/src/main/java/org/apache/paimon/format/avro/AvroBulkFormat.java#L84-L138) reads and exposes records one OCF block at a time. Sequential OCF access does not require retaining the whole file. ### Solution Implement incremental Avro data-file reading behind the existing FormatFileReader interface. - Read the OCF header and subsequent blocks through buffered FileRead access. Bound prefetch and queued decoded batches; emit the first batch before consuming the remaining file. - Decode only the records needed for the current bounded output batch rather than collecting a file-wide `Vec<Value>`. - Apply requested row ranges incrementally using original physical row positions, without allocating a boolean mask proportional to the entire file. Skip decoding irrelevant blocks when their record counts permit this safely; do not promise arbitrary random access without an index. - Preserve the existing type conversions, nested values, projection widening for predicate columns, exact residual filtering, row order and row-selection semantics. - Honor consumer backpressure. Dropping the stream or satisfying an upstream LIMIT should stop further reads/decoding after bounded in-flight work. - Integrate blocking decoding safely with the async runtime, and document memory bounds in terms of compressed/decompressed block sizes, output batches and bounded prefetch. A single very large Avro block must not be hidden behind a claim of a strict batch-sized memory limit. - Preserve errors for consumed corrupt/truncated blocks and the existing supported codecs. #### Acceptance criteria - [ ] A counting FileRead proves that the first output batch does not require a multi-block file to be fully fetched. - [ ] After consuming a short prefix and dropping the stream, additional I/O stays within the documented in-flight bound. - [ ] Full scans, row ranges spanning block boundaries, empty ranges/files, predicates using unprojected columns, and supported nested/logical types return the same results as the current reader. - [ ] Supported compressed codecs and malformed/truncated inputs are covered. - [ ] A reproducible comparison reports peak memory, time to first batch, total throughput and bytes read for full scans and prefix/LIMIT workloads, with fixed block and batch sizes. - [ ] Resource release is covered for normal completion, read/decode failure and early stream drop. ### Anything else? Scope: Avro **data-file** reads in `arrow/format/avro.rs`. The specialized manifest Avro decoder, manifest compaction and general benchmark infrastructure (#483) are separate work. This improvement should not require redesigning those components. Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These findings come from source inspection; no new runtime reproduction or performance measurement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
