SteNicholas opened a new pull request, #1449: URL: https://github.com/apache/paimon-rust/pull/1449
### Purpose Linked issue: #1444 Avro data-file reads currently fetch and decode the entire file before returning the first Arrow batch. Read OCF blocks on demand so short scans can stop after bounded in-flight work and memory depends on the current block and batch instead of all decoded records. ### Brief change log - Read OCF headers and blocks through a 64 KiB FileRead buffer. Yield between buffer refills so immediately ready cached reads remain cancellable. - Run schema parsing, decompression, datum decoding, Arrow conversion and residual filtering in blocking tasks. Limit each decode task to one batch of physical records, keep no background producer or decoded-batch queue, and release encoded block bytes once apache-avro has copied them. - Apply inclusive row ranges using original physical positions without a file-sized mask. Skip irrelevant block payloads while checking framing and sync markers, and stop after the final selected row. - Preserve supported codecs, nested and logical values, type conversion, projection widening for predicate columns, exact residual filtering and row order. Return errors for corrupt or truncated data when its framing, block or records are consumed. ### Tests - Streaming tests in `crates/paimon/src/arrow/format/avro/streaming_tests.rs`: lazy construction, bounded first-batch reads, consumer backpressure, pending and immediately ready read cancellation, and resource release on completion, failures and early drop. - Consolidate numeric and LIKE predicate coverage across projected and unprojected columns and physical row ranges. Cover full scans, cross-block ranges, empty selections and header-only files, all six supported codecs, large blocks, missing and reordered fields, numeric widening, nullable unions, nested and logical values, malformed input and I/O error propagation. - Add the ignored `compare_avro_reads` comparison in `crates/paimon/src/arrow/format/avro/streaming_tests/benchmark.rs`. It compares eager and streaming full scans and prefix/LIMIT workloads with fixed block and batch sizes, reporting peak RSS, time to first batch, throughput and bytes read. Compilation, builds, tests, validation and performance measurements have not been run, as requested. ### API and Format Keep the FormatFileReader interface and Avro OCF encoding unchanged. Reads become lazy, with I/O and decoding errors returned while polling the stream. Batches end at block boundaries and can be smaller after row selection or filtering. Memory includes the header, compressed and decompressed block, conversion scratch and input buffer; a large block or value can exceed the output batch's memory size. ### Documentation Add `benchmarks/avro/README.md` with the Java Paimon block traversal reference, physical row-selection semantics, memory and cancellation bounds, and a reproducible eager-versus-streaming comparison procedure. Document the per-block schema parsing tradeoff and state explicitly that no performance measurements accompany the change. 🤖 Assisted with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
