SteNicholas opened a new issue, #561:
URL: https://github.com/apache/paimon-cpp/issues/561

   ### Search before asking
   
   - [x] I searched the existing issues and open pull requests and found no 
issue tracking a bounded reader-construction window for sequential multi-file 
splits. Related work: #333 parallelizes file opening/building; #419 warms up 
the next already-built reader; #512 / #513 decouple raw warmup from batch 
prefetch.
   
   ### Motivation
   
   
[AbstractSplitRead::CreateRawFileReadersWithMetaImpl](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/abstract_split_read.cpp#L152)
 schedules construction for every data file and waits for all results before 
returning. 
[RawFileSplitRead](https://github.com/apache/paimon-cpp/blob/main/src/paimon/core/operation/raw_file_split_read.cpp#L139)
 then passes the fully built list to ConcatBatchReader for sequential 
consumption.
   
   The bounded build executor from #333 limits concurrent construction, but the 
number of completed reader stacks retained before the first batch still grows 
with the entire split. Each stack can hold streams, parsed file metadata, and 
format/prefetch state. Large small-file splits pay setup for files that may 
never be consumed if the caller stops early.
   
   
[ConcatBatchReader](https://github.com/apache/paimon-cpp/blob/main/src/paimon/common/reader/concat_batch_reader.cpp#L71)
 already releases readers at EOF and warms one file ahead. Those lifecycle 
improvements can be retained while making construction itself incremental.
   
   ### Solution
   
   Introduce lazy reader construction for paths that concatenate independent 
files:
   
   - Keep ordered per-file descriptors or owned creator state and construct 
only the current file plus a bounded lookahead window.
   - Retain bounded concurrent setup within that window and the existing 
next-file warmup, so a full scan still overlaps remote setup and reading.
   - Release completed reader stacks and refill the window as consumption 
advances.
   - Start with append/raw sequential paths. Keep the eager helpers available 
to callers that genuinely require all active inputs, including merge-on-read.
   - Ensure deferred creator state owns or safely retains its context, 
schema/mapping inputs, filesystem, and memory pool.
   - Preserve plan-level validation. Document that file-open/build failures 
outside the prepared window are reported when that window is reached, and 
propagate asynchronous preparation failures through the read API.
   
   No storage-format change is required. This should build on #333 and #419 
rather than replace their concurrency and warmup improvements.
   
   ### Anything else?
   
   Acceptance and validation:
   
   - [ ] Counting/gated filesystem tests show that construction and the first 
batch open only a bounded window of an N-file split.
   - [ ] Early close does not schedule unopened tail files; already submitted 
work is drained and released safely.
   - [ ] Outstanding reader resources stay bounded by the preparation window 
rather than total file count.
   - [ ] Preserve file order, predicate/bitmap/DV behavior, row kinds, file-row 
IDs, accumulated metrics, empty-file handling, and error propagation.
   - [ ] Verify safe lifetimes when the returned reader outlives its creator, 
plus cancellation and partial initialization failures.
   - [ ] Benchmark large small-file splits with full consumption and early 
termination. Compare first-batch latency, file-open/footer-read counts, peak 
memory/resources, and full-scan throughput.
   
   This issue concerns construction lifetime, independently of raw warmup 
scheduling in #512 / #513. It does not propose changing merge-on-read input 
requirements.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to