SteNicholas opened a new pull request, #564:
URL: https://github.com/apache/paimon-cpp/pull/564

   <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): 
<description> -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #561
   
   <!-- What is the purpose of the change -->
   Sequential append/raw reads constructed every file reader before returning, 
retaining open
   streams and format metadata for the entire split. Splits with many small 
files delayed the
   first batch and paid setup costs for files that early-stopping readers never 
consumed.
   
   The change prepares an ordered, bounded window of readers while retaining 
concurrent
   construction and next-file warmup:
   
   - **`ConcatBatchReader`**: adds lazy construction with a window of at most 
four reader stacks,
     including in-flight builds. It releases exhausted readers, retains their 
metrics and refills
     the window as consumption advances. Ready lookahead readers are warmed 
without blocking
     reads from the current file; skipped and empty files preserve the split's 
file order.
   - **`RawFileSplitRead`**: uses the lazy path for sequential append/raw reads 
and retains an
     independent creator with its context, copied partition, file metadata, 
schema inputs,
     filesystem and memory pools. Per-file predicates, deletion vectors, row 
kinds and file-row
     IDs retain their existing behavior. Merge-on-read keeps its eager 
input-construction helpers.
   - **Reader lifecycle**: validates plans eagerly and reports initial-window 
construction errors
     from `CreateReader`. Later file-open/build failures become terminal read 
errors when the
     affected file is reached. Close, destruction and failure cancel queued 
builds, drain running
     work and release prepared readers without scheduling the unopened tail. 
Returned batches
     remain valid after their readers are released.
   - **Sequential-reader benchmarks**: compare eager and lazy construction over 
128 or 1024 small
     files, for full scans and first-batch termination. Counters cover 
first-batch latency,
     file opens, initial footer/setup reads, peak retained streams, tracked 
pool allocation,
     bytes read and throughput, with an optional per-open delay to simulate 
remote setup latency.
   
   The construction bound applies to reader stacks, including work in flight. 
Per-file metadata
   and owned creation inputs still scale with the split's file count; batches 
retained by callers
   remain a separate memory cost. The construction window is independent of 
batch prefetch and
   warmup settings.
   
   ### Tests
   
   <!-- Describe tests added or updated and what they cover. -->
   UT:
   
   - `src/paimon/common/reader/concat_batch_reader_test.cpp`:
     - `TestSimple` and `TestSimpleWithBitmap`: reuse the existing data and 
bitmap cases for eager
       and lazy readers, including empty inputs and empty files across several 
batch sizes.
     - `TestWarmupLooksOneReaderAhead`: checks eager and lazy warmup ordering, 
including refilled
       windows, without warming beyond the next reader.
     - `LazyWindowPreservesOrderBitmapAndMetrics`: checks bounded live readers, 
skipped and empty
       files, ordered output, both read APIs, accumulated metrics and creator 
release at EOF.
     - `LazyEarlyCloseAndDestructionReleaseCreators`: verifies bounded 
first-batch construction,
       early/repeated close, destruction, creator cleanup and returned-batch 
validity.
     - `LazyInitialFailureDrainsPartialWindow` and 
`LazyDeferredFailuresAreTerminal`: cover partial
       initialization cleanup and terminal build, read and bitmap-conversion 
failures.
     - `LazyRefillOverlapsReadingAndCloseCancelsQueuedBuilds`: gates a refill 
to verify that reading
       continues while construction is in flight; close and destruction drain 
running work and
       prevent queued factories from opening additional files.
   - `src/paimon/core/operation/raw_file_split_read_test.cpp`:
     - `LazyInitialWindowBuildsConcurrentlyAndEarlyCloseStopsOpening`: uses a 
gated counting
       filesystem to check concurrent initial preparation, bounded first-batch 
opens and cleanup
       without opening the tail after early close.
     - `LazyReaderOutlivesCreatorWithPerFilePredicatesDvAndRowIds`: reads 
across multiple windows
       after releasing caller-owned state, checking per-file predicate 
ownership, deletion vectors,
       row kinds, row IDs, retained output and bounded resources.
     - `LazyValidationAndFileOpenFailures`: consolidates invalid-plan checks, 
including the null
       predicate-provider case, and verifies that validation opens no data 
files. It also checks
       initial and deferred missing-file errors, repeated failure propagation 
and stream cleanup.
   
   IT:
   
   No standalone integration tests are added. The raw-reader tests exercise 
real ORC files through
   `RawFileSplitRead`, including filesystem opens, filtering, deletion vectors 
and row tracking.
   
   ### API and Format
   
   <!-- Does this change affect API in include dir or storage format or 
protocol -->
   No public API under `include/`, stored file format or table protocol 
changes. File-open/build
   failures beyond the initial preparation window are now reported through 
`NextBatch` or
   `NextBatchWithBitmap` when the affected file is reached, instead of during 
`CreateReader`.
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   `docs/source/user_guide/prefetch.rst` describes bounded sequential 
preparation, deferred-error
   semantics, ownership and cleanup, and its relationship to next-file warmup.
   `docs/source/examples/benchmark.rst` documents the eager/lazy comparison, 
full-scan and early-stop
   cases, counters and measurement boundaries. It distinguishes tracked pool 
allocation from
   process RSS and simulated open latency from a complete remote-storage model.
   
   ### Generative AI tooling
   
   Generated-by: Codex (GPT-6)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to