SteNicholas opened a new pull request, #576:
URL: https://github.com/apache/paimon-cpp/pull/576

   <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): 
<description> -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: #561
   
   <!-- What is the purpose of the change -->
   Follow-up to #564, addressing the suggestions for a
   [configurable preparation 
window](https://github.com/apache/paimon-cpp/pull/564#discussion_r4597619301)
   and [lazy split 
initialization](https://github.com/apache/paimon-cpp/pull/564#issuecomment-6944276366).
   
   Append/raw reads used a fixed four-reader window, while 
`CreateReader(splits)` opened every
   split up front. A plan with S raw splits could retain up to 4 × S reader 
stacks before its
   first batch, including preparation for splits that an early-stopping caller 
never consumed.
   
   The change makes the preparation window configurable and opens splits only 
as they are reached:
   
   - **`ConcatBatchReader` and `RawFileSplitRead`**: add 
`read.preparation-window`, accepting
     positive 32-bit values with a default of 4. The window bounds retained 
data-file reader
     stacks, including in-flight builds, independently of build-executor 
parallelism, batch
     prefetch and warmup. A value of 1 retains only the current file reader.
   - **`AppendOnlyTableRead` and `KeyValueTableRead`**: open splits in input 
order on demand,
     closing each before opening the next. The returned reader owns its 
creation state and
     captures the selected delete handling, so it can outlive `TableRead` and 
the input split
     list. Closing before the first batch opens no files.
   - **Real-time read views**: validate and retain all views before consuming 
any tickets.
     Validation failures preserve tickets for retry; successful creation 
consumes them even
     when the reader is closed without reading. Retained views keep deferred 
reads valid
     across ticket expiry and committed-state refreshes.
   - **`LazyConcatBatchReader` lifecycle**: release each source factory after 
opening its reader
     and release pending factories and views on EOF, close or terminal read 
errors. Errors
     identify the affected source. Closing drains pending preparation before 
collecting final
     metrics, and returned batches remain valid after their readers are 
released.
   
   For raw disk splits, the supplied split list retains at most one preparation 
window of file
   readers. Split metadata and retained real-time views still scale with the 
plan; merge-on-read
   keeps eager construction of the current split's merge inputs.
   
   ### Tests
   
   <!-- Describe tests added or updated and what they cover. -->
   UT:
   
   - `src/paimon/core/core_options_test.cpp`:
     - Extend default, map and copy-assignment cases for the new option.
     - `TestReadPreparationWindow`: covers positive limits, nonpositive values, 
malformed input
       and integer overflow.
   - `src/paimon/common/reader/concat_batch_reader_test.cpp`:
     - Extend existing window, warmup, early-close and deferred-failure cases 
across window
       sizes and executor thread counts, including a window of 1 and windows 
larger than the
       file count.
     - Verify ordered output, bitmap handling, metrics, bounded retained 
readers, overlapping
       refill, cancellation and cleanup through both close and destruction.
   - `src/paimon/core/operation/raw_file_split_read_test.cpp`:
     - Extend real-file preparation and missing-file cases across configured 
windows.
     - Retain coverage for creator lifetime, per-file predicates, deletion 
vectors, row kinds
       and row IDs.
   - `src/paimon/core/table/format/lazy_concat_batch_reader_test.cpp`:
     - Consolidate source ordering, factory release, empty/skipped sources, 
bitmap selection
       and returned-batch lifetime cases.
     - Cover early/repeated close and destruction, terminal open/read/bitmap 
failures, and
       preservation of completed-reader and preparation metrics.
   - `src/paimon/core/table/source/table_read_test.cpp`:
     - `TestLazySplitsBoundResourcesAndOutliveCreator`: checks ordered 
multi-split output,
       bounded resources, ownership and early close for append, primary-key raw 
and merge reads.
     - `TestLazySplitFailureLeavesLaterSplitsUnopened`: checks deferred errors 
and cleanup
       without opening later splits.
     - Extend the primary-key read-entry case to verify captured delete 
handling.
   
   IT:
   
   - `test/inte/realtime_write_inte_test.cpp`:
     - `TestTicketExpiryPreservesCreatedReader`: verifies that an 
already-created deferred
       reader remains usable after unused tickets expire.
     - `TestRealtimeReaderFailureTimingAndTicketConsumption`: combines 
single/multi-split
       failure timing, retry and successful ticket-consumption checks.
     - `TestLazyMultiSplitReadersRetainViewsAndStopOpening`: covers append and 
primary-key
       split order, validation, ticket preservation, creator lifetime and early 
termination.
     - Update `TestPkRead` lifetime checks for deferred creation.
   - `test/inte/nested_column_pruning_inte_test.cpp` and
     `test/inte/scan_and_read_inte_test.cpp`:
     - Update schema/projection and unsupported-aggregation cases to expect 
split initialization
       errors during reading.
   
   ### API and Format
   
   <!-- Does this change affect API in include dir or storage format or 
protocol -->
   Adds the public option constant `Options::READ_PREPARATION_WINDOW` for
   `read.preparation-window`, with a default of 4 and positive 32-bit values.
   
   The append-only and primary-key implementations of 
`TableRead::CreateReader(splits)` now open
   splits on demand. Null splits and real-time tickets are validated at 
creation; other split
   initialization errors are terminal errors from `NextBatch` or 
`NextBatchWithBitmap` when the
   split is reached. Successful creation consumes real-time tickets and retains 
their views.
   
   Single-split creation keeps its initial-window error timing. There are no 
stored file-format
   or table-protocol changes.
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   `docs/source/user_guide/prefetch.rst` documents the option, configuration 
example, window-of-one
   behavior, lazy multi-split initialization, ownership, error timing and 
real-time ticket semantics.
   Public API comments describe the option and the multi-split reader contract.
   
   ### Generative AI tooling
   
   Generated-by: Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to