SteNicholas opened a new pull request, #576: URL: https://github.com/apache/paimon-cpp/pull/576
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #561 <!-- What is the purpose of the change --> Follow-up to #564, addressing the suggestions for a [configurable preparation window](https://github.com/apache/paimon-cpp/pull/564#discussion_r4597619301) and [lazy split initialization](https://github.com/apache/paimon-cpp/pull/564#issuecomment-6944276366). Append/raw reads used a fixed four-reader window, while `CreateReader(splits)` opened every split up front. A plan with S raw splits could retain up to 4 × S reader stacks before its first batch, including preparation for splits that an early-stopping caller never consumed. The change makes the preparation window configurable and opens splits only as they are reached: - **`ConcatBatchReader` and `RawFileSplitRead`**: add `read.preparation-window`, accepting positive 32-bit values with a default of 4. The window bounds retained data-file reader stacks, including in-flight builds, independently of build-executor parallelism, batch prefetch and warmup. A value of 1 retains only the current file reader. - **`AppendOnlyTableRead` and `KeyValueTableRead`**: open splits in input order on demand, closing each before opening the next. The returned reader owns its creation state and captures the selected delete handling, so it can outlive `TableRead` and the input split list. Closing before the first batch opens no files. - **Real-time read views**: validate and retain all views before consuming any tickets. Validation failures preserve tickets for retry; successful creation consumes them even when the reader is closed without reading. Retained views keep deferred reads valid across ticket expiry and committed-state refreshes. - **`LazyConcatBatchReader` lifecycle**: release each source factory after opening its reader and release pending factories and views on EOF, close or terminal read errors. Errors identify the affected source. Closing drains pending preparation before collecting final metrics, and returned batches remain valid after their readers are released. For raw disk splits, the supplied split list retains at most one preparation window of file readers. Split metadata and retained real-time views still scale with the plan; merge-on-read keeps eager construction of the current split's merge inputs. ### Tests <!-- Describe tests added or updated and what they cover. --> UT: - `src/paimon/core/core_options_test.cpp`: - Extend default, map and copy-assignment cases for the new option. - `TestReadPreparationWindow`: covers positive limits, nonpositive values, malformed input and integer overflow. - `src/paimon/common/reader/concat_batch_reader_test.cpp`: - Extend existing window, warmup, early-close and deferred-failure cases across window sizes and executor thread counts, including a window of 1 and windows larger than the file count. - Verify ordered output, bitmap handling, metrics, bounded retained readers, overlapping refill, cancellation and cleanup through both close and destruction. - `src/paimon/core/operation/raw_file_split_read_test.cpp`: - Extend real-file preparation and missing-file cases across configured windows. - Retain coverage for creator lifetime, per-file predicates, deletion vectors, row kinds and row IDs. - `src/paimon/core/table/format/lazy_concat_batch_reader_test.cpp`: - Consolidate source ordering, factory release, empty/skipped sources, bitmap selection and returned-batch lifetime cases. - Cover early/repeated close and destruction, terminal open/read/bitmap failures, and preservation of completed-reader and preparation metrics. - `src/paimon/core/table/source/table_read_test.cpp`: - `TestLazySplitsBoundResourcesAndOutliveCreator`: checks ordered multi-split output, bounded resources, ownership and early close for append, primary-key raw and merge reads. - `TestLazySplitFailureLeavesLaterSplitsUnopened`: checks deferred errors and cleanup without opening later splits. - Extend the primary-key read-entry case to verify captured delete handling. IT: - `test/inte/realtime_write_inte_test.cpp`: - `TestTicketExpiryPreservesCreatedReader`: verifies that an already-created deferred reader remains usable after unused tickets expire. - `TestRealtimeReaderFailureTimingAndTicketConsumption`: combines single/multi-split failure timing, retry and successful ticket-consumption checks. - `TestLazyMultiSplitReadersRetainViewsAndStopOpening`: covers append and primary-key split order, validation, ticket preservation, creator lifetime and early termination. - Update `TestPkRead` lifetime checks for deferred creation. - `test/inte/nested_column_pruning_inte_test.cpp` and `test/inte/scan_and_read_inte_test.cpp`: - Update schema/projection and unsupported-aggregation cases to expect split initialization errors during reading. ### API and Format <!-- Does this change affect API in include dir or storage format or protocol --> Adds the public option constant `Options::READ_PREPARATION_WINDOW` for `read.preparation-window`, with a default of 4 and positive 32-bit values. The append-only and primary-key implementations of `TableRead::CreateReader(splits)` now open splits on demand. Null splits and real-time tickets are validated at creation; other split initialization errors are terminal errors from `NextBatch` or `NextBatchWithBitmap` when the split is reached. Successful creation consumes real-time tickets and retains their views. Single-split creation keeps its initial-window error timing. There are no stored file-format or table-protocol changes. ### Documentation <!-- Does this change introduce a new feature --> `docs/source/user_guide/prefetch.rst` documents the option, configuration example, window-of-one behavior, lazy multi-split initialization, ownership, error timing and real-time ticket semantics. Public API comments describe the option and the multi-split reader contract. ### Generative AI tooling Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
