SteNicholas opened a new pull request, #564:
URL: https://github.com/apache/paimon-cpp/pull/564
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #561
<!-- What is the purpose of the change -->
Sequential append/raw reads constructed every file reader before returning,
retaining open
streams and format metadata for the entire split. Splits with many small
files delayed the
first batch and paid setup costs for files that early-stopping readers never
consumed.
The change prepares an ordered, bounded window of readers while retaining
concurrent
construction and next-file warmup:
- **`ConcatBatchReader`**: adds lazy construction with a window of at most
four reader stacks,
including in-flight builds. It releases exhausted readers, retains their
metrics and refills
the window as consumption advances. Ready lookahead readers are warmed
without blocking
reads from the current file; skipped and empty files preserve the split's
file order.
- **`RawFileSplitRead`**: uses the lazy path for sequential append/raw reads
and retains an
independent creator with its context, copied partition, file metadata,
schema inputs,
filesystem and memory pools. Per-file predicates, deletion vectors, row
kinds and file-row
IDs retain their existing behavior. Merge-on-read keeps its eager
input-construction helpers.
- **Reader lifecycle**: validates plans eagerly and reports initial-window
construction errors
from `CreateReader`. Later file-open/build failures become terminal read
errors when the
affected file is reached. Close, destruction and failure cancel queued
builds, drain running
work and release prepared readers without scheduling the unopened tail.
Returned batches
remain valid after their readers are released.
- **Sequential-reader benchmarks**: compare eager and lazy construction over
128 or 1024 small
files, for full scans and first-batch termination. Counters cover
first-batch latency,
file opens, initial footer/setup reads, peak retained streams, tracked
pool allocation,
bytes read and throughput, with an optional per-open delay to simulate
remote setup latency.
The construction bound applies to reader stacks, including work in flight.
Per-file metadata
and owned creation inputs still scale with the split's file count; batches
retained by callers
remain a separate memory cost. The construction window is independent of
batch prefetch and
warmup settings.
### Tests
<!-- Describe tests added or updated and what they cover. -->
UT:
- `src/paimon/common/reader/concat_batch_reader_test.cpp`:
- `TestSimple` and `TestSimpleWithBitmap`: reuse the existing data and
bitmap cases for eager
and lazy readers, including empty inputs and empty files across several
batch sizes.
- `TestWarmupLooksOneReaderAhead`: checks eager and lazy warmup ordering,
including refilled
windows, without warming beyond the next reader.
- `LazyWindowPreservesOrderBitmapAndMetrics`: checks bounded live readers,
skipped and empty
files, ordered output, both read APIs, accumulated metrics and creator
release at EOF.
- `LazyEarlyCloseAndDestructionReleaseCreators`: verifies bounded
first-batch construction,
early/repeated close, destruction, creator cleanup and returned-batch
validity.
- `LazyInitialFailureDrainsPartialWindow` and
`LazyDeferredFailuresAreTerminal`: cover partial
initialization cleanup and terminal build, read and bitmap-conversion
failures.
- `LazyRefillOverlapsReadingAndCloseCancelsQueuedBuilds`: gates a refill
to verify that reading
continues while construction is in flight; close and destruction drain
running work and
prevent queued factories from opening additional files.
- `src/paimon/core/operation/raw_file_split_read_test.cpp`:
- `LazyInitialWindowBuildsConcurrentlyAndEarlyCloseStopsOpening`: uses a
gated counting
filesystem to check concurrent initial preparation, bounded first-batch
opens and cleanup
without opening the tail after early close.
- `LazyReaderOutlivesCreatorWithPerFilePredicatesDvAndRowIds`: reads
across multiple windows
after releasing caller-owned state, checking per-file predicate
ownership, deletion vectors,
row kinds, row IDs, retained output and bounded resources.
- `LazyValidationAndFileOpenFailures`: consolidates invalid-plan checks,
including the null
predicate-provider case, and verifies that validation opens no data
files. It also checks
initial and deferred missing-file errors, repeated failure propagation
and stream cleanup.
IT:
No standalone integration tests are added. The raw-reader tests exercise
real ORC files through
`RawFileSplitRead`, including filesystem opens, filtering, deletion vectors
and row tracking.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No public API under `include/`, stored file format or table protocol
changes. File-open/build
failures beyond the initial preparation window are now reported through
`NextBatch` or
`NextBatchWithBitmap` when the affected file is reached, instead of during
`CreateReader`.
### Documentation
<!-- Does this change introduce a new feature -->
`docs/source/user_guide/prefetch.rst` describes bounded sequential
preparation, deferred-error
semantics, ownership and cleanup, and its relationship to next-file warmup.
`docs/source/examples/benchmark.rst` documents the eager/lazy comparison,
full-scan and early-stop
cases, counters and measurement boundaries. It distinguishes tracked pool
allocation from
process RSS and simulated open latency from a complete remote-storage model.
### Generative AI tooling
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]