kamcheungting-db opened a new pull request, #1113:
URL: https://github.com/apache/iceberg-cpp/pull/1113
## Summary
Introduce `FileScanTaskGroup` and `DataTableScan::PlanTasksStream()` (with
`PlanTasks()` as a collecting convenience), building on the validated byte
ranges of #976 and the range-aware Parquet reads of #1002.
- Files at or below `read.split.target-size` stay whole and go straight to
grouping without a file read. Larger Parquet files use manifest `split_offsets`
as boundaries when they are present, strictly increasing, and in bounds — again
with no file read. Only when the offsets are missing or unusable is the footer
inspected once for canonical row-group starts (`RowGroupStart`, from #1002).
- Boundary intervals coalesce toward the target size; the resulting ranges
partition `[0, file_size)` exactly, including the bytes before the first
boundary. A file with a single row group, or with no boundary that reaches the
target, stays whole. Avro and other formats are not split in this version.
- Tasks pack into groups with Java Iceberg's bin packing: bounded by
`read.split.planning-lookback` open groups, the largest open group closed
first, at most 1,024 tasks per group, and an oversized task as a singleton
group. Each task is weighted like Java's `TableScanUtil.planTasks`:
`max(range length + delete content bytes, (1 + delete file count) *
read.split.open-file-cost)`,
with deletion vectors counting their content size rather than their
container file size, matching `ScanTaskUtil.contentSizeInBytes`.
- Split settings resolve by scan option, then table property, then default
for `read.split.target-size`, `read.split.planning-lookback`, and
`read.split.open-file-cost`. Malformed values and nonpositive target/lookback
or negative open-file cost fail planning with a `Result` error.
- The core group API is format-neutral: split policies are registered per
file format through `internal::SplitPolicyRegistry`, and the Parquet bundle
registers its policy in `RegisterAll()`.
## Stream contract
`PlanTasksStream()` wraps `PlanFilesStream()`: it is fallible, single-pass,
and lazy, and it owns the underlying task stream and its metrics, so it can
outlive the scan. An executor configured through `PlanWith()` is borrowed and
must remain alive until the group stream is destroyed. Destroying the stream
mid-consumption releases the planning resources. Split settings are resolved
before any planning resources are created, so invalid values fail immediately.
## C++ and Java comparison
- The packer is a port of Java's `BinPacking.PackingIterator` with
`largestBinFirst=true`, so C++ and Java produce the same groups from the same
tasks and weights.
- When manifest offsets are unusable, Java's `BaseContentScanTask.split`
falls back to fixed-size byte ranges; this PR instead inspects the footer once
and uses canonical row-group starts. Boundaries need not match read offsets:
#1002 assigns row groups by midpoint, so offsets that disagree with canonical
starts cost balance, not correctness.
- Coalescing happens at split time — every emitted range reaches the target
size except the last — so a large file's splits land in singleton groups while
many small files pack together.
- Java's `ScanTaskUtil.contentSizeInBytes` counts a deletion vector's blob
size; this PR does the same, so delete-heavy tasks weigh the same in both
implementations.
## Testing
* `scan_task_planning_test` (new, bundle-independent; runs with
`ICEBERG_BUILD_BUNDLE=OFF`), fed by synthetic task streams and counting split
policies:
* settings defaults, scan-option-over-table-property precedence, and
malformed/invalid values;
* the Java weight formula, including deletion-vector content size,
open-file cost, and cost-overflow errors;
* split coverage and coalescing: ranges cover `[0, file size)` exactly,
including the prefix before the first boundary; whole-file cases for a small
file, an unsupported format, a single boundary, an empty file, and an
already-split task; splits share the data file, delete files, and residual
filter; policies are not consulted for files at or below the target size;
* packing: first-fit, lookback eviction with the largest group closed
first, oversized singleton groups, the 1,024-task cap, empty streams,
incremental consumption, and early destruction.
* Bundled `scan_test` additions, parameterized over manifest format versions:
* manifest split offsets are used without a footer read (the data path
does not exist, so planning can only succeed without one);
* a footer is read when offsets are missing, producing ranges that cover
the file exactly, each reaching the target except the last; a single-row-group
file stays whole;
* scan option beats table property, and malformed or invalid settings fail
planning;
* no-snapshot scans produce no groups; groups stream incrementally and
survive early destruction; delete files are carried into every split of a file
(format versions 2 and 3; version 1 has no delete manifests).
Validation performed: `scan_task_planning_test` passes in both the core-only
and bundled builds; the full `scan_test` passes; `pre-commit` is clean;
`clang-tidy 22.1.8` (the version CI pins) reports no findings on changed lines.
## Related
* Part-1 #976 — validated `FileScanTask` byte ranges
* Part-2a #1002 — range-aware Parquet task reading
* Part-2b #1062 — stable physical Arrow schema on task streams
* Part-2c #1065 — residual filter enforcement in the task reader
The bounded parallel reader and end-to-end validation follow in later parts
of this series.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
