JustinBinber commented on code in PR #11156:
URL: https://github.com/apache/paimon/pull/11156#discussion_r4630495769
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
plan.watermark(), snapshot == null ? null : snapshot.id(),
snapshot, (List) splits);
}
+ @Override
+ public SnapshotReader withIncrementalPlanning(DataFileMetaStore store,
IOManager ioManager) {
+ if (incrementalStore != null) {
+ throw new IllegalStateException("Incremental metadata resources
are already bound.");
+ }
+ Objects.requireNonNull(store, "Metadata store");
+ Objects.requireNonNull(ioManager, "IO manager");
+ incrementalStore = store;
+ incrementalIoManager = ioManager;
+ return this;
+ }
+
+ @Override
+ public Plan readIncrementally(DataFileMetaStore store, IOManager
ioManager) throws Exception {
+ try {
+ if (!(scan instanceof DataEvolutionFileStoreScan)
+ || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+ || scanMode != ScanMode.ALL) {
+ throw new UnsupportedOperationException(
+ "Incremental metadata planning requires a DE batch ALL
scan.");
+ }
+ IncrementalSplitPlanner planner = new
IncrementalSplitPlanner(store);
+ Snapshot snapshot =
+ ((DataEvolutionFileStoreScan) scan)
+ .forEachFileGroup(ioManager, planner::accept);
Review Comment:
Further corrected in
[`c94e5051d0`](https://github.com/apache/paimon/pull/11156/commits/c94e5051d0cf70e2955e1611a1ab0362bdd7a201).
The initial `62c90e2279` fallback was insufficient: a small row limit does not
bound the metadata in an overlapping manifest group, which the classic planner
materializes before applying the limit. That fallback is now removed.
LIMIT stays on the bounded streaming path. After external sorting, it emits
only the required prefix of complete dependency groups, counting provider rows
conservatively instead of summing overlapping column files. Actual scan
predicates, RowID ranges and deletion vectors disable this truncation,
including filters configured before constructing the snapshot reader. The
producer stops between complete groups, and selected-file metrics and resource
cleanup are preserved. Candidate metadata is still scanned/sorted first; this
does not claim LIMIT scans only the needed manifests or bounds the largest
group.
Added overlapping-manifest/provider regressions with classic `scan.plan()`
explicitly forbidden, LIMIT 1/3 replay-budget and metrics checks, and
preconfigured filter/range controls. Corrected-classpath RED on `62c90e2279` is
28 passed / 4 failed; the fix passes 32/32 planning cases. Final normal JDK8
Maven Core verification is 87/87, and Spark3.5.9 read regressions using the
current Core build are 11/11. The real 29-file LIMIT1 / 1KiB control passes.
The full 32768-file capacity case and performance measurements were not rerun.
[CI for the latest
commit](https://github.com/apache/paimon/actions/runs/44557136921) is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]