JustinBinber commented on code in PR #11156:
URL: https://github.com/apache/paimon/pull/11156#discussion_r4630495769


##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
                 plan.watermark(), snapshot == null ? null : snapshot.id(), 
snapshot, (List) splits);
     }
 
+    @Override
+    public SnapshotReader withIncrementalPlanning(DataFileMetaStore store, 
IOManager ioManager) {
+        if (incrementalStore != null) {
+            throw new IllegalStateException("Incremental metadata resources 
are already bound.");
+        }
+        Objects.requireNonNull(store, "Metadata store");
+        Objects.requireNonNull(ioManager, "IO manager");
+        incrementalStore = store;
+        incrementalIoManager = ioManager;
+        return this;
+    }
+
+    @Override
+    public Plan readIncrementally(DataFileMetaStore store, IOManager 
ioManager) throws Exception {
+        try {
+            if (!(scan instanceof DataEvolutionFileStoreScan)
+                    || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+                    || scanMode != ScanMode.ALL) {
+                throw new UnsupportedOperationException(
+                        "Incremental metadata planning requires a DE batch ALL 
scan.");
+            }
+            IncrementalSplitPlanner planner = new 
IncrementalSplitPlanner(store);
+            Snapshot snapshot =
+                    ((DataEvolutionFileStoreScan) scan)
+                            .forEachFileGroup(ioManager, planner::accept);

Review Comment:
   Further corrected in 
[`c94e5051d0`](https://github.com/apache/paimon/pull/11156/commits/c94e5051d0cf70e2955e1611a1ab0362bdd7a201).
 The initial `62c90e2279` fallback was insufficient: a small row limit does not 
bound the metadata in an overlapping manifest group, which the classic planner 
materializes before applying the limit. That fallback is now removed.
   
   LIMIT stays on the bounded streaming path. After external sorting, it emits 
only the required prefix of complete dependency groups, counting provider rows 
conservatively instead of summing overlapping column files. Actual scan 
predicates, RowID ranges and deletion vectors disable this truncation, 
including filters configured before constructing the snapshot reader. The 
producer stops between complete groups, and selected-file metrics and resource 
cleanup are preserved. Candidate metadata is still scanned/sorted first; this 
does not claim LIMIT scans only the needed manifests or bounds the largest 
group.
   
   Added overlapping-manifest/provider regressions with classic `scan.plan()` 
explicitly forbidden, LIMIT 1/3 replay-budget and metrics checks, and 
preconfigured filter/range controls. Corrected-classpath RED on `62c90e2279` is 
28 passed / 4 failed; the fix passes 32/32 planning cases. Final normal JDK8 
Maven Core verification is 87/87, and Spark3.5.9 read regressions using the 
current Core build are 11/11. The real 29-file LIMIT1 / 1KiB control passes. 
The full 32768-file capacity case and performance measurements were not rerun. 
[CI for the latest 
commit](https://github.com/apache/paimon/actions/runs/44557136921) is pending.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to