JustinBinber commented on code in PR #11156:
URL: https://github.com/apache/paimon/pull/11156#discussion_r4630494696
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -61,6 +69,50 @@ abstract class PaimonBaseScan(table: InnerTable)
plan.splits().asScala.toArray
}
+ private def getIncrementalInputSplits: Array[Split] = {
+ val io = SparkUtils.createIOManager()
+ var store: DataFileMetaStore = null
+ try {
+ store = new DataFileMetaStore(
+ Paths.get(io.pickTempDir()),
+ coreOptions.dataEvolutionMetadataMaxDiskSize())
+ val scan =
+ readBuilder.newScan(actual =>
actual.newSnapshotReader().withIncrementalPlanning(store, io))
Review Comment:
Fixed in
[`62c90e2279`](https://github.com/apache/paimon/pull/11156/commits/62c90e227975ef2e4345ffce9a8ed6f3e46a5d33).
Replay planning is now gated to compatible single-snapshot FileStore reads.
Fallback/chain tables and non-FileStore wrappers such as audit-log keep their
existing planner, so multiple physical readers no longer append to one sealed
store and audit-log never receives an unsupported snapshot-reader factory. This
preserves their original memory boundary rather than claiming replay support
for these shapes.
Real nonempty Spark regressions compare fallback values/RowIDs and partition
precedence, filtered fallback reads and audit-log rows against the ordinary
path. Both reproduced on the original implementation and pass after the change.
The combined Spark suite passes 11/11 on JDK8/Spark3.5.9. The upstream content
conflict has also been resolved.
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -61,6 +69,50 @@ abstract class PaimonBaseScan(table: InnerTable)
plan.splits().asScala.toArray
}
+ private def getIncrementalInputSplits: Array[Split] = {
+ val io = SparkUtils.createIOManager()
+ var store: DataFileMetaStore = null
+ try {
+ store = new DataFileMetaStore(
+ Paths.get(io.pickTempDir()),
+ coreOptions.dataEvolutionMetadataMaxDiskSize())
+ val scan =
+ readBuilder.newScan(actual =>
actual.newSnapshotReader().withIncrementalPlanning(store, io))
+ val splits = planInputSplits(scan.asInstanceOf[InnerTableScan])
+ if (splits.isEmpty) {
+ // Empty-table planning may finish before the snapshot reader seals
its store.
+ store.close()
+ io.close()
+ return splits
+ }
+ // Sort spill is finished; replay files remain owned by the returned
split slices.
+ io.close()
+ store.withAutomaticCleanup()
Review Comment:
Fixed in
[`62c90e2279`](https://github.com/apache/paimon/pull/11156/commits/62c90e227975ef2e4345ffce9a8ed6f3e46a5d33).
Incremental and latest-delta scan modes now use the ordinary planner instead
of installing an unused or incompatible replay store. Internal batch-scan modes
also retain their planner. The ordinary single-snapshot ALL path still uses
replay planning.
Added real two-commit Spark reads for explicit DELTA, AUTO (which selects
DELTA here), and latest-delta. Off mode returns the second row; the old
streaming path fails; the corrected streaming reads return that same row and
leave snapshot 2 unchanged. Together with fallback/audit and existing replay
read controls, the final Spark suite passes 11/11.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
plan.watermark(), snapshot == null ? null : snapshot.id(),
snapshot, (List) splits);
}
+ @Override
+ public SnapshotReader withIncrementalPlanning(DataFileMetaStore store,
IOManager ioManager) {
+ if (incrementalStore != null) {
+ throw new IllegalStateException("Incremental metadata resources
are already bound.");
+ }
+ Objects.requireNonNull(store, "Metadata store");
+ Objects.requireNonNull(ioManager, "IO manager");
+ incrementalStore = store;
+ incrementalIoManager = ioManager;
+ return this;
+ }
+
+ @Override
+ public Plan readIncrementally(DataFileMetaStore store, IOManager
ioManager) throws Exception {
+ try {
+ if (!(scan instanceof DataEvolutionFileStoreScan)
+ || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+ || scanMode != ScanMode.ALL) {
+ throw new UnsupportedOperationException(
+ "Incremental metadata planning requires a DE batch ALL
scan.");
+ }
+ IncrementalSplitPlanner planner = new
IncrementalSplitPlanner(store);
+ Snapshot snapshot =
+ ((DataEvolutionFileStoreScan) scan)
+ .forEachFileGroup(ioManager, planner::accept);
Review Comment:
Fixed in
[`62c90e2279`](https://github.com/apache/paimon/pull/11156/commits/62c90e227975ef2e4345ffce9a8ed6f3e46a5d33).
Small positive limits reuse the classic planner's early termination only when
there are no data predicates, RowID restrictions or deletion vectors. This
reuse is limited to row limits up to 1024; larger limits keep streamed
dependency-group planning, avoiding an unrestricted materialization fallback.
The real 29-file LIMIT 1 regression reproduces the old replay-budget failure
and now produces one split, returns the same first row, and stays within 1 KiB.
Separate filtered/RowID-range controls check that rows excluded from the result
are not counted toward the limit. All 81 selected Core cases pass through the
normal JDK8 Maven reactor, including the new real-table cases and existing
incremental planning/cleanup regressions.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
plan.watermark(), snapshot == null ? null : snapshot.id(),
snapshot, (List) splits);
}
+ @Override
+ public SnapshotReader withIncrementalPlanning(DataFileMetaStore store,
IOManager ioManager) {
+ if (incrementalStore != null) {
+ throw new IllegalStateException("Incremental metadata resources
are already bound.");
+ }
+ Objects.requireNonNull(store, "Metadata store");
+ Objects.requireNonNull(ioManager, "IO manager");
+ incrementalStore = store;
+ incrementalIoManager = ioManager;
+ return this;
+ }
+
+ @Override
+ public Plan readIncrementally(DataFileMetaStore store, IOManager
ioManager) throws Exception {
+ try {
+ if (!(scan instanceof DataEvolutionFileStoreScan)
+ || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+ || scanMode != ScanMode.ALL) {
+ throw new UnsupportedOperationException(
+ "Incremental metadata planning requires a DE batch ALL
scan.");
+ }
+ IncrementalSplitPlanner planner = new
IncrementalSplitPlanner(store);
+ Snapshot snapshot =
+ ((DataEvolutionFileStoreScan) scan)
+ .forEachFileGroup(ioManager, planner::accept);
+ planner.finish();
+ store.seal();
+ if (options.scanPlanSortPartition()) {
+ planner.splits.sort(
+ (left, right) ->
+ partitionComparator()
+ .compare(left.bucket.partition,
right.bucket.partition));
+ }
+ Set<Pair<BinaryRow, Integer>> buckets = new HashSet<>();
+ for (SpilledSplit split : planner.splits) {
+ buckets.add(Pair.of(split.bucket.partition,
split.bucket.bucket));
+ }
+ Map<Pair<BinaryRow, Integer>, Map<String, DeletionFile>> deletions
=
+ deletionVectors ? scanDvIndex(snapshot, buckets) :
Collections.emptyMap();
+ List<Split> splits = new ArrayList<>();
+ for (SpilledSplit split : planner.splits) {
+ List<DataFileMeta> files = store.slice(split.start,
split.start + split.count);
+ IncrementalBucket bucket = split.bucket;
+ DataSplit.Builder builder =
+ DataSplit.builder()
+ .withSnapshot(
+ snapshot == null ? FIRST_SNAPSHOT_ID -
1 : snapshot.id())
+ .withPartition(bucket.partition)
+ .withBucket(bucket.bucket)
+ .withTotalBuckets(bucket.totalBuckets)
+ .withBucketPath(bucket.path)
+ .withDataFiles(files)
+ .rawConvertible(split.rawConvertible);
+ if (deletionVectors) {
+ builder.withDataDeletionFiles(
+ getDeletionFiles(
+ files,
+ deletions.getOrDefault(
+ Pair.of(bucket.partition,
bucket.bucket),
+ Collections.emptyMap())));
+ }
+ splits.add(builder.build());
+ }
+ return new PlanImpl(
+ snapshot == null ? null : snapshot.watermark(),
+ snapshot == null ? null : snapshot.id(),
+ snapshot,
+ splits);
+ } catch (Exception | Error failure) {
+ try {
+ store.close();
+ } catch (IOException cleanupFailure) {
+ failure.addSuppressed(cleanupFailure);
+ }
+ throw failure;
+ }
+ }
+
+ private final class IncrementalSplitPlanner {
+ private final DataFileMetaStore store;
+ private final List<SpilledSplit> splits = new ArrayList<>();
+ private IncrementalBucket current;
+ private DataEvolutionSplitGenerator.BatchPacker<IOException> packer;
+ private int nextFile;
+
+ private IncrementalSplitPlanner(DataFileMetaStore store) {
+ this.store = store;
+ }
+
+ private void accept(List<ManifestEntry> group) throws IOException {
+ // Partition only after the complete global group has been
filtered. Providers from
+ // one partition may be interleaved with another partition in the
group's source order.
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> partitions =
groupByPartFiles(group);
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>>
partition :
+ partitions.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
+ partition.getValue().entrySet()) {
+ if (current == null
+ || current.bucket != bucket.getKey()
+ || !current.partition.equals(partition.getKey())) {
+ finish();
Review Comment:
Fixed in
[`62c90e2279`](https://github.com/apache/paimon/pull/11156/commits/62c90e227975ef2e4345ffce9a8ed6f3e46a5d33).
The streaming planner now keeps a bounded LRU of partition/bucket packers
under a shared pending-file count instead of finishing a packer at every
partition transition. Eviction finishes complete provider groups; the existing
1024-file threshold is shared across packers, and an oversized connected group
is still kept intact.
Real alternating-partition Parquet controls now produce the same split count
and rows as classic planning, with deletion vectors both disabled and enabled.
Added an eviction case checking every complete provider group survives the
global bound. The 31 incremental/planning cases pass, and the normal Core
reactor passes all 81 selected cases. No new large-workload timing claim is
made from these controls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]