JingsongLi commented on code in PR #11156:
URL: https://github.com/apache/paimon/pull/11156#discussion_r4624999084


##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -61,6 +69,50 @@ abstract class PaimonBaseScan(table: InnerTable)
     plan.splits().asScala.toArray
   }
 
+  private def getIncrementalInputSplits: Array[Split] = {
+    val io = SparkUtils.createIOManager()
+    var store: DataFileMetaStore = null
+    try {
+      store = new DataFileMetaStore(
+        Paths.get(io.pickTempDir()),
+        coreOptions.dataEvolutionMetadataMaxDiskSize())
+      val scan =
+        readBuilder.newScan(actual => 
actual.newSnapshotReader().withIncrementalPlanning(store, io))

Review Comment:
   **[P1] Keep ordinary fallback-branch and audit reads working in streaming 
mode**
   
   A partitioned Data Evolution table with one row on main and a different 
partition on its fallback branch returns both rows in a plain Spark SELECT with 
the option off. Streaming mode throws `Metadata store is sealed`: the factory 
shares one store, and the main reader seals it before the fallback reader 
appends to it. The Java table control also shows that an audit-log read 
inherits the option but cannot use this factory because the wrapper is not a 
FileStoreTable. Gate the planner to compatible scan/table shapes, or give each 
physical snapshot reader independently owned storage.



##########
paimon-core/src/main/java/org/apache/paimon/operation/FileEntryMergeBuffer.java:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.operation;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.manifest.FileEntry;
+import org.apache.paimon.manifest.FileKind;
+import org.apache.paimon.manifest.SimpleFileEntry;
+import org.apache.paimon.manifest.SimpleFileEntryWithDV;
+import org.apache.paimon.operation.ManifestEntryExternalSort.BinaryRowSorter;
+import 
org.apache.paimon.operation.ManifestEntryExternalSort.ExternalSortConfig;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.IOUtils;
+import org.apache.paimon.utils.MutableObjectIterator;
+import org.apache.paimon.utils.ThrowingConsumer;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.paimon.utils.Preconditions.checkArgument;
+import static org.apache.paimon.utils.Preconditions.checkState;
+import static org.apache.paimon.utils.SerializationUtils.deserializeBinaryRow;
+import static org.apache.paimon.utils.SerializationUtils.serializeBinaryRow;
+
+/**
+ * Spillable equivalent of validating delta entries and then merging base and 
delta entries with
+ * {@link FileEntry#mergeEntries(Iterable)}. Survivors retain the original 
insertion order.
+ *
+ * <p>The two sorters share one configured sort-buffer budget. Reduction 
retains only the current
+ * identity's state and the first invalid histories, rather than a map of all 
file identities. The
+ * encoded-size limit covers input and survivor rows; it is a logical metadata 
budget, not a bound
+ * on physical spill I/O (merging runs may temporarily coexist). A single 
entry still has to fit in
+ * the sort buffer. The caller owns this buffer until consumption finishes, 
including on failure.
+ */
+public final class FileEntryMergeBuffer implements AutoCloseable {
+
+    private static final RowType ROW_TYPE =
+            DataTypes.ROW(
+                    DataTypes.BYTES(), // complete identity, including DV 
subtype
+                    DataTypes.INT(), // base before delta
+                    DataTypes.BIGINT(), // original ordinal within source
+                    DataTypes.TINYINT(), // kind
+                    DataTypes.INT(), // total buckets (not part of identity)
+                    DataTypes.BYTES(), // min key
+                    DataTypes.BYTES(), // max key
+                    DataTypes.BIGINT(), // row count
+                    DataTypes.BIGINT()); // nullable first row id
+
+    private final IOManager ioManager;
+    private final boolean ownedIOManager;
+    private final BinaryRowSorter identities;
+    private final BinaryRowSorter survivors;
+    private final InternalRowSerializer serializer = new 
InternalRowSerializer(ROW_TYPE);
+    private final long maxEncodedSize;
+    private final long[] ordinals = new long[2];
+    private long encodedSize;
+    private boolean finished;
+    private boolean closed;
+    @Nullable private Failure deltaFailure;
+    @Nullable private Failure combinedFailure;
+
+    public FileEntryMergeBuffer(CoreOptions options, @Nullable IOManager 
ioManager)
+            throws Exception {
+        maxEncodedSize = options.dataEvolutionMetadataMaxDiskSize();
+        ExternalSortConfig configured = ExternalSortConfig.from(options, 
ioManager);
+        long pages = configured.bufferSize / configured.pageSize;
+        checkArgument(pages >= 6, "Metadata merge requires at least six 
sort-buffer pages.");

Review Comment:
   **[P2] Handle valid four-page sort buffers before committing updates**
   
   I used `page-size=64 kb` and `sort-spill-buffer-size=256 kb` on real Data 
Evolution tables. All modes accept the schema and publish the first row. A 
one-column update then publishes snapshot 2 in off and lightweight modes, while 
streaming fails with `Metadata merge requires at least six sort-buffer pages`. 
Keep the legacy conflict check for smaller buffers or validate and document the 
minimum when streaming is enabled.



##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
                 plan.watermark(), snapshot == null ? null : snapshot.id(), 
snapshot, (List) splits);
     }
 
+    @Override
+    public SnapshotReader withIncrementalPlanning(DataFileMetaStore store, 
IOManager ioManager) {
+        if (incrementalStore != null) {
+            throw new IllegalStateException("Incremental metadata resources 
are already bound.");
+        }
+        Objects.requireNonNull(store, "Metadata store");
+        Objects.requireNonNull(ioManager, "IO manager");
+        incrementalStore = store;
+        incrementalIoManager = ioManager;
+        return this;
+    }
+
+    @Override
+    public Plan readIncrementally(DataFileMetaStore store, IOManager 
ioManager) throws Exception {
+        try {
+            if (!(scan instanceof DataEvolutionFileStoreScan)
+                    || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+                    || scanMode != ScanMode.ALL) {
+                throw new UnsupportedOperationException(
+                        "Incremental metadata planning requires a DE batch ALL 
scan.");
+            }
+            IncrementalSplitPlanner planner = new 
IncrementalSplitPlanner(store);
+            Snapshot snapshot =
+                    ((DataEvolutionFileStoreScan) scan)
+                            .forEachFileGroup(ioManager, planner::accept);

Review Comment:
   **[P2] Retain limit-aware planning for bounded reads**
   
   On a real Data Evolution table with 29 separate one-row files, no data 
predicate or deletion vectors, and `withLimit(1)`, the classic planner produced 
one split and returned the first row. The streaming planner replayed all groups 
and exceeded a 1 KiB metadata disk budget. Keep the existing early limit under 
its safe conditions, or fall back to the normal planner for those bounded scans.



##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
                 plan.watermark(), snapshot == null ? null : snapshot.id(), 
snapshot, (List) splits);
     }
 
+    @Override
+    public SnapshotReader withIncrementalPlanning(DataFileMetaStore store, 
IOManager ioManager) {
+        if (incrementalStore != null) {
+            throw new IllegalStateException("Incremental metadata resources 
are already bound.");
+        }
+        Objects.requireNonNull(store, "Metadata store");
+        Objects.requireNonNull(ioManager, "IO manager");
+        incrementalStore = store;
+        incrementalIoManager = ioManager;
+        return this;
+    }
+
+    @Override
+    public Plan readIncrementally(DataFileMetaStore store, IOManager 
ioManager) throws Exception {
+        try {
+            if (!(scan instanceof DataEvolutionFileStoreScan)
+                    || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+                    || scanMode != ScanMode.ALL) {
+                throw new UnsupportedOperationException(
+                        "Incremental metadata planning requires a DE batch ALL 
scan.");
+            }
+            IncrementalSplitPlanner planner = new 
IncrementalSplitPlanner(store);
+            Snapshot snapshot =
+                    ((DataEvolutionFileStoreScan) scan)
+                            .forEachFileGroup(ioManager, planner::accept);
+            planner.finish();
+            store.seal();
+            if (options.scanPlanSortPartition()) {
+                planner.splits.sort(
+                        (left, right) ->
+                                partitionComparator()
+                                        .compare(left.bucket.partition, 
right.bucket.partition));
+            }
+            Set<Pair<BinaryRow, Integer>> buckets = new HashSet<>();
+            for (SpilledSplit split : planner.splits) {
+                buckets.add(Pair.of(split.bucket.partition, 
split.bucket.bucket));
+            }
+            Map<Pair<BinaryRow, Integer>, Map<String, DeletionFile>> deletions 
=
+                    deletionVectors ? scanDvIndex(snapshot, buckets) : 
Collections.emptyMap();
+            List<Split> splits = new ArrayList<>();
+            for (SpilledSplit split : planner.splits) {
+                List<DataFileMeta> files = store.slice(split.start, 
split.start + split.count);
+                IncrementalBucket bucket = split.bucket;
+                DataSplit.Builder builder =
+                        DataSplit.builder()
+                                .withSnapshot(
+                                        snapshot == null ? FIRST_SNAPSHOT_ID - 
1 : snapshot.id())
+                                .withPartition(bucket.partition)
+                                .withBucket(bucket.bucket)
+                                .withTotalBuckets(bucket.totalBuckets)
+                                .withBucketPath(bucket.path)
+                                .withDataFiles(files)
+                                .rawConvertible(split.rawConvertible);
+                if (deletionVectors) {
+                    builder.withDataDeletionFiles(
+                            getDeletionFiles(
+                                    files,
+                                    deletions.getOrDefault(
+                                            Pair.of(bucket.partition, 
bucket.bucket),
+                                            Collections.emptyMap())));
+                }
+                splits.add(builder.build());
+            }
+            return new PlanImpl(
+                    snapshot == null ? null : snapshot.watermark(),
+                    snapshot == null ? null : snapshot.id(),
+                    snapshot,
+                    splits);
+        } catch (Exception | Error failure) {
+            try {
+                store.close();
+            } catch (IOException cleanupFailure) {
+                failure.addSuppressed(cleanupFailure);
+            }
+            throw failure;
+        }
+    }
+
+    private final class IncrementalSplitPlanner {
+        private final DataFileMetaStore store;
+        private final List<SpilledSplit> splits = new ArrayList<>();
+        private IncrementalBucket current;
+        private DataEvolutionSplitGenerator.BatchPacker<IOException> packer;
+        private int nextFile;
+
+        private IncrementalSplitPlanner(DataFileMetaStore store) {
+            this.store = store;
+        }
+
+        private void accept(List<ManifestEntry> group) throws IOException {
+            // Partition only after the complete global group has been 
filtered. Providers from
+            // one partition may be interleaved with another partition in the 
group's source order.
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> partitions = 
groupByPartFiles(group);
+            for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> 
partition :
+                    partitions.entrySet()) {
+                for (Map.Entry<Integer, List<ManifestEntry>> bucket :
+                        partition.getValue().entrySet()) {
+                    if (current == null
+                            || current.bucket != bucket.getKey()
+                            || !current.partition.equals(partition.getKey())) {
+                        finish();

Review Comment:
   **[P2] Preserve split coalescing across interleaved partitions**
   
   Global row-ID order can alternate partitions across successive commits. Each 
transition finishes this packer before it can coalesce later small files for 
the same partition. With two seed files and 24 one-row Parquet commits 
alternating two partitions, the same table and default sizing produced 3 splits 
with classic planning and 25 with streaming. Consider bounded per-partition 
packing or a partition-local spill ordering.



##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -61,6 +69,50 @@ abstract class PaimonBaseScan(table: InnerTable)
     plan.splits().asScala.toArray
   }
 
+  private def getIncrementalInputSplits: Array[Split] = {
+    val io = SparkUtils.createIOManager()
+    var store: DataFileMetaStore = null
+    try {
+      store = new DataFileMetaStore(
+        Paths.get(io.pickTempDir()),
+        coreOptions.dataEvolutionMetadataMaxDiskSize())
+      val scan =
+        readBuilder.newScan(actual => 
actual.newSnapshotReader().withIncrementalPlanning(store, io))
+      val splits = planInputSplits(scan.asInstanceOf[InnerTableScan])
+      if (splits.isEmpty) {
+        // Empty-table planning may finish before the snapshot reader seals 
its store.
+        store.close()
+        io.close()
+        return splits
+      }
+      // Sort spill is finished; replay files remain owned by the returned 
split slices.
+      io.close()
+      store.withAutomaticCleanup()

Review Comment:
   **[P1] Preserve nonempty DELTA incremental reads**
   
   On Spark 3.5.9 I committed two rows to a Data Evolution table and read it 
with `incremental-between=1,2` and `incremental-between-scan-mode=delta`. Off 
mode returns the second row. Enabling streaming alone throws `Metadata store 
must be sealed before reading`. The DELTA planner returns regular splits 
without sealing the unused incremental store. Route DELTA through the normal 
planner and cover it with a nonempty Spark read.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to