JingsongLi commented on code in PR #11156:
URL: https://github.com/apache/paimon/pull/11156#discussion_r4624999084
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -61,6 +69,50 @@ abstract class PaimonBaseScan(table: InnerTable)
plan.splits().asScala.toArray
}
+ private def getIncrementalInputSplits: Array[Split] = {
+ val io = SparkUtils.createIOManager()
+ var store: DataFileMetaStore = null
+ try {
+ store = new DataFileMetaStore(
+ Paths.get(io.pickTempDir()),
+ coreOptions.dataEvolutionMetadataMaxDiskSize())
+ val scan =
+ readBuilder.newScan(actual =>
actual.newSnapshotReader().withIncrementalPlanning(store, io))
Review Comment:
**[P1] Keep ordinary fallback-branch and audit reads working in streaming
mode**
A partitioned Data Evolution table with one row on main and a different
partition on its fallback branch returns both rows in a plain Spark SELECT with
the option off. Streaming mode throws `Metadata store is sealed`: the factory
shares one store, and the main reader seals it before the fallback reader
appends to it. The Java table control also shows that an audit-log read
inherits the option but cannot use this factory because the wrapper is not a
FileStoreTable. Gate the planner to compatible scan/table shapes, or give each
physical snapshot reader independently owned storage.
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileEntryMergeBuffer.java:
##########
@@ -0,0 +1,433 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.operation;
+
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.data.BinaryRow;
+import org.apache.paimon.data.GenericRow;
+import org.apache.paimon.data.serializer.InternalRowSerializer;
+import org.apache.paimon.disk.IOManager;
+import org.apache.paimon.manifest.FileEntry;
+import org.apache.paimon.manifest.FileKind;
+import org.apache.paimon.manifest.SimpleFileEntry;
+import org.apache.paimon.manifest.SimpleFileEntryWithDV;
+import org.apache.paimon.operation.ManifestEntryExternalSort.BinaryRowSorter;
+import
org.apache.paimon.operation.ManifestEntryExternalSort.ExternalSortConfig;
+import org.apache.paimon.types.DataTypes;
+import org.apache.paimon.types.RowType;
+import org.apache.paimon.utils.IOUtils;
+import org.apache.paimon.utils.MutableObjectIterator;
+import org.apache.paimon.utils.ThrowingConsumer;
+
+import javax.annotation.Nullable;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.apache.paimon.utils.Preconditions.checkArgument;
+import static org.apache.paimon.utils.Preconditions.checkState;
+import static org.apache.paimon.utils.SerializationUtils.deserializeBinaryRow;
+import static org.apache.paimon.utils.SerializationUtils.serializeBinaryRow;
+
+/**
+ * Spillable equivalent of validating delta entries and then merging base and
delta entries with
+ * {@link FileEntry#mergeEntries(Iterable)}. Survivors retain the original
insertion order.
+ *
+ * <p>The two sorters share one configured sort-buffer budget. Reduction
retains only the current
+ * identity's state and the first invalid histories, rather than a map of all
file identities. The
+ * encoded-size limit covers input and survivor rows; it is a logical metadata
budget, not a bound
+ * on physical spill I/O (merging runs may temporarily coexist). A single
entry still has to fit in
+ * the sort buffer. The caller owns this buffer until consumption finishes,
including on failure.
+ */
+public final class FileEntryMergeBuffer implements AutoCloseable {
+
+ private static final RowType ROW_TYPE =
+ DataTypes.ROW(
+ DataTypes.BYTES(), // complete identity, including DV
subtype
+ DataTypes.INT(), // base before delta
+ DataTypes.BIGINT(), // original ordinal within source
+ DataTypes.TINYINT(), // kind
+ DataTypes.INT(), // total buckets (not part of identity)
+ DataTypes.BYTES(), // min key
+ DataTypes.BYTES(), // max key
+ DataTypes.BIGINT(), // row count
+ DataTypes.BIGINT()); // nullable first row id
+
+ private final IOManager ioManager;
+ private final boolean ownedIOManager;
+ private final BinaryRowSorter identities;
+ private final BinaryRowSorter survivors;
+ private final InternalRowSerializer serializer = new
InternalRowSerializer(ROW_TYPE);
+ private final long maxEncodedSize;
+ private final long[] ordinals = new long[2];
+ private long encodedSize;
+ private boolean finished;
+ private boolean closed;
+ @Nullable private Failure deltaFailure;
+ @Nullable private Failure combinedFailure;
+
+ public FileEntryMergeBuffer(CoreOptions options, @Nullable IOManager
ioManager)
+ throws Exception {
+ maxEncodedSize = options.dataEvolutionMetadataMaxDiskSize();
+ ExternalSortConfig configured = ExternalSortConfig.from(options,
ioManager);
+ long pages = configured.bufferSize / configured.pageSize;
+ checkArgument(pages >= 6, "Metadata merge requires at least six
sort-buffer pages.");
Review Comment:
**[P2] Handle valid four-page sort buffers before committing updates**
I used `page-size=64 kb` and `sort-spill-buffer-size=256 kb` on real Data
Evolution tables. All modes accept the schema and publish the first row. A
one-column update then publishes snapshot 2 in off and lightweight modes, while
streaming fails with `Metadata merge requires at least six sort-buffer pages`.
Keep the legacy conflict check for smaller buffers or validate and document the
minimum when streaming is enabled.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
plan.watermark(), snapshot == null ? null : snapshot.id(),
snapshot, (List) splits);
}
+ @Override
+ public SnapshotReader withIncrementalPlanning(DataFileMetaStore store,
IOManager ioManager) {
+ if (incrementalStore != null) {
+ throw new IllegalStateException("Incremental metadata resources
are already bound.");
+ }
+ Objects.requireNonNull(store, "Metadata store");
+ Objects.requireNonNull(ioManager, "IO manager");
+ incrementalStore = store;
+ incrementalIoManager = ioManager;
+ return this;
+ }
+
+ @Override
+ public Plan readIncrementally(DataFileMetaStore store, IOManager
ioManager) throws Exception {
+ try {
+ if (!(scan instanceof DataEvolutionFileStoreScan)
+ || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+ || scanMode != ScanMode.ALL) {
+ throw new UnsupportedOperationException(
+ "Incremental metadata planning requires a DE batch ALL
scan.");
+ }
+ IncrementalSplitPlanner planner = new
IncrementalSplitPlanner(store);
+ Snapshot snapshot =
+ ((DataEvolutionFileStoreScan) scan)
+ .forEachFileGroup(ioManager, planner::accept);
Review Comment:
**[P2] Retain limit-aware planning for bounded reads**
On a real Data Evolution table with 29 separate one-row files, no data
predicate or deletion vectors, and `withLimit(1)`, the classic planner produced
one split and returned the first row. The streaming planner replayed all groups
and exceeded a 1 KiB metadata disk budget. Keep the existing early limit under
its safe conditions, or fall back to the normal planner for those bounded scans.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -411,6 +436,181 @@ public Plan read() {
plan.watermark(), snapshot == null ? null : snapshot.id(),
snapshot, (List) splits);
}
+ @Override
+ public SnapshotReader withIncrementalPlanning(DataFileMetaStore store,
IOManager ioManager) {
+ if (incrementalStore != null) {
+ throw new IllegalStateException("Incremental metadata resources
are already bound.");
+ }
+ Objects.requireNonNull(store, "Metadata store");
+ Objects.requireNonNull(ioManager, "IO manager");
+ incrementalStore = store;
+ incrementalIoManager = ioManager;
+ return this;
+ }
+
+ @Override
+ public Plan readIncrementally(DataFileMetaStore store, IOManager
ioManager) throws Exception {
+ try {
+ if (!(scan instanceof DataEvolutionFileStoreScan)
+ || !(splitGenerator instanceof DataEvolutionSplitGenerator)
+ || scanMode != ScanMode.ALL) {
+ throw new UnsupportedOperationException(
+ "Incremental metadata planning requires a DE batch ALL
scan.");
+ }
+ IncrementalSplitPlanner planner = new
IncrementalSplitPlanner(store);
+ Snapshot snapshot =
+ ((DataEvolutionFileStoreScan) scan)
+ .forEachFileGroup(ioManager, planner::accept);
+ planner.finish();
+ store.seal();
+ if (options.scanPlanSortPartition()) {
+ planner.splits.sort(
+ (left, right) ->
+ partitionComparator()
+ .compare(left.bucket.partition,
right.bucket.partition));
+ }
+ Set<Pair<BinaryRow, Integer>> buckets = new HashSet<>();
+ for (SpilledSplit split : planner.splits) {
+ buckets.add(Pair.of(split.bucket.partition,
split.bucket.bucket));
+ }
+ Map<Pair<BinaryRow, Integer>, Map<String, DeletionFile>> deletions
=
+ deletionVectors ? scanDvIndex(snapshot, buckets) :
Collections.emptyMap();
+ List<Split> splits = new ArrayList<>();
+ for (SpilledSplit split : planner.splits) {
+ List<DataFileMeta> files = store.slice(split.start,
split.start + split.count);
+ IncrementalBucket bucket = split.bucket;
+ DataSplit.Builder builder =
+ DataSplit.builder()
+ .withSnapshot(
+ snapshot == null ? FIRST_SNAPSHOT_ID -
1 : snapshot.id())
+ .withPartition(bucket.partition)
+ .withBucket(bucket.bucket)
+ .withTotalBuckets(bucket.totalBuckets)
+ .withBucketPath(bucket.path)
+ .withDataFiles(files)
+ .rawConvertible(split.rawConvertible);
+ if (deletionVectors) {
+ builder.withDataDeletionFiles(
+ getDeletionFiles(
+ files,
+ deletions.getOrDefault(
+ Pair.of(bucket.partition,
bucket.bucket),
+ Collections.emptyMap())));
+ }
+ splits.add(builder.build());
+ }
+ return new PlanImpl(
+ snapshot == null ? null : snapshot.watermark(),
+ snapshot == null ? null : snapshot.id(),
+ snapshot,
+ splits);
+ } catch (Exception | Error failure) {
+ try {
+ store.close();
+ } catch (IOException cleanupFailure) {
+ failure.addSuppressed(cleanupFailure);
+ }
+ throw failure;
+ }
+ }
+
+ private final class IncrementalSplitPlanner {
+ private final DataFileMetaStore store;
+ private final List<SpilledSplit> splits = new ArrayList<>();
+ private IncrementalBucket current;
+ private DataEvolutionSplitGenerator.BatchPacker<IOException> packer;
+ private int nextFile;
+
+ private IncrementalSplitPlanner(DataFileMetaStore store) {
+ this.store = store;
+ }
+
+ private void accept(List<ManifestEntry> group) throws IOException {
+ // Partition only after the complete global group has been
filtered. Providers from
+ // one partition may be interleaved with another partition in the
group's source order.
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> partitions =
groupByPartFiles(group);
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>>
partition :
+ partitions.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
+ partition.getValue().entrySet()) {
+ if (current == null
+ || current.bucket != bucket.getKey()
+ || !current.partition.equals(partition.getKey())) {
+ finish();
Review Comment:
**[P2] Preserve split coalescing across interleaved partitions**
Global row-ID order can alternate partitions across successive commits. Each
transition finishes this packer before it can coalesce later small files for
the same partition. With two seed files and 24 one-row Parquet commits
alternating two partitions, the same table and default sizing produced 3 splits
with classic planning and 25 with streaming. Consider bounded per-partition
packing or a partition-local spill ordering.
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -61,6 +69,50 @@ abstract class PaimonBaseScan(table: InnerTable)
plan.splits().asScala.toArray
}
+ private def getIncrementalInputSplits: Array[Split] = {
+ val io = SparkUtils.createIOManager()
+ var store: DataFileMetaStore = null
+ try {
+ store = new DataFileMetaStore(
+ Paths.get(io.pickTempDir()),
+ coreOptions.dataEvolutionMetadataMaxDiskSize())
+ val scan =
+ readBuilder.newScan(actual =>
actual.newSnapshotReader().withIncrementalPlanning(store, io))
+ val splits = planInputSplits(scan.asInstanceOf[InnerTableScan])
+ if (splits.isEmpty) {
+ // Empty-table planning may finish before the snapshot reader seals
its store.
+ store.close()
+ io.close()
+ return splits
+ }
+ // Sort spill is finished; replay files remain owned by the returned
split slices.
+ io.close()
+ store.withAutomaticCleanup()
Review Comment:
**[P1] Preserve nonempty DELTA incremental reads**
On Spark 3.5.9 I committed two rows to a Data Evolution table and read it
with `incremental-between=1,2` and `incremental-between-scan-mode=delta`. Off
mode returns the second row. Enabling streaming alone throws `Metadata store
must be sealed before reading`. The DELTA planner returns regular splits
without sealing the unused incremental store. Route DELTA through the normal
planner and cover it with a nonempty Spark read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]