JingsongLi commented on code in PR #11051:
URL: https://github.com/apache/paimon/pull/11051#discussion_r4522614628


##########
paimon-core/src/main/java/org/apache/paimon/operation/KeyValueFileStoreWrite.java:
##########
@@ -254,7 +255,9 @@ protected MergeTreeWriter createWriter(
 
     @Override
     protected boolean requiresCompactionBaseline() {
-        return options.deletionVectorsEnabled() && !options.writeOnly();
+        return !options.writeOnly()

Review Comment:
   **[P1] Keep write-only ingestion working after enabling the vector index**
   
   I extended the promoted-file integration fixture on `28bc1dd077`. I wrote 
and compacted a DV-enabled PK table before enabling the index, then inserted a 
row with `write-only=true`. The insert failed before the APPEND commit with `A 
primary-key compaction or vector backfill message is missing its restored 
snapshot`. The index maintainer restores the promoted APPEND file and generates 
an index-only COMPACT increment during ordinary prepareCommit. The writer omits 
the snapshot baseline here. Could we keep promoted-file backfill on the 
compaction path for write-only writers and add this insert as a regression case?



##########
paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeyVectorScan.java:
##########
@@ -135,7 +135,25 @@ public Plan scan() {
                                                 == vectorFieldId
                                         && (partitionFilter == null
                                                 || 
partitionFilter.test(entry.partition())));
-        return plan(snapshot.id(), snapshotPlan.splits(), vectorIndexEntries);
+        if (filter == null || snapshotPlan.splits().isEmpty()) {
+            return plan(snapshot.id(), snapshotPlan.splits(), 
vectorIndexEntries);
+        }
+
+        // Index validity and direct-read eligibility depend on the complete 
bucket. Keep
+        // pruned files in the plan with empty row ranges so they cannot 
produce candidates.
+        SnapshotReader completeReader =

Review Comment:
   **[P2] Bound the second metadata scan to partitions and buckets selected by 
the first plan**
   
   A vector query with a partition predicate in `withFilter` prunes that 
partition through `SnapshotReader.withFilter`. The new complete-reader pass 
only applies a separately supplied partition filter and drops bucket pruning as 
well. It reads and assembles splits for all other partitions or buckets before 
discarding them below. This makes a selective query do table-wide metadata work 
on the workload described in #11047. Could the second pass use the 
partition/bucket keys of the surviving splits, while retaining every file 
within those keys? The Python second scan has the same issue.



##########
paimon-core/src/main/java/org/apache/paimon/operation/commit/SnapshotConflictChecker.java:
##########
@@ -197,14 +226,19 @@ private void checkBucketCompaction(
                 ranges.values().stream().mapToLong(range -> 
range.to).min().getAsLong() + 1;
         for (long id = firstSnapshot; id < newSnapshotId; id++) {
             Snapshot snapshot = snapshotManager.snapshot(id);
+            boolean compactOrOverwrite =
+                    snapshot.commitKind() == CommitKind.COMPACT
+                            || snapshot.commitKind() == CommitKind.OVERWRITE;
             if (snapshot.commitUser().equals(commitUser)
-                    || (snapshot.commitKind() != CommitKind.COMPACT
-                            && snapshot.commitKind() != CommitKind.OVERWRITE)) 
{
+                    || (!compactOrOverwrite
+                            && (snapshot.commitKind() != CommitKind.APPEND
+                                    || vectorBackfills.isEmpty()))) {
                 continue;
             }
             Set<Pair<BinaryRow, Integer>> buckets = new HashSet<>();
             for (Map.Entry<Pair<BinaryRow, Integer>, Range> entry : 
ranges.entrySet()) {
-                if (entry.getValue().to < id) {
+                if (entry.getValue().to < id

Review Comment:
   **[P2] Avoid rejecting backfills for appends outside their source set**
   
   With deletion vectors enabled and merge-on-read disabled, the batch view 
excludes L0. A concurrent writer adding only an L0 file in the same bucket 
changes neither the backfill's visible data nor its deletion vectors or payload 
coverage, but the bucket-wide APPEND check still rejects the backfill. On a 
continuously ingested bucket, a build longer than the checkpoint interval can 
fail on every attempt. Could the conflict check compare the eligible source set 
and its DV/index state, or exempt a pure L0 append when the batch view excludes 
it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to