JingsongLi commented on code in PR #11051:
URL: https://github.com/apache/paimon/pull/11051#discussion_r4522614628
##########
paimon-core/src/main/java/org/apache/paimon/operation/KeyValueFileStoreWrite.java:
##########
@@ -254,7 +255,9 @@ protected MergeTreeWriter createWriter(
@Override
protected boolean requiresCompactionBaseline() {
- return options.deletionVectorsEnabled() && !options.writeOnly();
+ return !options.writeOnly()
Review Comment:
**[P1] Keep write-only ingestion working after enabling the vector index**
I extended the promoted-file integration fixture on `28bc1dd077`. I wrote
and compacted a DV-enabled PK table before enabling the index, then inserted a
row with `write-only=true`. The insert failed before the APPEND commit with `A
primary-key compaction or vector backfill message is missing its restored
snapshot`. The index maintainer restores the promoted APPEND file and generates
an index-only COMPACT increment during ordinary prepareCommit. The writer omits
the snapshot baseline here. Could we keep promoted-file backfill on the
compaction path for write-only writers and add this insert as a regression case?
##########
paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeyVectorScan.java:
##########
@@ -135,7 +135,25 @@ public Plan scan() {
== vectorFieldId
&& (partitionFilter == null
||
partitionFilter.test(entry.partition())));
- return plan(snapshot.id(), snapshotPlan.splits(), vectorIndexEntries);
+ if (filter == null || snapshotPlan.splits().isEmpty()) {
+ return plan(snapshot.id(), snapshotPlan.splits(),
vectorIndexEntries);
+ }
+
+ // Index validity and direct-read eligibility depend on the complete
bucket. Keep
+ // pruned files in the plan with empty row ranges so they cannot
produce candidates.
+ SnapshotReader completeReader =
Review Comment:
**[P2] Bound the second metadata scan to partitions and buckets selected by
the first plan**
A vector query with a partition predicate in `withFilter` prunes that
partition through `SnapshotReader.withFilter`. The new complete-reader pass
only applies a separately supplied partition filter and drops bucket pruning as
well. It reads and assembles splits for all other partitions or buckets before
discarding them below. This makes a selective query do table-wide metadata work
on the workload described in #11047. Could the second pass use the
partition/bucket keys of the surviving splits, while retaining every file
within those keys? The Python second scan has the same issue.
##########
paimon-core/src/main/java/org/apache/paimon/operation/commit/SnapshotConflictChecker.java:
##########
@@ -197,14 +226,19 @@ private void checkBucketCompaction(
ranges.values().stream().mapToLong(range ->
range.to).min().getAsLong() + 1;
for (long id = firstSnapshot; id < newSnapshotId; id++) {
Snapshot snapshot = snapshotManager.snapshot(id);
+ boolean compactOrOverwrite =
+ snapshot.commitKind() == CommitKind.COMPACT
+ || snapshot.commitKind() == CommitKind.OVERWRITE;
if (snapshot.commitUser().equals(commitUser)
- || (snapshot.commitKind() != CommitKind.COMPACT
- && snapshot.commitKind() != CommitKind.OVERWRITE))
{
+ || (!compactOrOverwrite
+ && (snapshot.commitKind() != CommitKind.APPEND
+ || vectorBackfills.isEmpty()))) {
continue;
}
Set<Pair<BinaryRow, Integer>> buckets = new HashSet<>();
for (Map.Entry<Pair<BinaryRow, Integer>, Range> entry :
ranges.entrySet()) {
- if (entry.getValue().to < id) {
+ if (entry.getValue().to < id
Review Comment:
**[P2] Avoid rejecting backfills for appends outside their source set**
With deletion vectors enabled and merge-on-read disabled, the batch view
excludes L0. A concurrent writer adding only an L0 file in the same bucket
changes neither the backfill's visible data nor its deletion vectors or payload
coverage, but the bucket-wide APPEND check still rejects the backfill. On a
continuously ingested bucket, a build longer than the checkpoint interval can
fail on every attempt. Could the conflict check compare the eligible source set
and its DV/index state, or exempt a pure L0 append when the batch view excludes
it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]