JingsongLi commented on code in PR #11072:
URL: https://github.com/apache/paimon/pull/11072#discussion_r4532741620
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1675,17 +1675,25 @@ public boolean replaceManifestList(
@Nullable String indexManifest,
@Nullable Long nextRowId,
@Nullable Map<String, String> properties) {
- return replaceManifestListOnce(
- latest,
- totalRecordCount,
- baseManifestList,
- deltaManifestList,
- indexManifest,
- nextRowId,
- properties);
+ Snapshot snapshot =
+ replaceManifestListOnce(
+ latest,
+ totalRecordCount,
+ baseManifestList,
+ deltaManifestList,
+ indexManifest,
+ nextRowId,
+ properties);
+ if (snapshot == null) {
+ return false;
+ }
+ // Like regular commits, run post-commit callbacks after releasing the
commit lock.
+ commitCallbacks.forEach(callback ->
callback.onManifestReplacement(snapshot));
Review Comment:
[P2] Make callback failures recoverable through the repair procedure
The replacement snapshot has already been committed before this callback
runs. If writing Iceberg metadata or publishing its catalog pointer fails,
rerunning `remove_unexisting_manifests` finds no missing manifests and returns
`false` before invoking any callback. The repair also uses a fresh
`Repair-table-UUID`, so ordinary writer retries cannot recover this commit. I
reproduced this by failing one metadata write: a second `execute()` returned
`false` and Iceberg still exposed the pre-repair row. The new retry test
directly constructs a callback with the original repair user, which is not the
recovery path available to someone retrying the procedure. Please make an
already-committed replacement's callbacks recoverable and add a
failure-then-procedure-retry test.
##########
paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java:
##########
@@ -1053,13 +1071,31 @@ private void createMetadataWithBase(
.filter(meta -> meta.content() ==
IcebergManifestFileMeta.Content.DELETES)
.collect(Collectors.toList());
- Map<String, Pair<BinaryRow, DataFileMeta>> removedFiles = new
LinkedHashMap<>();
+ // Manifest replacement can change the base files without recording a
delta. Detect the
+ // empty-delta overwrite on retries as well, when the original
callback context is gone.
+ boolean fullSnapshot =
+ fileChangesCollector == null
+ || (snapshot.commitKind() ==
Snapshot.CommitKind.OVERWRITE
+ && table.store()
+ .manifestListFactory()
+ .create()
+ .readDeltaManifests(snapshot)
+ .isEmpty());
+ List<DataSplit> snapshotSplits =
+ fullSnapshot
+ ?
table.newSnapshotReader().withSnapshot(snapshotId).read().dataSplits()
+ .stream()
+ .filter(DataSplit::rawConvertible)
Review Comment:
[P1] Preserve live published files when a split requires merging
For a primary-key table without Iceberg deletion vectors, a previously
published highest-level file can overlap a later L0 update. The batch planner
then puts both files into a non-raw-convertible split. Filtering that split out
makes `collectSnapshotFileChanges` treat the still-live highest-level file as
removed, so a manifest replacement drops all of its previously visible rows
from Iceberg, including keys that were not updated. I reproduced this with an
unchanged Paimon file set: replacing only the manifest lists reduced the
Iceberg result from four rows to two. Please reconcile against the snapshot's
live manifest entries while preserving the publication rules, rather than using
`rawConvertible` as the test for file existence, and cover the Lmax + L0
overlap case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]