JingsongLi commented on code in PR #11072:
URL: https://github.com/apache/paimon/pull/11072#discussion_r4532741620


##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1675,17 +1675,25 @@ public boolean replaceManifestList(
             @Nullable String indexManifest,
             @Nullable Long nextRowId,
             @Nullable Map<String, String> properties) {
-        return replaceManifestListOnce(
-                latest,
-                totalRecordCount,
-                baseManifestList,
-                deltaManifestList,
-                indexManifest,
-                nextRowId,
-                properties);
+        Snapshot snapshot =
+                replaceManifestListOnce(
+                        latest,
+                        totalRecordCount,
+                        baseManifestList,
+                        deltaManifestList,
+                        indexManifest,
+                        nextRowId,
+                        properties);
+        if (snapshot == null) {
+            return false;
+        }
+        // Like regular commits, run post-commit callbacks after releasing the 
commit lock.
+        commitCallbacks.forEach(callback -> 
callback.onManifestReplacement(snapshot));

Review Comment:
   [P2] Make callback failures recoverable through the repair procedure
   
   The replacement snapshot has already been committed before this callback 
runs. If writing Iceberg metadata or publishing its catalog pointer fails, 
rerunning `remove_unexisting_manifests` finds no missing manifests and returns 
`false` before invoking any callback. The repair also uses a fresh 
`Repair-table-UUID`, so ordinary writer retries cannot recover this commit. I 
reproduced this by failing one metadata write: a second `execute()` returned 
`false` and Iceberg still exposed the pre-repair row. The new retry test 
directly constructs a callback with the original repair user, which is not the 
recovery path available to someone retrying the procedure. Please make an 
already-committed replacement's callbacks recoverable and add a 
failure-then-procedure-retry test.



##########
paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java:
##########
@@ -1053,13 +1071,31 @@ private void createMetadataWithBase(
                         .filter(meta -> meta.content() == 
IcebergManifestFileMeta.Content.DELETES)
                         .collect(Collectors.toList());
 
-        Map<String, Pair<BinaryRow, DataFileMeta>> removedFiles = new 
LinkedHashMap<>();
+        // Manifest replacement can change the base files without recording a 
delta. Detect the
+        // empty-delta overwrite on retries as well, when the original 
callback context is gone.
+        boolean fullSnapshot =
+                fileChangesCollector == null
+                        || (snapshot.commitKind() == 
Snapshot.CommitKind.OVERWRITE
+                                && table.store()
+                                        .manifestListFactory()
+                                        .create()
+                                        .readDeltaManifests(snapshot)
+                                        .isEmpty());
+        List<DataSplit> snapshotSplits =
+                fullSnapshot
+                        ? 
table.newSnapshotReader().withSnapshot(snapshotId).read().dataSplits()
+                                .stream()
+                                .filter(DataSplit::rawConvertible)

Review Comment:
   [P1] Preserve live published files when a split requires merging
   
   For a primary-key table without Iceberg deletion vectors, a previously 
published highest-level file can overlap a later L0 update. The batch planner 
then puts both files into a non-raw-convertible split. Filtering that split out 
makes `collectSnapshotFileChanges` treat the still-live highest-level file as 
removed, so a manifest replacement drops all of its previously visible rows 
from Iceberg, including keys that were not updated. I reproduced this with an 
unchanged Paimon file set: replacing only the manifest lists reduced the 
Iceberg result from four rows to two. Please reconcile against the snapshot's 
live manifest entries while preserving the publication rules, rather than using 
`rawConvertible` as the test for file existence, and cover the Lmax + L0 
overlap case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to