junmuz commented on code in PR #11072:
URL: https://github.com/apache/paimon/pull/11072#discussion_r4540387001
##########
paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java:
##########
@@ -1053,13 +1071,31 @@ private void createMetadataWithBase(
.filter(meta -> meta.content() ==
IcebergManifestFileMeta.Content.DELETES)
.collect(Collectors.toList());
- Map<String, Pair<BinaryRow, DataFileMeta>> removedFiles = new
LinkedHashMap<>();
+ // Manifest replacement can change the base files without recording a
delta. Detect the
+ // empty-delta overwrite on retries as well, when the original
callback context is gone.
+ boolean fullSnapshot =
+ fileChangesCollector == null
+ || (snapshot.commitKind() ==
Snapshot.CommitKind.OVERWRITE
+ && table.store()
+ .manifestListFactory()
+ .create()
+ .readDeltaManifests(snapshot)
+ .isEmpty());
+ List<DataSplit> snapshotSplits =
+ fullSnapshot
+ ?
table.newSnapshotReader().withSnapshot(snapshotId).read().dataSplits()
+ .stream()
+ .filter(DataSplit::rawConvertible)
Review Comment:
Addressed. Repair reconciliation now reads live manifest entries and
preserves previously published files by physical path, including files in
splits requiring merging. Added a regression for the Lmax + L0 overlap case
that verifies Iceberg retains all four previously visible rows. The test passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]