badalprasadsingh commented on code in PR #2624:
URL: https://github.com/apache/iceberg-go/pull/2624#discussion_r4597357399
##########
table/transaction_delete.go:
##########
@@ -1016,30 +1024,48 @@ func (t *Transaction) writePositionDeletesForFiles(ctx
context.Context, fs io.IO
}
}
}
- for _, ref := range streamedPathOrder {
- if _, ok := seen[ref]; !ok {
- seen[ref] = struct{}{}
- referenced = append(referenced, ref)
+ if meta.formatVersion < 3 {
+ for _, ref := range streamedPathOrder {
+ if _, ok := seen[ref]; !ok {
+ seen[ref] = struct{}{}
+ referenced = append(referenced, ref)
+ }
}
}
- // Supersede the old DVs only after the merged replacements are staged,
so a
- // write failure above leaves the existing DVs untouched.
- for _, dvFile := range existingDVs {
- updater.removeDeletionVector(dvFile)
+ // Supersede only deletes for files that received a replacement DV.
+ // File-scoped legacy position deletes are now represented by that DV;
+ // shared position-delete files must stay live for their other targets.
+ for dataFilePath, deleteFiles := range oldPositionDeletes.fileScoped {
+ if _, replaced := seen[dataFilePath]; !replaced {
+ continue
+ }
+ for _, deleteFile := range deleteFiles {
+ updater.removeDeleteFile(deleteFile)
+ }
+ }
+ for dataFilePath, dvFile := range existingDVs {
+ if _, replaced := seen[dataFilePath]; replaced {
+ updater.removeDeletionVector(dvFile)
+ }
}
Review Comment:
no test covers a v3 delete that finds no new rows in a file with a dv -> It
relies on the seen check here and the "formatVersion < 3" guard at line1027.
without either, the dv is dropped and deleted rows come back.
please add a test
##########
table/transaction_delete.go:
##########
@@ -1016,30 +1024,48 @@ func (t *Transaction) writePositionDeletesForFiles(ctx
context.Context, fs io.IO
}
}
}
- for _, ref := range streamedPathOrder {
- if _, ok := seen[ref]; !ok {
- seen[ref] = struct{}{}
- referenced = append(referenced, ref)
+ if meta.formatVersion < 3 {
+ for _, ref := range streamedPathOrder {
+ if _, ok := seen[ref]; !ok {
+ seen[ref] = struct{}{}
+ referenced = append(referenced, ref)
+ }
}
}
- // Supersede the old DVs only after the merged replacements are staged,
so a
- // write failure above leaves the existing DVs untouched.
- for _, dvFile := range existingDVs {
- updater.removeDeletionVector(dvFile)
+ // Supersede only deletes for files that received a replacement DV.
+ // File-scoped legacy position deletes are now represented by that DV;
+ // shared position-delete files must stay live for their other targets.
+ for dataFilePath, deleteFiles := range oldPositionDeletes.fileScoped {
+ if _, replaced := seen[dataFilePath]; !replaced {
+ continue
+ }
+ for _, deleteFile := range deleteFiles {
+ updater.removeDeleteFile(deleteFile)
+ }
Review Comment:
removing these files makes the commit "non-replayable", so if another writer
commits first -> this delete now fails with "ErrCommitFailed" instead of
retrying
the delete docs (transaction_delete.go ;ine275, table.go line422) name only
dv removals, could you update both?
##########
table/arrow_utils.go:
##########
@@ -2481,28 +2481,14 @@ func positionDeleteRecordsToDataFilesDV(ctx
context.Context, rootLocation string
return metadata.PartitionSpecByID(int(id))
})
- // Seed the writer with any deletion vector the referenced data
file
- // already carries so its previously deleted positions survive
into the
- // merged DV. The scan that produced args.itr does NOT apply
the existing
- // DVs (makePositionDeleteRecordsForFilter builds its
FileScanTasks
- // without DeleteFiles), so the incoming batches may re-report
positions
- // already in the old DV; re-Setting them on top of the seed is
a no-op
- // because the bitmap is idempotent. The seed is what
guarantees the old
- // positions survive — do not remove it, or a second delete
would drop
- // the earlier deletes and resurrect those rows. The spec
allows at most
- // one DV per data file, so the caller supersedes the old DV
once this
- // merged one is written.
+ // Seed a file's existing deletes only after a new position is
found for
+ // it. A filter can select files from their stats even when no
live row
+ // matches; seeding those files eagerly would turn an empty
delete into a
+ // DV rewrite. The scan does not apply existing DVs, so skip
positions
+ // already present in the seed before deciding that a
replacement is
+ // needed.
hasEntries := false
- for filePath, bitmap := range args.existingDVs {
- pCtx, ok := partitionContextByFilePath[filePath]
- if !ok {
- yield(nil, fmt.Errorf("unexpected missing
partition context for existing deletion vector path %s", filePath))
-
- return
- }
- writer.Load(filePath, bitmap, pCtx.specID,
pCtx.partitionData)
- hasEntries = true
- }
+ seededPaths := make(map[string]struct{}, len(args.existingDVs))
Review Comment:
this now seeds older pos-deletes too, and only once a new position is found,
but the doc of "recordWritingArgs.existingDVs" still describes only dvs, and
both it and" existingDVBitmaps" have dv-only names.
better: can you rename them (say "existingDeletes") and update that doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]