junmuz commented on issue #5387:
URL: https://github.com/apache/paimon/issues/5387#issuecomment-6528898763
I have seen similar issue and was curious around solving this problem where
compaction can discard a delete’s sequence, allowing a later insert with an
older sequence to recreate the row.
@leaves12138 @JingsongLi I have been thinking about introducing an optional
tombstone index that
• Store the latest delete sequence per primary key in separate files
within the same partition and bucket as its data.
• Check this state during write processing to reject stale events before
they become visible.
• Let compaction remove a tombstone once a newer live version is durably
retained. Newer deletes advance the tombstone.
• Commit tombstone and data changes consistently through snapshot metadata.
I plan to keep these files outside normal data scans and Iceberg data
manifests. The goal is to preserve existing queries and native -D events, with
additional costs concentrated on write lookups, storage, and compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
