jackye1995 opened a new pull request #3480: URL: https://github.com/apache/iceberg/pull/3480
Add new hook in `RewriteFiles` snapshot update to allow accepting a sequence number that is used for all new data files. That will force the manifest writer to produce `ManifestFile` with the provided sequence number instead of `-1`. Also add a new property in `RewriteDataFiles` to use this feature through config `use-starting-sequence-number`. When enabled, the sequence number when compaction starts will be used for commit. This whole mechanism solves the issue today in CDC where compaction has conflicts with new equality delete files. With this change, `RewriteFiles` can go through as long as the newly added data files don't have new position deletes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
