cbb330 commented on issue #18237: URL: https://github.com/apache/iceberg/issues/18237#issuecomment-5899146646
Closing this in favor of a different approach. While working through the design, I found that a reference's creation time doesn't bound what we actually care about. Our problem is compliant data. E.g. when a branch that split from `main` keeps rows that `main` later deletes or edits (for example, member data purges). In this case, what matters is how long a branch has been missing `main`'s changes, and that isn't necessarily how long the reference has existed, like for example: - A branch created today from an older snapshot holds that snapshot's data. - A branch created from another branch holds that branch's data, however recently it was created. With that framing in mind, v3 row lineage already provides what's needed at the data level. `_row_id` and `_last_updated_sequence_number` let a maintenance job compare a branch's rows with `main`'s and find rows the branch still holds that `main` has since deleted or edited. The sequence numbers of `main`'s delete files and rewritten files, mapped to snapshot times, show how old those changes are. If that snapshot has already expired, the change is older than the retention window, which is enough for a compliance decision. None of this needs a spec change, so we plan to handle it in OpenHouse's table maintenance. If others need creation timestamps for lifecycle policies, independent of data compliance, I'm happy to revisit. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
