Hi everyone, To avoid passing copies of a file around for comments, I put the doc for commit sequence numbers into Google so we can comment on a central copy: https://docs.google.com/document/d/1jr4Ah8oceOmo6fwxG_0II4vKDUHUKScb/edit?usp=sharing&ouid=100239850723655533404&rtpof=true&sd=true
Ryan On Fri, May 16, 2025 at 2:51 AM Maninderjit Singh < parmar.maninder...@gmail.com> wrote: > Thanks for the updated proposal Drew! > My preference for using the catalog authored timestamp is to minimize > changes to the REST spec so we can have good backwards compatibility. I > have quickly put together a draft proposal on how this should work. Looking > forward to feedback and discussion. > > Draft Proposal: Catalog‑Authored Timestamps for Apache Iceberg REST > Catalog > <https://drive.google.com/open?id=1KVgUJc1WgftHfLz118vMbEE7HV8_pUDk4s-GJFDyAOE> > > Thanks, > Maninder > > On Wed, May 14, 2025 at 6:12 PM Drew <img...@gmail.com> wrote: > >> Hi everyone, >> >> Thank you for feedback on the MTT proposal and during community sync. >> Based on it, Jagdeep and I have iterated on the document and added a second >> option to use *Catalog CommitSequenceNumbers*. Looking forward to >> getting more feedback on the proposal, where to add more details or >> approach/changes to consider. We appreciate everyone's time on this! >> >> The option introduces *Catalog CommitSequenceNumbers(CSNs)*, which allow >> clients/engines to read a consistent view of multiple tables without >> needing to register a transaction context with the catalog. This removes >> the need of registering a transaction context with Catalog, thus removing >> the need of transaction bookkeeping on the catalog side. For aborting >> transactions early, clients can use LoadTable with and without CSN to >> figure out if there is already a conflicting write on any of the tables >> being modified. Also removed the section where transactions were staging >> commits on Catalog, and changed the proposal to align with Eduard's PR >> around staging changes locally before commit ( >> https://github.com/apache/iceberg/pull/6948). >> >> Jagdeep also clarified in an example in a previous email where a workload >> may require multi table snapshot isolation, even if the tables are being >> updated without Multi-Table commit API. Though most MTT transactions will >> commit using the multi table commit API. >> >> Maninder, for the approach of "common notion of time between clients and >> catalog" - I spent some time thinking about it, but cannot find a feasible >> way to do this. Yes, the catalogs can use a high precision clock, but >> clients cannot use Catalog Timestamp from API calls to set local clock due >> to network latency for request/response. For example, different requests to >> the same Catalog servers can return different timestamps based on network >> latency. Also what if a client works with more than 1 Catalog. If you want >> to do a rough write-up or share a reference implementation that uses such >> an approach, I will be happy to brainstorm it more. Let us know! >> >> Here is the link to updated proposal >> >> >> <https://docs.google.com/document/d/1jr4Ah8oceOmo6fwxG_0II4vKDUHUKScb/edit?usp=sharing&ouid=100384647237395649950&rtpof=true&sd=true> >> Thanks Again! >> - Drew >> >
