Hey everyone, Thanks for putting this proposal together, Jagdeep! I just wanted to share a few quick thoughts from my side. There’ve been some past discussions around this topic, and we have some catalog implementations take on server-side responsibility for transaction logic for their own means. So it’s clear that catalogs can play an active role in having transactional capabilities.
What I think this proposal is getting at is allowing such catalogs to be able to facilitate transactions, and to track and remain aware of the transactional state. That awareness is key for enabling consistent, isolated reads across multiple tables. I think we should first clarify the following goals: - *What does it actually mean for a catalog to support transactions?* More specifically, what’s the expected scope and contents of a transaction? In the case of the multi-table commit API, it seems scoped to tables, not to broader catalog-level state like in some other implementations. - *What isolation guarantees should we expect?* As well as what mechanisms ensure those guarantees are upheld (e.g. rebase, conflict checks)? - *And how much should the catalog vs. the client be responsible for tracking this transactional state?* Looking forward to hearing more thoughts! Drew On Tue, Apr 22, 2025 at 11:10 AM Jagdeep Sidhu <sidhujagde...@gmail.com> wrote: > Hi Fokko, > > Thank you for detailed feedback! I need to think through your comments and > update/reply accordingly. > > But I want to discuss a key goal for writing this proposal (MTMS = > "Multi-Table Multi-Statement"): > For MTMS txns, many un-coordinated engines could be operating on the same > set of tables in a Catalog. In this scenario, lets say an engine starts a > MTMS txn, which needs to read through different tables during transaction > lifetime. It needs to know exactly which version of a table it should load > to provide SNAPSHOT ISOLATION consistent reads across tables (we can even > ignore commits and assume this txn runs read only queries). That is the key > problem this proposal aims to solve - provide SNAPSHOT ISOLATION consistent > reads across tables when multiple engines are operating on the same tables. > > As you suggest, this kind of book-keeping can be implemented in engines > using some sort of algorithm, but that does not help if multiple different > engines are operating on the same set of tables. Catalog is already in the > commit path when engines use IRC. By adding this sort of functionality to > Catalog, we can make it possible for MTMS transactions to simply ask the > Catalog about which version of table it needs to load to provide cross > table SNAPSHOT ISOLATION consistent reads. > > While the proposal has additional changes to further simplify how engines > can implement MTMS transactions, what I described above is the key reason > for this proposal. Looking forward to hearing what you think! > > -Jagdeep > > On Tue, Apr 22, 2025 at 2:22 AM Fokko Driesprong <fo...@apache.org> wrote: > >> Hey Jagdeep, >> >> Thanks for proposing this. In general, it depends on what entity of the >> triangle (Table-format, Catalog, Engine) we want to make responsible for >> what operation. Where this proposal delegates more to the catalog, and what >> is done today in the engine itself. I went over it and added some context >> and comments. Let me know what you think! >> >> Kind regards, >> Fokko >> >> >> Op di 22 apr 2025 om 04:42 schreef Jagdeep Sidhu <sidhujagde...@gmail.com >> >: >> >>> Hi Iceberg dev community, >>> >>> cc: Dru - I have been collaborating with him. >>> >>> I want to start this email thread to discuss an IRC API proposal to >>> enable engines to implement Multi-Statement Multi-Table Transactions. More >>> details in the document: >>> >>> >>> https://docs.google.com/document/d/1jr4Ah8oceOmo6fwxG_0II4vKDUHUKScb/edit?usp=sharing&ouid=117626409673211358817&rtpof=true&sd=true >>> Proposal issue: https://github.com/apache/iceberg/issues/12865 >>> >>> This document discusses challenges for engines to implement such >>> support, and proposes APIs that Catalog can implement to enable such >>> functionality. Looking forward to feedback from Iceberg dev community. >>> >>> Thank you in advance! >>> -Jagdeep >>> >>> >>>
