Hi all, Ayush and Prithvi recently contributed a couple of interesting PRs: [4939], [5035].
It looks like people are starting to encounter consistency issues in JDBC persistence. The PRs provide valuable insight into the underlying issues. They offer incremental fixes that can work. However, I believe it is time for the Polaris community to review and improve this area of the codebase holistically. By this, I mean finding a solution that can be applied to all persistence backends (in-memory, JDBC, NoSQL) and addresses these aspects: * Supporting concurrent and consistent changes where the service reads and validates current catalog state, then commits a change (e.g. name clashes during renames). * Supporting consistent but independent changes to RBAC grants and MetaStore entities. This independence is needed to support external authorizers like OPA and Ranger. * Supporting atomic changes across multiple similar entities. * Supporting authorization-based filtering of list operations (cf. [4831]). * Supporting credential-vending decisions that are rooted in the exact state of the catalog. * Supporting server-side retries for transient persistence failures (e.g. RDBMS Tx serializability failures). Please share your comments and ideas. [4831] https://github.com/apache/polaris/pull/4831 [4939] https://github.com/apache/polaris/pull/4939 [5035] https://github.com/apache/polaris/pull/5035 Thanks, Dmitri
