Hey all, I sketched out some design details for next-step features in Polaris Federation based on discussion with Pooja, Rulin, Harish, and Teja: https://docs.google.com/document/d/19Vm0Uy-EyEYtgd2-bZpPICODOB3rYQJvHeqL1ODOOLc/edit?tab=t.0
At a high level, currently the REST-based Federation MVP basically only does two things: - Catalog-level RBAC - Simple passthrough to the remote REST Catalog Per the original federation proposal, the responsibilities of a catalog federation layer ultimately go beyond "simple passthrough", and higher-level features entail making the federation layer actually an "active" translation layer. The three features outlined in the document are closely interrelated and also serve as prerequisites for future plans: - Credential-vending using the Polaris StorageConfig - Polaris fetching the TableMetadata (and potentially other metadata/manifest files) - particularly applicable if the remote catalog is an implementation that only returns a metadata filename instead of providing inline TableMetadata JSON - Namespace/Table-level RBAC In a nutshell, the main technical work for these next steps is to refactor the remote-catalog client to live under a Polaris "wrapper" (referred to as the "decorating delegator catalog" in the document) which formalizes the responsibilities of this federation in terms of things like credential-vending, construction of FileIO via FileIOFactory, etc. Any input is welcome. Cheers, Dennis
