In my opinion, this is unnecessary, it's well solved by load balancers/proxies.
On Mon, Dec 9, 2024, 8:12 AM Vladimir Ozerov <voze...@querifylabs.com> wrote: > Hi, > > Catalog is a critical part of Iceberg infrastructure and may require > highly available setup. In similar services (e.g., HMS, etc) this is often > done as follows: > > 1. Start several service instances > 2. Decide which one is coordinator via etcd, Zookeper, Ratis, etc > 3. Expose HA endpoint to a client: multiple endpoints OR a single > endpoint via proxy > > Currently, there is no way to expose multiple endpoints to a REST server. > This may work in some cases if you hide multiple REST server instances > behind a proxy/balancer. But this proxy requires own HA setup, which > complicates the overall deployment. > > I'd like to ask the community whether we can extend REST specification > with multiple endpoints to support HA REST catalog without proxies. This > extension could contain two essential parts: > > 1. How to provide multiple endpoints to RESTSessionCatalog. This could > be encoded into URL or as additional property > 2. Some additional headers and/or error codes to allow a REST server > instance communicate peer endpoints and the current coordinator > > A very rough example how this can work in practice. Configuration: > > mycatalog.endpoint=https://host1 > mycatalog.endpoints=https://host1,https://host2,https://host3 > > REST server response headers with additional endpoints: > > X-HA-Coordinator: https://host2 > X-HA-Endpoints: https://host1,https://host2,https://host3 > > WDYT? > > Regards, > Vladimir >
