FANNG1 commented on PR #11280: URL: https://github.com/apache/gravitino/pull/11280#issuecomment-4905086562
Thanks for all the review comments and suggestions. Let me first explain our use case. We are building a multi-tenant compute platform. Each tenant is assigned separate storage resources, such as buckets and AK/SK credentials. For each tenant, we create separate Iceberg REST catalogs and Lance catalogs. As the number of tenants grows, especially when new tenants are added, maintaining client-side Spark catalog configuration becomes difficult. In some environments, upgrading or changing client-side configuration is also not easy. So dynamic discovery and automatic registration of Iceberg and Lance REST catalogs is a real pain point for us. I agree that the current proposal can make setup easier under certain conditions, but it also introduces some understanding cost and complexity when we try to extend it to more cases. Another possible direction is to add a `catalogs` API to the Iceberg/Lance REST catalog service. The API could return the catalogs managed by the REST service, and the client side could discover and register them automatically from that API. This direction may also be easier to extend to other connectors, such as Flink and Trino, because the discovery logic would live closer to the REST catalog service instead of being tied to Spark-specific Gravitino catalog mapping. If the current Gravitino-driven mapping approach feels too complicated, what do you think about this alternative direction? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
