Hi Spark developers, I'd like to get feedback on SPARK-59849, which proposes an optional fallback resolver for unknown V2 catalog names.
We use Spark with Apache Gravitino, managing over 1,300 catalogs. Currently, the connector must list and register all visible catalog names through spark.sql.catalog.<name> during driver initialization, even when an application accesses only one catalog. Spark supports lazy instantiation of configured catalogs, but an unconfigured catalog name cannot be discovered through an external resolver. The proposed behavior is: - Keep existing catalog configuration as the first choice. - For an unconfigured name, consult an explicitly configured resolver. - Cache a successfully resolved catalog. - Preserve existing behavior when no resolver is configured. This would allow external catalog services to resolve catalogs on first use and discover catalogs created after application startup. Would this extension be acceptable in principle? Is there an existing extension point or alternative approach that I should consider? If the direction is welcome, I can work on a minimal implementation and tests. JIRA: https://issues.apache.org/jira/browse/SPARK-59849 Proposal: https://github.com/apache/spark/issues/58444 Thanks, Jie Hu --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
