Hu Jie created SPARK-59849:
------------------------------
Summary: [SQL] Support fallback resolver for unknown V2 catalog
names
Key: SPARK-59849
URL: https://issues.apache.org/jira/browse/SPARK-59849
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.3.2
Environment: Spark 3.3.2 with Apache Gravitino Spark Connector.
The same static V2 catalog registration model appears to still exist in the
current master branch, where unknown catalog names must be configured through
spark.sql.catalog.<catalog_name> before SQL analysis.
Reporter: Hu Jie
This issue was first opened on GitHub, but since Spark generally tracks logical
issues in JIRA, I am filing it here to make it easier for the community to
triage and discuss.
GitHub issue: https://github.com/apache/spark/issues/58444
h3. Problem
We are using Spark with Apache Gravitino and observed that all top-level V2
catalogs need to be registered eagerly through Spark configuration:
{code}
spark.sql.catalog.<catalog_name>=<catalog_class>
{code}
This appears to be a general limitation of Spark's V2 catalog lookup model
rather than a version-specific bug. When Spark resolves a multipart identifier
such as:
{code:sql}
SELECT * FROM catalog_a.db.table
{code}
CatalogManager.catalog("catalog_a") eventually calls Catalogs.load("catalog_a",
conf). If spark.sql.catalog.catalog_a is not configured, Spark throws
CatalogNotFoundException immediately.
Spark supports lazy instantiation for configured catalogs, but it does not
currently support lazy discovery for catalog names managed by an external
catalog service.
h3. Motivation
External catalog services, such as Apache Gravitino or similar enterprise
catalog services, may manage a large number of catalogs. Since Spark requires
each top-level catalog name to be configured before analysis, connectors have
to eagerly materialize all visible catalog names into Spark configuration
during application startup.
This has several drawbacks:
* Spark application startup becomes slower as the number of catalogs grows.
* Catalogs created after Spark application startup are not naturally
discoverable.
* Connectors have to preload catalog names only to satisfy Spark's static
registration model.
* The actual catalog implementation may still be initialized lazily, but the
top-level catalog name must be eagerly registered.
For example, an external catalog service connector may currently need to list
visible catalogs at driver initialization time and register entries like:
{code}
spark.sql.catalog.catalog_a=...
spark.sql.catalog.catalog_b=...
spark.sql.catalog.catalog_c=...
{code}
even if the Spark application only accesses one of them.
h3. Proposal
Introduce an optional fallback resolver for unknown V2 catalog names.
One possible API shape could be:
{code:scala}
trait CatalogResolver {
def resolveCatalog(name: String, conf: SQLConf): Option[CatalogPlugin]
}
{code}
Resolvers could be configured statically, for example:
{code}
spark.sql.catalog.fallbackResolvers=com.example.ExternalCatalogResolver
{code}
Then CatalogManager.catalog(name) could follow this order:
# Return the cached catalog if it has already been loaded.
# Try the existing spark.sql.catalog.<name> based loading path.
# If the catalog is not configured, invoke fallback resolvers in configured
order.
# If a resolver returns a CatalogPlugin, cache it in CatalogManager.
# If no resolver resolves the catalog, throw the existing
CatalogNotFoundException.
The default behavior would remain unchanged when no fallback resolver is
configured.
h3. Compatibility
This should be fully backward compatible:
* No behavior changes unless fallback resolvers are explicitly configured.
* Existing spark.sql.catalog.<name> configuration keeps priority.
* Existing CatalogNotFoundException behavior remains when no resolver matches.
* Resolved catalogs can follow the same caching and lifecycle behavior as other
V2 catalogs.
h3. Summary
Spark already supports lazy instantiation of configured V2 catalogs. This
proposal is to add optional lazy discovery for unknown V2 catalog names, which
would help integrations with external catalog services avoid eagerly
registering every visible catalog at application startup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
