jarredhj0214 opened a new issue, #13117:
URL: https://github.com/apache/gravitino/issues/13117
### Describe the feature
Add a job-level configuration to allow Spark applications to skip Gravitino
Spark connector initialization while keeping the global spark.plugins
configuration unchanged.
Proposed configuration:
spark.sql.gravitino.enabled=false
Default value:
spark.sql.gravitino.enabled=true
When spark.sql.gravitino.enabled=false, GravitinoSparkPlugin can still be
loaded by Spark, but the Gravitino driver plugin should skip all Gravitino
initialization logic.
### Motivation
Some platforms inject Gravitino Spark connector configurations globally, for
example:
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=...
spark.sql.gravitino.metalake=...
However, not all Spark workloads are compatible with Gravitino today. For
example, TiSpark jobs may fail during startup or execution when the Gravitino
Spark connector is initialized.
In these cases, users need a per-job way to opt out of Gravitino behavior
without requiring the platform to remove the global spark.plugins
configuration. This would make it easier to adopt Gravitino incrementally while
still supporting incompatible or not-yet-supported Spark workloads.
### Describe the solution
Introduce a new Spark configuration:
public static final String GRAVITINO_ENABLED =
GRAVITINO_PREFIX + "enabled";
The default behavior should remain unchanged:
spark.sql.gravitino.enabled=true
At the very beginning of GravitinoDriverPlugin.init(), before reading or
validating spark.sql.gravitino.uri and spark.sql.gravitino.metalake, check this
flag:
if (!conf.getBoolean(GravitinoSparkConfig.GRAVITINO_ENABLED, true)) {
LOG.info("Gravitino Spark connector is disabled.");
return Collections.emptyMap();
}
When disabled, the connector should:
- Not validate spark.sql.gravitino.uri
- Not validate spark.sql.gravitino.metalake
- Not create a GravitinoClient
- Not access the Gravitino server
- Not call loadRelationalCatalogs()
- Not register Gravitino catalogs into Spark
- Not inject Gravitino SQL extensions
Spark should continue running with native Spark, TiSpark, or user explicitly
configured catalogs.
### Additional context
This switch is not intended to:
- Remove spark.plugins itself
- Avoid class loading of the Gravitino connector jar
- Disable Gravitino or hybrid extensions manually configured in
spark.sql.extensions
- Disable Gravitino catalog classes manually configured in
spark.sql.catalog.xxx
Those configurations should still be controlled by the job submission side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
