cloud-fan commented on a change in pull request #26071: [SPARK-29412][SQL]
refine the document of v2 session catalog config
URL: https://github.com/apache/spark/pull/26071#discussion_r333051695
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1976,11 +1977,19 @@ object SQLConf {
.stringConf
.createOptional
- val V2_SESSION_CATALOG = buildConf("spark.sql.catalog.session")
- .doc("A catalog implementation that will be used in place of the Spark
built-in session " +
- "catalog for v2 operations. The implementation may extend
`CatalogExtension` to be " +
- "passed the Spark built-in session catalog, so that it may delegate
calls to the " +
- "built-in session catalog.")
+ val V2_SESSION_CATALOG_IMPLEMENTATION =
+ buildConf(s"spark.sql.catalog.${CatalogManager.SESSION_CATALOG_NAME}")
+ .doc("A catalog implementation that will be used in place of the Spark
Catalog for v2 " +
+ "operations (e.g. create table using a v2 source, alter a v2 table).
The Spark Catalog " +
+ "is the current catalog by default, and supports all kinds of catalog
operations like " +
+ "CREATE TABLE USING v1/v2 source, VIEW/FUNCTION related operations,
etc. This config is " +
+ "used to extend the Spark Catalog and inject custom logic to v2
operations, while other" +
+ "operations still go through the Spark Catalog. The catalog
implementation specified " +
+ "by this config should extend `CatalogExtension` to be passed the
Spark Catalog, " +
+ "so that it can delegate calls to Spark Catalog. Otherwise, the
implementation " +
+ "should figure out a way to access the Spark Catalog or its underlying
meta-store " +
+ "by itself. It's important to make the implementation share the
underlying meta-store " +
+ "of the Spark Catalog and act as an extension, instead of a separated
catalog.")
Review comment:
This is a complicated config and I'm trying my best to explain it.
cc some SQL guys to get fresh eyes and see if they can understand it.
@dongjoon-hyun @viirya @maropu @HyukjinKwon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
