WeiWenda opened a new pull request #27349: [SPARK-30617][SQL] Stop check values of spark.sql.catalogImplementation to improve expansibility URL: https://github.com/apache/spark/pull/27349 ### What changes were proposed in this pull request? When user config spark.sql.catalogImplementation with value not in in-memory/hive, check if below properties is configured. If configured then instantiate SessionState with provided Class, or else throw Exception as usual. ``` spark.sql.catalogImplementation.[value of spark.sql.catalogImplementation].builder spark.sql.catalogImplementation.[value of spark.sql.catalogImplementation].externalCatalog ``` For example: ``` spark.sql.catalogImplementation = qihoo spark.sql.catalogImplementation.qihoo.builder = org.apache.spark.sql.qihoo.QihooSessionStateBuilder spark.sql.catalogImplementation.qihoo.externalCatalog = org.apache.spark.sql.qihoo.QihooExternalCatalog ``` ### Why are the changes needed? We have implemented a complex ExternalCatalog which is used for retrieving multi isomerism database's metadata(sush as elasticsearch、postgresql), so that we can make a mixture query between hive and our online data. But as spark require that value of spark.sql.catalogImplementation must be one of in-memory/hive, we have to modify SparkSession and rebuild spark to make our project work. Finally, we hope spark removing above restriction, so that it's will be much easier to let us keep pace with new spark version. Thanks! ### Does this PR introduce any user-facing change? no ### How was this patch tested? no
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
