10110346 commented on a change in pull request #24174: [SPARK-27238][SQL] Add
fine-grained configurations to handle `convertMetastoreParquet ` and
`convertMetastoreOrc`
URL: https://github.com/apache/spark/pull/24174#discussion_r268469532
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -97,6 +97,13 @@ private[spark] object HiveUtils extends Logging {
.booleanConf
.createWithDefault(true)
+ val CONVERT_METASTORE_PARQUET_EXCLUDED_TABLES =
+ buildConf("spark.sql.hive.convertMetastoreParquet.excludedTables")
+ .doc("A comma-separated list of Parquet table names, which do not use the
built-in Parquet" +
+ "reader and writer when \"spark.sql.hive.convertMetastoreParquet\" is
true.")
Review comment:
Thanks @HyukjinKwon
Take another example, in a SQL statement, this SQL statement needs to read
many hive Parquet tables, but `TableA` can't use the built-in Parquet reader
and writer, if we disable `spark.sql.hive.convertMetastoreParquet`, the other
tables can't use the built-in Parquet reader and writer too, we know, the
performance of built-in Parquet reader and writer is much better, this will
affect the performance of the SQL statement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
