pan3793 commented on code in PR #53608:
URL: https://github.com/apache/spark/pull/53608#discussion_r2646829084
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala:
##########
@@ -239,6 +239,16 @@ private[spark] object HiveUtils extends Logging {
.booleanConf
.createWithDefault(true)
+ val PROPAGATE_HIVE_CONFS =
+ buildConf("spark.sql.hive.propagateHiveConfs")
+ .version("4.2.0")
+ .doc("Propagate the specific Hive configs from Spark session to Hive
conf, to " +
+ "mitigate some common use cases that 'SET hive.foo=bar' does not takes
effect. " +
+ "Currently, this only takes effect for loadDynamicPartitions.")
+ .stringConf
+ .toSequence
+ .createWithDefault(Seq("hive.exec.max.dynamic.partitions"))
Review Comment:
this one is the most frequent case from my daily support, the current
implementation just demonstrates the idea of how it works, I'd like to hear
more suggestions on how to control the Hive config list for propagation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
