ajithme commented on a change in pull request #27041: [SPARK-25061][SQL]
Precedence for ThriftServer hiveconf commandline parameter
URL: https://github.com/apache/spark/pull/27041#discussion_r366534585
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##########
@@ -177,8 +177,12 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default
values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
+
+ // not to lose command line overwritten properties
+ // make a copy overridden props so that it can be reinserted finally
+ val overriddenHiveProps = HiveConf.getConfSystemProperties.asScala
val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey ->
kv.getValue) ++
- sparkConf.getAll.toMap ++ extraConfig).toMap
+ sparkConf.getAll.toMap ++ overriddenHiveProps ++ extraConfig).toMap
Review comment:
@yhuai Sure, I have updated the PR with reasonable pointers for the order.
Does it suffice it now.?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
