ajithme commented on issue #27041: [SPARK-25061][SQL] Precedence for ThriftServer hiveconf commandline parameter URL: https://github.com/apache/spark/pull/27041#issuecomment-588125267 > Thanks for pinging me @cloud-fan > > We can override hive configurations in many ways. > > Take `hive.metastore.uris` for an example, we can reset it via `--conf spark.hadoop.hive.metastore.uris=thrift://example.com:9083`, or `--conf spark.hive.hive.metastore.uris=thrift://example.com:9083` or `--hiveconf hive.metastore.uris=thrift://example.com:9083` or maybe(not sure) `--conf spark.driver.extraJavaOptions=-Dxxx`. > > This PR seems to prefer `--hiveconf` than others. > > Personally, I prefer those spark configurations always have higher precedence than other type > configurations including hive/hadoop/java/system, etc, as we are writing spark applications. Thanks @yaooqinn for your thoughts. This seems little confusing to know who is overriding as per the documentation mentioned in https://spark.apache.org/docs/latest/sql-distributed-sql-engine.html#running-the-thrift-jdbcodbc-server I agree with your opinion of having sparkConf as most precedence, but command line (--hiveconf) should be preferred over config file (hive-site.xml) For the case you mentioned (X marks the conf is used) type | case 1 | case 2 | case 3 | case 4 ------------ | ------------- | ------------- | ------------- | ------------- --conf spark.hadoop.hive.* | X | X | - | - --conf spark.hive.hive.* | X | - | X | - --hiveconf | X | X | X | X hive-site.xml | X | X | X | X **Preference** | ? | --conf spark.hadoop.hive.* | --conf spark.hive.* | --hiveconf so do you mean, in case 1, 2, 3 where spark.* conf is used it must get preference.? how about case 4.? i prefer in case 4 --hiveconf has precedence and rest cases spark conf can have higher precedence
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
