Github user cxzl25 commented on the issue:
https://github.com/apache/spark/pull/22887
user set hadoop conf can't overwrite spark-defaults.conf
**SparkHadoopUtil.get.appendS3AndSparkHadoopConfigurations** overwrite the
user-set spark.hadoop with the default configuration
(sparkSession.sparkContext.conf)
@gengliangwang @cloud-fan @gatorsmile
Could you please give some comments when you have time?
Thanks so much.
https://github.com/apache/spark/blob/80813e198033cd63cc6100ee6ffe7d1eb1dff27b/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala#L85-L89
## test:
### spark-defaults.conf
```
spark.hadoop.mapreduce.input.fileinputformat.split.maxsize 2
```
### spark-shell
```scala
val hadoopConfKey="mapreduce.input.fileinputformat.split.maxsize"
spark.conf.get("spark.hadoop."+hadoopConfKey) // 2
var hadoopConf=spark.sessionState.newHadoopConf
hadoopConf.get(hadoopConfKey) // 2
spark.conf.set(hadoopConfKey,1) // set 1
hadoopConf=spark.sessionState.newHadoopConf
hadoopConf.get(hadoopConfKey) // 1
//org.apache.spark.sql.hive.HadoopTableReader append Conf
org.apache.spark.deploy.SparkHadoopUtil.get.appendS3AndSparkHadoopConfigurations(spark.sparkContext.getConf,
hadoopConf)
//org.apache.spark.sql.hive.HadoopTableReader _broadcastedHadoopConf
hadoopConf.get("mapreduce.input.fileinputformat.split.maxsize") // 2
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
