[
https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836750#comment-16836750
]
Xingbo Jiang commented on SPARK-27520:
--------------------------------------
The major problem of `SparkContext.hadoopConfiguration` is it's globally
effective, which means when you change the hadoopConf then all other jobs share
the same SparkContext shall be affected. We shall introduce a way that can
limit the hadoopConf change to minimal scope. Also, maybe we shall deprecate
SparkContext.hadoopConfiguration, instead, user shall use hadoopConfiguration
with ThreadLocal so the changes are more unlikely to affect others.
> Introduce a global config system to replace hadoopConfiguration
> ---------------------------------------------------------------
>
> Key: SPARK-27520
> URL: https://issues.apache.org/jira/browse/SPARK-27520
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Xingbo Jiang
> Priority: Major
>
> hadoopConf can be accessed via `SparkContext.hadoopConfiguration` from both
> user code and Spark internal. The configuration is mainly used to read files
> from hadoop-supported file system(eg. get URI/get FileSystem/add security
> credentials/get metastore connect url/etc.)
> We shall keep a global config that users can set and use that to track the
> hadoop configurations, and avoid using `SparkContext.hadoopConfiguration`,
> maybe we shall mark it as deprecate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
