[
https://issues.apache.org/jira/browse/SPARK-23573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Li updated SPARK-23573:
----------------------------
Summary: Create linter rule to prevent misuse of
SparkContext.hadoopConfiguration in SQL modules (was: Ban the usage of
SparkContext.hadoopConfiguration)
> Create linter rule to prevent misuse of SparkContext.hadoopConfiguration in
> SQL modules
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-23573
> URL: https://issues.apache.org/jira/browse/SPARK-23573
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiao Li
> Priority: Major
>
> SparkContext.hadoopConfiguration should not be used directly in SQL module.
> Instead, one should always use sessionState.newHadoopConfiguration(), which
> blends in configs set per session.
> The idea is to add the linter rule to ban it.
> - Restrict the linter rule to the components that use SQL. use the parameter
> `scalastyleSources`?
> - Exclude genuinely valid uses, like e.g. in SessionState (ok, that can be
> done per case with scalastyle:off in the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
