[
https://issues.apache.org/jira/browse/SPARK-23573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuaki Ishizaki updated SPARK-23573:
-------------------------------------
Description:
SparkContext.hadoopConfiguration should not be used directly in SQL module.
Instead, one should always use sessionState.newHadoopConfiguration(), which
blends in configs set per session.
The idea is to add the linter rule to ban it.
- Restrict the linter rule to the components that use SQL. use the parameter
`scalastyleSources`?
- Exclude genuinely valid uses, like e.g. in SessionState (ok, that can be
done per case with scalastyle:off in the code.
was:
SparkContext.hadoopConfiguration should not be used directly in SQL module.
Instead, one should always use sessionState.newHadoopConfiguration(), which
blends in configs set per session.
The idea is to add the linter rule to ban it.
- Restrict the linter rule to the components that use SQL. use the parameter
`scalastyleSources`?
- Exclude genuinely valid uses, like e.g. in SessionState (ok, that can be done
per case with scalastyle:off in the code.
> Create linter rule to prevent misuse of SparkContext.hadoopConfiguration in
> SQL modules
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-23573
> URL: https://issues.apache.org/jira/browse/SPARK-23573
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiao Li
> Priority: Major
>
> SparkContext.hadoopConfiguration should not be used directly in SQL module.
> Instead, one should always use sessionState.newHadoopConfiguration(), which
> blends in configs set per session.
> The idea is to add the linter rule to ban it.
> - Restrict the linter rule to the components that use SQL. use the parameter
> `scalastyleSources`?
> - Exclude genuinely valid uses, like e.g. in SessionState (ok, that can be
> done per case with scalastyle:off in the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
