Xiao Li created SPARK-23573:

             Summary: Ban the usage of SparkContext.hadoopConfiguration
                 Key: SPARK-23573
                 URL: https://issues.apache.org/jira/browse/SPARK-23573
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: Xiao Li

SparkContext.hadoopConfiguration should not be used directly in SQL module.
Instead, one should always use sessionState.newHadoopConfiguration(), which 
blends in configs set per session.

The idea is to add the linter rule to ban it. 

- Restrict the linter rule to the components that use SQL. use the parameter 
- Exclude genuinely valid uses, like e.g. in SessionState (ok, that can be done 
per case with scalastyle:off in the code.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to