HyukjinKwon opened a new pull request #28459:
URL: https://github.com/apache/spark/pull/28459
### What changes were proposed in this pull request?
This PR proposes to deprecate 'spark.sql.optimizer.metadataOnly'
configuration and remove it in the future release.
### Why are the changes needed?
This optimization can cause a potential correctness issue, see also
SPARK-26709.
Also, it seems difficult to extend the optimization. Basically you should
whitelist all available functions. It costs some maintenance overhead, see also
SPARK-31590.
Looks we should just better let users use `SparkSessionExtensions` instead
if they must use, and remove it in Spark side.
### Does this PR introduce _any_ user-facing change?
Yes, setting `spark.sql.optimizer.metadataOnly` will shod a deprecation
warning:
```scala
scala> spark.conf.unset("spark.sql.optimizer.metadataOnly")
```
```
20/05/06 12:57:23 WARN SQLConf: The SQL config
'spark.sql.optimizer.metadataOnly' has been
deprecated in Spark v3.0 and may be removed in the future. Avoid to depend
on this optimization
to prevent a potential correctness issue. If you must use, use
'SparkSessionExtensions' instead to
inject it as a custom rule.
```
```scala
scala> spark.conf.set("spark.sql.optimizer.metadataOnly", "true")
```
```
20/05/06 12:57:44 WARN SQLConf: The SQL config
'spark.sql.optimizer.metadataOnly' has been
deprecated in Spark v3.0 and may be removed in the future. Avoid to depend
on this optimization
to prevent a potential correctness issue.If you must use, use
'SparkSessionExtensions' instead to
inject it as a custom rule.
```
### How was this patch tested?
Manually tested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
