szehon-ho commented on code in PR #57593:
URL: https://github.com/apache/spark/pull/57593#discussion_r3668313160
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -314,18 +315,17 @@ object Analyzer {
/**
* In case ANSI value wasn't persisted for a view or a UDF, we set it to
`true` in case Spark
- * version used to create the view is 4.0.0 or higher. We set it to `false`
in case Spark version
- * is lower than 4.0.0 or if the Spark version wasn't stored (in that case
we assume that the
- * value is `false`)
+ * version used to create the view is 4.0.0 or higher (ANSI SQL mode became
the default in
+ * Spark 4.0, see SPARK-44444). We set it to `false` in case Spark version
is lower than 4.0.0
+ * or if the Spark version wasn't stored / can't be parsed (in that case we
assume that the
+ * value is `false`).
*/
def trySetAnsiValue(sqlConf: SQLConf, createSparkVersion: String = ""): Unit
= {
Review Comment:
Agree this needs a direct regression test — `HiveSQLViewSuite` / "make sure
we can resolve view created by old version of Spark" does not cover the fixed
behavior.
That test never asserts ANSI / `Cast.evalMode`; it only checks rows and
schema. `ExternalCatalogWithListener.createTable` rewrites `createVersion` to
`SPARK_VERSION`, so under dual-run a *partial* fix can surface as
`LOGICAL_PLAN_COMPARISON_MISMATCH`, but a regression that puts **both**
analyzers back on LEGACY (the current master coincidence: `"5.x"` and `""` both
miss `startsWith("4.")`) still passes.
A focused check of `trySetAnsiValue` / `View.effectiveSQLConf` for
`createVersion` in `{"3.5.0", "4.0.0", "5.0.0", "", "bogus"}` (and that
single-pass + fixed-point agree when ANSI is absent from `viewSQLConfigs`)
would lock this in. `AlwaysPersistedConfigsSuite` already has "ANSI value is
set to false if not persisted for views" — natural place to add the positive
cases.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ViewResolver.scala:
##########
@@ -102,7 +102,10 @@ class ViewResolver(
val (resolvedChild, usedViewResolutionContext) =
withViewResolutionContext(unresolvedView) {
SQLConf.withExistingConf(
- View.effectiveSQLConf(unresolvedView.desc.viewSQLConfigs,
unresolvedView.isTempView)
+ View.effectiveSQLConf(
+ configs = unresolvedView.desc.viewSQLConfigs,
+ isTempView = unresolvedView.isTempView,
+ createSparkVersion = unresolvedView.desc.createVersion)
Review Comment:
Good catch aligning this with `ViewResolution.resolve`.
Minor follow-up: `SessionCatalog.parseMetricViewDefinition` still calls
`View.effectiveSQLConf` without `createSparkVersion` (same omission this
fixes). `fromCatalogTable` already forwards it. Low practical risk if metric
views always persist ANSI via `CapturesConfig`, but worth fixing for
consistency / dual-run safety when ANSI is absent from `viewSQLConfigs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
