gengliangwang opened a new pull request, #59926:
URL: https://github.com/apache/spark/pull/59926
### What changes were proposed in this pull request?
`CollationTypeCoercion` checks every expression it visits for an input with
indeterminate collation. The check computed the data type of every child first,
and only then asked whether the expression accepts such an input:
```scala
expression.children.exists(child =>
shouldFailWithIndeterminateCollation(expression, getDataTypeSafe(child)))
```
`shouldFailWithIndeterminateCollation(expression, dataType)` is
`!canContainIndeterminateCollation(expression) &&
hasIndeterminateCollation(dataType)`, and almost every expression can contain
indeterminate collation, so the child types were thrown away.
This PR checks the expression first and computes the child types only when
the expression does not accept an indeterminate collation:
```scala
!canContainIndeterminateCollation(expression) &&
expression.children.exists(child =>
hasIndeterminateCollation(getDataTypeSafe(child)))
```
The first operand does not depend on the child, so the verdict is the same
for every expression.
### Why are the changes needed?
Some of those child types are expensive to ask for. The `WithField` and
`DropField` operations of `UpdateFields` have no data type:
`StructFieldsOperation.dataType` throws `SparkException.internalError`, and
`getDataTypeSafe` catches it and returns `NullType`. So analyzing a plan that
updates struct fields, for example `col("s").withField("c",
lit(1)).dropFields("b")`, built and discarded one exception, stack trace
included, for each such operation every time type coercion visited the
`UpdateFields`. `UpdateFields` itself accepts an indeterminate collation, so
after this change its operations are not asked for a data type at all.
### Does this PR introduce _any_ user-facing change?
No. Which expressions fail with `INDETERMINATE_COLLATION_IN_EXPRESSION` is
unchanged.
### How was this patch tested?
New `IndeterminateCollationCheckSuite` (catalyst) with three tests:
- `CollationTypeCoercion` does not ask the field operations of an
`UpdateFields` for a data type. The test uses a `DropField` that counts the
requests.
- An expression that does not accept an indeterminate collation input,
`upper(x)` over an `IndeterminateStringType` attribute, still fails with
`INDETERMINATE_COLLATION_IN_EXPRESSION`.
- An expression that accepts one, `length(x)`, is returned unchanged.
Without the change to `CollationTypeCoercion`, the first test fails and the
other two pass. With it, all three pass, and so do the 12 tests of the existing
`IndeterminateCollationTestSuite`:
```
build/sbt "catalyst/testOnly
org.apache.spark.sql.catalyst.analysis.IndeterminateCollationCheckSuite"
build/sbt "sql/testOnly
org.apache.spark.sql.collation.IndeterminateCollationTestSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
