srielau opened a new pull request, #58585: URL: https://github.com/apache/spark/pull/58585
### What changes were proposed in this pull request? Fix two CHAR comparison padding edge cases: - Preserve each foldable `IN` list expression in its original position while computing padding lengths, including `NULL` entries. - Recognize analyzer-inserted `CAST(attribute AS STRING)` nodes used to coerce untyped `NULL`, without treating user-specified casts as CHAR comparisons. - Accumulate padding requirements across every struct field instead of retaining only the last field's result. - Preserve nullable parent structs when rebuilding nested values for comparison. JIRA: https://issues.apache.org/jira/browse/SPARK-59278 ### Why are the changes needed? The previous `IN` rewrite partitioned null values away from non-null values and then zipped the original expression list with only the non-null lengths. If `NULL` preceded a matching literal, the matching literal could be dropped and a true result became null. Nested struct padding also overwrote its `needPadding` state for each field. Padding required by an earlier field was therefore lost when a later field needed no rewrite. Activating that rewrite additionally required preserving a null parent struct rather than rebuilding it as a non-null struct of null fields. ### Does this PR introduce _any_ user-facing change? Yes. CHAR comparisons now preserve SQL three-valued `IN`/`NOT IN` semantics when `NULL` precedes a matching literal, and nested struct comparisons apply all required CHAR padding while preserving parent nulls. ### How was this patch tested? Added regression tests covering: - `IN` and `NOT IN` with `NULL` before and between matching and non-matching literals. - Implicit analyzer casts versus user-specified `CAST(... AS STRING)`. - Multi-field nested struct padding. - Null parent structs with regular and null-safe equality. Ran: ``` sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ -Dsbt.override.build.repos=true \ 'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-59278' sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ -Dsbt.override.build.repos=true \ 'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-34833' \ 'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-50847' \ 'sql/testOnly org.apache.spark.sql.FileSourceCharVarcharTestSuite org.apache.spark.sql.DSV2CharVarcharTestSuite -- -z SPARK-51732' dev/scalastyle sql ``` All selected tests and Scala style checks passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Auto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
