srielau opened a new pull request, #58584: URL: https://github.com/apache/spark/pull/58584
### What changes were proposed in this pull request? Support first-class CHAR/VARCHAR across Hive extension boundaries when first-class semantics are enabled: - Convert Catalyst CHAR/VARCHAR to Hive CHAR/VARCHAR object inspectors while preserving length. - Convert Hive CHAR/VARCHAR inspectors back to first-class Catalyst types for UDF, UDAF, and UDTF results. - Wrap Java and writable Hive CHAR/VARCHAR values using their declared lengths. - Preserve constant and nested CHAR/VARCHAR inspector shapes. - Support CHAR/VARCHAR output from script TRANSFORM with and without Hive SerDe. - Continue exposing Hive CHAR/VARCHAR results as STRING under legacy semantics. Hive object inspectors cannot carry Spark collation metadata. Collated inputs are accepted, while Hive function results retain the CHAR/VARCHAR length with the default collation. JIRA: https://issues.apache.org/jira/browse/SPARK-59277 ### Why are the changes needed? With `spark.sql.charVarchar.standardSemantics.enabled=true`, Catalyst keeps CHAR/VARCHAR as first-class types. Hive extension paths still handled them as unsupported types or downgraded Hive CHAR/VARCHAR results to STRING. This prevented Hive UDF/UDAF/UDTF functions and script TRANSFORM from operating consistently on scalar, collated, and nested CHAR/VARCHAR values. ### Does this PR introduce _any_ user-facing change? Yes. When first-class CHAR/VARCHAR semantics are enabled, Hive UDF/UDAF/UDTF functions and script TRANSFORM now accept and return CHAR/VARCHAR values while preserving declared lengths. Legacy flag-off behavior remains unchanged. ### How was this patch tested? Added focused coverage for: - Java, writable, constant, collated, and nested Hive object inspectors. - Legacy STRING fallback. - Hive scalar UDF, UDAF, and UDTF execution. - Script TRANSFORM with Hive SerDe and without SerDe. Ran: ``` sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ -Dsbt.override.build.repos=true \ 'hive/testOnly org.apache.spark.sql.hive.HiveInspectorSuite org.apache.spark.sql.hive.execution.HiveUDFSuite org.apache.spark.sql.hive.execution.HiveUDAFSuite org.apache.spark.sql.hive.execution.HiveScriptTransformationSuite -- -z SPARK-59277' sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ -Dsbt.override.build.repos=true \ 'sql/testOnly org.apache.spark.sql.execution.SparkScriptTransformationSuite -- -z SPARK-59277' dev/scalastyle sql ``` All 10 focused tests and Scala style checks passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Auto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
