srielau commented on code in PR #59302:
URL: https://github.com/apache/spark/pull/59302#discussion_r4234771790
##########
sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala:
##########
@@ -3265,7 +3265,8 @@ class FileSourceCharVarcharTestSuite extends
CharVarcharTestSuite with SharedSpa
test("SPARK-58794: EXTERNAL TABLE CHAR/VARCHAR pad, assignment, and
oversize") {
// External file tables use the same store assignment on write and pad +
- // length-check on scan. Hive TRANSFORM is out of scope for this epic.
+ // length-check on scan. Hive TRANSFORM uses the same assignment via
Project
+ // (SPARK-60090); covered in BaseScriptTransformationSuite.
Review Comment:
Named all three suites in that comment.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:
##########
@@ -87,6 +87,21 @@ abstract class BaseScriptTransformationSuite extends
QueryTest {
parameters = Map("dataType" -> sqlType))
}
+ protected def checkExceedLimitLength(exception: Throwable, limit: String):
Unit = {
+ val runtimeException = Iterator.iterate[Throwable](exception)(_.getCause)
+ .takeWhile(_ != null)
+ .collectFirst {
+ case s: org.apache.spark.SparkRuntimeException
Review Comment:
Imported `SparkRuntimeException` next to the other Spark exceptions.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1856,6 +1857,21 @@ class AstBuilder extends DataTypeAstBuilder
schemaLess
)
)
+ // Under standard semantics, assign declared CHAR/VARCHAR (pad / length
check) in a
+ // Project so the query result type matches AS (...). Flag off and
preserve-only keep
+ // the historical STRING result with no pad and no EXCEED_LIMIT_LENGTH.
+ if (conf.charVarcharStandardSemantics &&
+ attributes.exists(a =>
CharVarcharUtils.getRawType(a.metadata).isDefined)) {
+ Project(
+ attributes.map { a =>
+ Alias(
+ CharVarcharUtils.stringLengthCheck(a, a),
+ a.name)(explicitMetadata =
Some(CharVarcharUtils.cleanMetadata(a.metadata)))
Review Comment:
Same limitation as scan-side `stringLengthCheck`: `CreateNamedStruct` does
not copy nested field metadata or inner `NOT NULL`. Noted in the PR description
rather than special-casing TRANSFORM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
