srielau commented on code in PR #59302:
URL: https://github.com/apache/spark/pull/59302#discussion_r4238103128
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SparkScriptTransformationSuite.scala:
##########
@@ -62,38 +207,33 @@ class SparkScriptTransformationSuite extends
BaseScriptTransformationSuite with
|SELECT TRANSFORM('abcdef') USING 'cat' AS (c CHAR(4))
|FROM VALUES (1) input(dummy)""".stripMargin)
withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
- val exception = intercept[Exception] {
- sql("SELECT * FROM v_overflow").collect()
- }
- val runtimeException = exception match {
- case s: org.apache.spark.SparkRuntimeException => s
- case other =>
-
other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
- }
- checkError(
- exception = runtimeException,
- condition = "EXCEED_LIMIT_LENGTH",
- parameters = Map("limit" -> "4"))
+ checkExceedLimitLength(
+ intercept[Exception](sql("SELECT * FROM v_overflow").collect()),
+ "4")
}
}
}
- withSQLConf(
- SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true",
- SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+ assert(uncaughtExceptionHandler.exception.isEmpty)
+ }
+
+ test("SPARK-60090: TRANSFORM view created at default follows the caller
session flag") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ // PERSISTED stores the create-time flag when it is set true (see the test
above).
+ // The default false is not stored on the view, so a later session with
the flag on
Review Comment:
Done. Renamed to `TRANSFORM view created with flag off uses stored schema
cast` and reworded the comment: PERSISTED parses under the stored flag (off),
so there is no assignment Project; the CHAR(4) result and padding come from the
view output cast against the stored schema.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1856,6 +1858,26 @@ class AstBuilder extends DataTypeAstBuilder
schemaLess
)
)
+ // First-class CHAR/VARCHAR (standard semantics or
preserveCharVarcharTypeInfo):
+ // assign declared types in a Project so the result matches AS (...). Flag
off keeps
+ // the historical STRING result with no pad and no EXCEED_LIMIT_LENGTH.
The check can
+ // be pruned like scan-side assignment when the column is unreferenced.
+ if (conf.charVarcharFirstClassTypes &&
+ schema.exists(f => CharVarcharUtils.hasCharVarchar(f.dataType))) {
+ Project(
+ attributes.zip(schema).map { case (a, field) =>
+ if (CharVarcharUtils.hasCharVarchar(field.dataType)) {
+ Alias(
+ CharVarcharUtils.stringLengthCheck(a, field.dataType),
+ a.name)(explicitMetadata =
Some(CharVarcharUtils.cleanMetadata(a.metadata)))
+ } else {
+ a
Review Comment:
Added `TRANSFORM mixed CHAR and non-CHAR columns` in
`BaseScriptTransformationSuite` (`AS (id INT, name CHAR(4), note STRING)`), so
it runs on both the no-SerDe and Hive SerDe paths. It asserts the full schema
order and the padded CHAR value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
