cloud-fan commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r3892187814
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:
##########
@@ -437,6 +437,11 @@ object OrcUtils extends Logging {
s"array<${getOrcSchemaString(a.elementType)}>"
case m: MapType =>
s"map<${getOrcSchemaString(m.keyType)},${getOrcSchemaString(m.valueType)}>"
+ // Under standard semantics, keep Spark responsible for CHAR/VARCHAR
assignment and scan
+ // checks. Native ORC would truncate or pad before Spark can validate the
original value.
+ // Preserve-only mode retains the native constrained schema and its legacy
enforcement.
+ case _: CharType | _: VarcharType if
SQLConf.get.charVarcharStandardSemantics =>
Review Comment:
**Blocking (P1):** This reads the task-side caller configuration, but
`spark.sql.charVarchar.standardSemantics.enabled` is persisted with views. If a
view is resolved with standard semantics and its caller later disables the
setting, this branch requests native ORC `VARCHAR`; ORC can then truncate
`abcdef` to `abcd` before the view's already-resolved Spark length check sees
the value. That makes the persisted view caller-dependent and silently bypasses
`EXCEED_LIMIT_LENGTH`. Please carry the analyzed/view-bound semantics into ORC
reader construction instead, and add a permanent-view regression that flips the
caller setting.
##########
connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:
##########
@@ -3724,6 +3724,82 @@ abstract class AvroSuite
}
}
+ test("SPARK-58814: Avro infers nested CHAR/VARCHAR schema and values") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ val input = spark.range(1).selectExpr(
+ "cast('ab' AS CHAR(4)) AS c",
+ "cast('xy' AS VARCHAR(3)) AS v",
+ "named_struct('c', cast('z' AS CHAR(2))) AS s",
+ "array(cast('q' AS VARCHAR(2))) AS a",
+ "map(cast('k' AS CHAR(2)), cast('v' AS VARCHAR(2))) AS m")
+ input.write.mode("overwrite").format("avro").save(path)
+
+ val readBack = spark.read.format("avro").load(path)
+ assert(DataType.equalsIgnoreNullability(readBack.schema, input.schema))
+ checkAnswer(
+ readBack.selectExpr("concat('<', c, '>')", "v", "concat('<', s.c,
'>')"),
Review Comment:
**Non-blocking (P2):** This case creates the array and map, but the
projection only materializes `c`, `v`, and `s.c`. Column pruning can skip
decoding both collection fields, so a regression in array/map conversion or
CHAR-key padding would still pass. Please select `a` and `m` as well and assert
`Seq("q")` and `Map("k " -> "v")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
