srielau commented on code in PR #59167: URL: https://github.com/apache/spark/pull/59167#discussion_r4226883875
########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala: ########## @@ -60,11 +60,50 @@ class JacksonParser( // to a value in a field for `InternalRow`. private type ValueConverter = JsonParser => AnyRef + // CHAR/VARCHAR is STRING-on-the-wire in JSON. Parse as unbounded STRING, then assign. + private val physicalSchema: DataType = + CharVarcharUtils.replaceCharVarcharWithStringAlways(schema) + // `ValueConverter`s for the root schema for all fields in the schema - private val rootConverter = makeRootConverter(schema) + private val rootConverter = makeRootConverter(physicalSchema) // Lazy: a schema that is not a struct never reaches either top-level-array path. - private lazy val arrayElementConverter = makeConverter(schema) + private lazy val arrayElementConverter = makeConverter(physicalSchema) + + private val assignExpr: Expression = + CharVarcharUtils.assignAfterParse( + BoundReference(0, physicalSchema, nullable = true), + schema) + + private val needsAssignment = + SQLConf.get.charVarcharStandardSemantics && Review Comment: To put the baseline more precisely: we are comparing to this JSON code from before `spark.sql.charVarchar.standardSemantics.enabled` applied here, not to current master. In that state, JSON did not do standard-semantics assignment. CHAR/VARCHAR on the wire was parsed as STRING. That includes preserve-only (`preserveCharVarcharTypeInfo=true` and the flag false): no parser pad and no `EXCEED_LIMIT_LENGTH`. This PR only adds assignment when the flag is true. We are not restoring later in-parser CHAR handling for the flag-off path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
