srielau commented on code in PR #59302:
URL: https://github.com/apache/spark/pull/59302#discussion_r4234807893
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -277,34 +247,6 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
data => IntervalUtils.microsToDuration(
IntervalUtils.castStringToDTInterval(UTF8String.fromString(data),
start, end)),
converter)
- case dt @ (_: ArrayType | _: MapType | _: StructType)
- if standardCharVarcharSemantics &&
CharVarcharUtils.hasCharVarchar(dt) =>
- val physicalType =
ScriptTransformationIOSchema.toUnboundedStringType(dt)
- // JSON object keys are strings. Restore non-string keys inside the
malformed
- // wrapper, then apply CHAR/VARCHAR (and CHAR/VARCHAR map-key dedup)
outside it.
- val jsonType =
ScriptTransformationIOSchema.toJsonMapKeyType(physicalType)
- val tz = Some(conf.sessionLocalTimeZone)
- val complexTypeFactory = JsonToStructs(
- jsonType,
- ioschema.outputSerdeProps.toMap,
- Literal(null),
- tz)
- val restoreJsonKeys = if (jsonType.sameType(physicalType)) {
- identity[Any] _
- } else {
- ScriptTransformationIOSchema.makeJsonToTargetRestorer(
- jsonType, physicalType, tz)
- }
- val toDeclaredType =
ScriptTransformationIOSchema.makeJsonToTargetRestorer(
- physicalType, dt, tz)
- val parser = wrapperConvertException(
- data => restoreJsonKeys(
- complexTypeFactory.nullSafeEval(UTF8String.fromString(data))),
- identity)
- (data: String) => {
- val parsed = parser(data)
- if (parsed == null) null else toDeclaredType(parsed)
- }
case _: ArrayType | _: MapType | _: StructType =>
Review Comment:
To clarify the last sentence: a **targeted parse-time** error for `MAP<INT,
CHAR/VARCHAR>` (non-string map key mixed with CHAR/VARCHAR, no SerDe,
first-class types on) would not touch `MAP<INT, STRING>`. Empty or null
`MAP<INT, STRING>` rows that work today would keep working.
The empty/null caveat only applies if we rejected non-string map keys in
`outputFieldWriters` at runtime. That path is shared with `MAP<INT, STRING>`,
so a broad exec-side check could start failing rows that currently succeed.
We are not doing either check in this PR. `MAP<INT, CHAR(n)>` without SerDe
stays the same as `MAP<INT, STRING>`: JSON keys are strings, then
`UnsafeProjection` can `ClassCastException` on `getInt`. Called out in the PR
description as out of scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
