srielau commented on code in PR #58545:
URL: https://github.com/apache/spark/pull/58545#discussion_r3945782529
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -636,9 +640,8 @@ class JacksonParser(
}
}
- // The JSON map will never have null or duplicated map keys, it's safe to
create a
- // ArrayBasedMapData directly here.
- val mapData = ArrayBasedMapData(keys.toArray, values.toArray)
+ val mapData = new ArrayBasedMapBuilder(keyType, valueType).from(
Review Comment:
Fixed in 25d1584897e. JSON now uses ArrayBasedMapBuilder only for declared
CHAR/VARCHAR key types and retains direct ArrayBasedMapData construction for
ordinary STRING keys. XML likewise uses the policy-aware builder only for
CHAR/VARCHAR and retains its historical toMap last-wins path otherwise. Added
compatibility coverage with standard semantics both disabled and enabled.
Tested with: sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64
-Dsbt.override.build.repos=true "sql/testOnly *BasicCharVarcharTestSuite -- -z
SPARK-59274" (3 passed).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:
##########
@@ -636,9 +640,8 @@ class JacksonParser(
}
}
- // The JSON map will never have null or duplicated map keys, it's safe to
create a
- // ArrayBasedMapData directly here.
- val mapData = ArrayBasedMapData(keys.toArray, values.toArray)
+ val mapData = new ArrayBasedMapBuilder(keyType, valueType).from(
Review Comment:
Fixed in 25d1584897e. JacksonParser now uses one root-cause extractor for
DUPLICATED_MAP_KEY and rethrows it before generic partial-result handling in
convertMap and convertObject, as well as at the top-level parser boundary.
Added regressions for a collision nested as a map value and a collision after
an unrelated malformed struct field; both fail under EXCEPTION, while nested
LAST_WIN remains covered. The targeted SPARK-59274 suite passed all 3 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
