Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22237#discussion_r220668461
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -554,18 +554,30 @@ case class JsonToStructs(
@transient
lazy val converter = nullableSchema match {
case _: StructType =>
- (rows: Seq[InternalRow]) => if (rows.length == 1) rows.head else null
+ (rows: Iterator[InternalRow]) => if (rows.hasNext) rows.next() else
null
case _: ArrayType =>
- (rows: Seq[InternalRow]) => rows.head.getArray(0)
+ (rows: Iterator[InternalRow]) => if (rows.hasNext)
rows.next().getArray(0) else null
case _: MapType =>
- (rows: Seq[InternalRow]) => rows.head.getMap(0)
+ (rows: Iterator[InternalRow]) => if (rows.hasNext)
rows.next().getMap(0) else null
}
- @transient
- lazy val parser =
- new JacksonParser(
- nullableSchema,
- new JSONOptions(options + ("mode" -> FailFastMode.name),
timeZoneId.get))
+ @transient lazy val parser = {
+ val parsedOptions = new JSONOptions(options, timeZoneId.get)
+ val mode = parsedOptions.parseMode
+ if (mode != PermissiveMode && mode != FailFastMode) {
--- End diff --
The `JSONOptions` is shared among build-in json functions like `from_json`
and JSON datasource. And the formal one use 3 modes - `FAILFAST`,
`DROPMALFORMED` and `PERMISSIVE`. I am not sure how the `mode` mode can be
replaced. The approach that I could image is to inherit from `JSONOptions` and
add new `val`. The `mode` itself cannot be removed because it is used in
`FailureSafeParser` for example, in particular `DropMalformedMode` is handled
explicitly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
