Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r212928747 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -554,18 +554,22 @@ case class JsonToStructs( @transient lazy val converter = nullableSchema match { case _: StructType => - (rows: Seq[InternalRow]) => if (rows.length == 1) rows.head else null + (rows: Iterator[InternalRow]) => if (rows.hasNext) rows.next() else null case _: ArrayType => - (rows: Seq[InternalRow]) => rows.head.getArray(0) + (rows: Iterator[InternalRow]) => if (rows.hasNext) rows.next().getArray(0) else null case _: MapType => - (rows: Seq[InternalRow]) => rows.head.getMap(0) + (rows: Iterator[InternalRow]) => if (rows.hasNext) rows.next().getMap(0) else null } - @transient - lazy val parser = - new JacksonParser( - nullableSchema, - new JSONOptions(options + ("mode" -> FailFastMode.name), timeZoneId.get)) + @transient lazy val parsedOptions = new JSONOptions(options, timeZoneId.get) + @transient lazy val rawParser = new JacksonParser(nullableSchema, parsedOptions) + @transient lazy val createParser = CreateJacksonParser.utf8String _ + @transient lazy val parser = new FailureSafeParser[UTF8String]( + input => rawParser.parse(input, createParser, identity[UTF8String]), + parsedOptions.parseMode, --- End diff -- It is not handled by `JacksonParser`, and the behavior is somehow similar to `PermissiveMode` as @HyukjinKwon pointed out at https://github.com/apache/spark/pull/22237/files#r212850156, but not exactly the same. Seems now the `PermissiveMode` on `FailureSafeParser` has different result on corrupted records. I noticed that some existing tests maybe changed due to that.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org