cloud-fan commented on code in PR #59138:
URL: https://github.com/apache/spark/pull/59138#discussion_r4279818527
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala:
##########
@@ -1479,6 +1479,31 @@ abstract class JsonSuite
}
}
+ gridTest("multiline top level JSON array ends the document at a syntax error
in a nested object")(
+ Seq("PERMISSIVE", "DROPMALFORMED")) { mode =>
+ withSQLConf(
+ SQLConf.JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY.key -> "true",
+ SQLConf.JSON_ENABLE_PARTIAL_RESULTS.key -> "true") {
+ withTempPath { file =>
+ // Resuming inside the second element would return `t`'s object as a
row.
+ val document = """[{"a":1},{"s":{"x":1 2},"t":{"a":9}},{"a":2}]"""
Review Comment:
**Non-blocking (P2):** This document fails inside `s`, so after the error
the parser sits in the element's object context and `context.inArray` alone
rejects the resume. The other conjunct, `context.getParent.inRoot`, is what
rejects a failure that leaves the parser inside an array nested in the element,
and no streaming fixture reaches that position. If the guard were narrowed to
`context.inArray`, every existing JsonSuite and archive streaming test would
still pass.
I checked this with a jackson-core 2.22.2 simulation of the
`convertObject`/`convertArray` semantics. With the narrowed guard,
`[{"a":1},{"arr":[{"x":1 2},{"a":5}]},{"a":2}]` (schema `a INT, arr
ARRAY<STRUCT<x: INT>>, _corrupt_record STRING`) resumes inside `arr`. It emits
`(5, null, null)` with no corrupt marker, reads `arr`'s `]` as the end of the
top-level array, and silently drops `{"a":2}`. With the current guard, the
document ends after the corrupt row, as intended. Could you add this document
(or a similar array-of-structs case) to this gridTest, either as a second
document dimension or as a sibling test, so both halves of the guard are pinned?
**Verification:**
- **Regression:** Show that the nested-array case fails when the
getParent.inRoot conjunct is removed from the resume guard: an inner array
value is emitted as a row and the trailing element is silently dropped. It
passes with the full guard.
- **Behavior:** Assert the exact result rows for the nested-array document
in PERMISSIVE and DROPMALFORMED under both the v1 and v2 JSON readers,
alongside the existing nested-object case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
