Tim Meehan created SPARK-59417:
----------------------------------
Summary: Stream multiline top-level JSON arrays during JSON reads
Key: SPARK-59417
URL: https://issues.apache.org/jira/browse/SPARK-59417
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.0
Reporter: Tim Meehan
When Spark reads multiline JSON whose root value is an array of structs,
{{JacksonParser}} currently materializes the entire array before returning
rows. For large arrays, parser-side memory usage therefore grows with the
complete document size.
Add an opt-in SQL configuration that allows elements of a top-level array to be
emitted lazily from a single Jackson parser. The existing eager behavior
remains the default for compatibility and staged rollout.
The streaming path should:
* Keep parser-side memory bounded rather than materializing the full array.
* Preserve rows emitted before a later malformed element.
* Handle late failures according to PERMISSIVE, DROPMALFORMED, and FAILFAST
modes.
* Leave existing eager parser callers unchanged.
{{
Configuration: }}
{{spark.sql.json.enableStreamingTopLevelArray (default false)}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
