cloud-fan commented on code in PR #58704:
URL: https://github.com/apache/spark/pull/58704#discussion_r4107507137
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -7034,6 +7034,26 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val JSON_STREAM_MULTILINE_TOP_LEVEL_ARRAY =
+ buildConf("spark.sql.json.enableStreamingTopLevelArray")
+ .doc("When true, multiline JSON file reads stream the elements of a
top-level array one at " +
+ "a time instead of materializing the entire array before returning
rows. This applies " +
+ "only to reads into a struct schema that take top-level arrays as
structs, and has no " +
+ "effect on reads using the `singleVariantColumn` or
`explodeEmbeddedArray` option. " +
+ "Streaming also makes an array element, rather than the whole
document, the record " +
+ "that a parse mode applies to, since rows already emitted cannot be
withdrawn: " +
+ "PERMISSIVE fills the corrupt record column for the malformed element
only, leaving " +
+ "it null on the valid rows of the same document, and DROPMALFORMED
drops that " +
+ "element rather than the whole document. An element whose failure
leaves the parser " +
+ "at an unknown position, such as a nested value of the wrong shape,
still ends the " +
+ "document, as does a failure outside any element, such as a syntax
error between two " +
+ "elements or a missing closing bracket. It can be overwritten by the
JSON option " +
+ "`enableStreamingTopLevelArray`.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
Thanks, that rationale makes sense. The explicit false/true grids already
pin both semantic branches, and the declaration makes the compatibility default
visible, so I’m fine leaving out the separate unset case.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:4105396987","thread_id":"inline:4105396987","verdict_sha256":"fbfd6156813f99061dd3aa8e7b04d155ef214ed6f88f7eb390b7ddbab21cebbd"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
