cloud-fan commented on code in PR #57414:
URL: https://github.com/apache/spark/pull/57414#discussion_r3654286161
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:
##########
@@ -357,6 +270,93 @@ object MultiLineJsonDataSource extends JsonDataSource {
}
}
+ /**
+ * Infers a multi-line JSON schema when at least one input is an archive.
Each archive entry
+ * (streamed via `SupportsArchiveFormat`, never unpacked to disk) and each
loose file is one whole
+ * JSON document, and all feed a single [[JsonInferSchema]] pass -- exactly
as a directory of the
+ * same files would infer. Single-line archive inference does not come here:
the Text data source
+ * reads archives directly, so it flows through
[[TextInputJsonDataSource.infer]] like any
+ * directory read. A corrupt/missing input is skipped as a unit when the
ignore flags are set --
Review Comment:
Please describe this as skipping the remainder of the archive, not the whole
input. Earlier entries have already been merged into the inferred schema before
`hasNext` reaches a corrupt later header, and replacing the remaining delegate
with `Iterator.empty` cannot retract them. The XML helper has the same
overstatement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
