benibus commented on code in PR #14355:
URL: https://github.com/apache/arrow/pull/14355#discussion_r1028261142
##########
cpp/src/arrow/json/reader.h:
##########
@@ -60,5 +50,49 @@ class ARROW_EXPORT TableReader {
ARROW_EXPORT Result<std::shared_ptr<RecordBatch>> ParseOne(ParseOptions
options,
std::shared_ptr<Buffer> json);
+/// \brief A class that reads a JSON file incrementally
+///
+/// JSON data is read from a stream in fixed-size blocks (configurable with
+/// `ReadOptions::block_size`). Each block is converted to a `RecordBatch`.
Yielded
+/// batches have a consistent schema but may differ in row count.
+///
+/// The supplied `ParseOptions` are used to determine a schema on the first
non-empty
+/// block. Afterwards, the schema is frozen and unexpected fields will be
ignored on
+/// subsequent reads (unless `UnexpectedFieldBehavior::Error` was specified).
Review Comment:
Do you mean overloads that replace `ParseOptions` with just a schema? You
can already (optionally) provide a schema with `ParseOptions::explicit_schema`,
which will be used if it's present. The main purpose of the
"read-the-first-block dance" is to handle cases where:
- A schema isn't supplied in `ParseOptions` and we need to infer one somehow
- A schema _is_ supplied but `ParseOptions::unexpected_field_behavior` is
set to `InferType`, so the output schema may differ
For simplicity's sake, the implementation always sets the output schema from
the first batch, although it shouldn't differ from the input schema if it was
provided and `UnexpectedFieldBehavior::InferType` wasn't specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
