taepper commented on code in PR #50945:
URL: https://github.com/apache/arrow/pull/50945#discussion_r3986657057
##########
cpp/src/arrow/json/chunker.cc:
##########
@@ -165,6 +91,78 @@ class ParsingBoundaryFinder : public BoundaryFinder {
int64_t* out_pos, int64_t* num_found) override {
return Status::NotImplemented("ParsingBoundaryFinder::FindNth");
}
+
+ private:
+ simdjson::ondemand::parser parser_;
+ // A persistent buffer to keep padded contents for simdjson.
+ // This should be more efficient than allocating a new padded_string
everytime.
+ std::string buffer_;
+
+ simdjson::padded_string_view GetPaddedStringView(std::string_view partial,
+ std::string_view block =
{}) {
+ // Adjust buffer size without copying old contents.
+ buffer_.clear();
+ buffer_.reserve(partial.size() + block.size() +
simdjson::SIMDJSON_PADDING);
+ buffer_.append(partial);
+ buffer_.append(block);
+ // XXX Hopefully this upholds for all std::string implementations
+ DCHECK_GE(buffer_.capacity() - buffer_.size(), simdjson::SIMDJSON_PADDING);
+ auto view = simdjson::padded_string_view(buffer_);
+ DCHECK(view.has_sufficient_padding());
+ return view;
+ }
+
+ // Consume the first or last JSON object (depending on `until_end`)
+ // and return the consumed JSON byte length, or 0 if no valid document
+ // can be parsed.
+ Result<size_t> ConsumeWholeObject(simdjson::padded_string_view input, bool
until_end) {
+ simdjson::ondemand::document_stream stream;
+ // XXX Should be pass a specific batch_size?
+ // The default value used by simdjson is 1MB, probably enough for most
purposes.
Review Comment:
We could use `batch_size` equal to `input.size()`, if we do not want to fail
in this case:
```
const size_t batch_size = std::max<size_t>(input.size(), 1);
RETURN_NOT_OK(ToStatus(parser_.iterate_many(input,
batch_size).get(stream)));
```
But I agree that it could be acceptable, given that the chunker for
newline-delimited data does not have this limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
