pitrou commented on code in PR #50945:
URL: https://github.com/apache/arrow/pull/50945#discussion_r4005219327
##########
cpp/src/arrow/json/chunker.cc:
##########
@@ -165,14 +92,94 @@ class ParsingBoundaryFinder : public BoundaryFinder {
int64_t* out_pos, int64_t* num_found) override {
return Status::NotImplemented("ParsingBoundaryFinder::FindNth");
}
+
+ private:
+ MemoryPool* pool_;
+ simdjson::ondemand::parser parser_;
+ // A persistent buffer to keep padded contents for simdjson.
+ // This should be more efficient than allocating a new padded_string
everytime.
+ std::shared_ptr<ResizableBuffer> buffer_;
+
+ Result<simdjson::padded_string_view> GetPaddedStringView(std::string_view
partial,
+ std::string_view
block = {}) {
+ const auto data_size = partial.size() + block.size();
+ const auto required_size = data_size + simdjson::SIMDJSON_PADDING;
+ if (!buffer_) {
+ ARROW_ASSIGN_OR_RAISE(buffer_, AllocateResizableBuffer(
+ /*size=*/0, pool_));
Review Comment:
That would be a bit different, because `Reserve` only adjusts the capacity,
not the size.
Currently, this doesn't make any difference in the allocation behavior, but
we could imagine that it would in the future.
(one possible improvement is that it's really a piece of scratch space: we
don't care about the old contents, but resizing or reserving the buffer copies
the old contents over anyway; see https://github.com/apache/arrow/pull/40774
for an experiment on that topic)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
