comath opened a new pull request, #5165:
URL: https://github.com/apache/arrow-rs/pull/5165
# Which issue does this PR close?
Closes #5153
# Rationale for this change
There are ways to memmap buffers, if you know the header information, but no
easy way to use the existing IPC file format in a zero-copy way.
# What changes are included in this PR?
Reused the IPC reader code to read the header and footer information from an
IPC file, but swapped out the read/copy of the buffer for the Buffer creation
with `Buffer::from_custom_allocation`. Minor savings can be had by not
allocating the arbitrary-sized footer and message read, but the easy way to do
this requires a nightly API in the Cursor object.
This duplicated some code, but it's messy to deduplicate because most of it
is boilerplate around reading the flatbuffer messages. The better way to
deduplicate is to change the `FileReader` to optionally accept a buffer and
conditionally use that in the 2 locations where buffers are actually read.
Perfectly happy to do it this way as well.
# Are there any user-facing changes?
There's a new struct that mirrors the current IPC reader struct that passes
back zero-copy buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
