comath commented on PR #5165:
URL: https://github.com/apache/arrow-rs/pull/5165#issuecomment-1841768298
I have a way that goes:
```rust
/// Arrow File reader
pub struct FileReader<R: Read + Seek> {
/// Buffered file reader that supports reading and seeking
reader: BufReader<R>,
/// Optional Buffer for when we want to map the underlying data without
copying
buffer: Option<BufferReader>,
......
}
impl FileReader<Cursor<BufferReader>> {
pub fn try_new_map(buffer: Buffer, projection: Option<Vec<usize>>) ->
Result<Self, ArrowError> {
let buffer = BufferReader { inter: buffer };
let cursor = Cursor::new(buffer.clone());
Self::try_new_inter(cursor, projection, Some(buffer))
}
}
impl<R: Read + Seek> FileReader<R> {
pub fn try_new(reader: R, projection: Option<Vec<usize>>) ->
Result<Self, ArrowError> {
Self::try_new_inter(reader, projection, None)
}
fn try_new_inter(
reader: R,
projection: Option<Vec<usize>>,
buffer: Option<BufferReader>,
) -> Result<Self, ArrowError> {
let mut reader = BufReader::new(reader);
.......
}
}
```
It then checks if the optional buffer is there during the dictionary loop
and `maybe_next` call. If there is a buffer it maps the memory instead of
copying it. The `BufferReader` is a struct wrapping `Buffer` that just
implements `AsRef<[u8]>` so that we can put a `Cursor` on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
