chitralverma opened a new issue, #7909:
URL: https://github.com/apache/opendal/issues/7909
# Feature Description
Add an optional read path that fills a **caller-owned destination buffer**
directly, so that capable services and transports can write bytes into memory
the caller already owns without OpenDAL first materializing an intermediate
owned `Buffer`/`Bytes`.
Today OpenDAL's reader pipeline always yields owned `Buffer`/`Bytes` chunks.
The closest public API, `Reader::read_into(&mut impl BufMut, range)`, still
consumes those owned chunks from the reader stream and copies them into the
caller's `BufMut`:
```rust
pub async fn read_into(
&self,
buf: &mut impl BufMut,
range: impl Into<BytesRange>,
) -> Result<usize> {
let mut stream = self.clone().into_stream(range).await?;
let mut read = 0;
loop {
let Some(bs) = stream.try_next().await? else {
return Ok(read);
};
read += bs.len();
buf.put(bs); // copy into caller memory
}
}
```
With `&mut [u8]` as the `BufMut`, `bytes::BufMut::put_slice` performs a
`copy_from_slice` into the caller's allocation. So the effective path is:
```text
backend/transport -> OpenDAL Buffer/Bytes -> caller-owned buffer
one memcpy
```
The request is for an API where the caller supplies the destination and, on
capable backends, the backend writes into it directly:
```text
backend/transport -> caller-owned destination (no intermediate data
buffer)
```
# Problem and Solution
## Problem
Many callers do not own the shape of their destination memory — it is
dictated by a foreign API, a hardware constraint, or a preallocated pool — and
they require OpenDAL to *fill that specific memory*. In these cases OpenDAL's
`Buffer`-first model forces an extra copy that has no semantic purpose.
Concrete cases where the caller owns the destination and cannot adopt an
OpenDAL `Buffer`:
- **FFI / foreign filesystem bridges.** C/C++ `pread`-style contracts
(`read(void *buffer, size_t nr_bytes, off_t offset)`) hand OpenDAL a raw
destination pointer by value and require it to be filled. The buffer is owned
and freed by the foreign runtime; OpenDAL cannot return or substitute its own
`Buffer`, so today every such bridge pays a memcpy per read.
- **Columnar / Parquet readers.** Arrow-style readers assemble column chunks
into preallocated, correctly-aligned buffers (often part of a larger arena or a
`MutableBuffer`). They want positioned range reads landed straight into those
buffers rather than into a transient `Buffer` that is immediately copied and
dropped.
- **Buffer pools and arenas.** Query engines and network servers read into
slabs from a fixed-capacity pool to bound allocation and improve cache
locality. A `Buffer`-returning read defeats the pool because the data lands in
OpenDAL-owned memory first.
- **Special-purpose memory.** Destinations such as `mmap` regions, GPU
pinned host memory, `O_DIRECT`-aligned buffers, or `io_uring`-registered
buffers must be written in place. They cannot be replaced by an OpenDAL
allocation.
The extra copy is most visible exactly where network latency does *not* hide
memory bandwidth: local `fs`, fast attached storage, the in-memory `memory`
service, and cache hits (e.g. a warm cache layer). For large scans, many
concurrent positioned reads, or high-throughput links, the redundant copy shows
up as CPU cycles, memory bandwidth, and allocator pressure.
## Proposed solution
Introduce a caller-owned buffer read at two levels, gated by capability with
a safe fallback:
1. **Public high-level method** on `Reader`, e.g.
```rust
/// Read exactly `dst.len()` bytes for `range` into caller-owned memory.
pub async fn read_exact_into(
&self,
dst: &mut [u8],
range: Range<u64>,
) -> Result<()>;
```
with a fixed-capacity, `pread`-like contract that matches positioned
reads.
2. **Optional raw capability** on the positioned-read path so capable
services fill the destination directly, for example an additive method
alongside the existing `PositionRead::read_at`:
```rust
async fn read_at_into(
handle: &Self::Handle,
offset: u64,
dst: &mut [u8],
) -> Result<usize>;
```
Services that can write into a foreign destination (starting with `fs`
via platform positioned reads, and `memory` via `copy_from_slice`) implement
it. Everything else falls back to the existing `read_at` + one copy, so no
service
is required to change and behavior is preserved.
The existing `read`/`read_into`/`fetch`/stream APIs and their
zero-copy-within-OpenDAL semantics stay exactly as they are; this is purely
additive.
A pragmatic first increment: land the raw capability + fs implementation +
generic fallback + the `Reader` method, benchmark it against `read_into`, then
consider extending to transports and other services. A full design (capability
advertisement, layer composition — retry/timeout/cancellation with borrowed
memory, short-read/EOF semantics, `MaybeUninit` vs initialized slices) is worth
an RFC; i'll be happy to draft that if this issue makes sense.
# Additional Context
- [ ] Yes, I am willing to contribute to the development of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
