chitralverma opened a new issue, #5943:
URL: https://github.com/apache/opendal/issues/5943
### Feature Description
Current implementation of `read` and `open` are all eager operations and the
data is read into memory before its written to destination.
suggestion is to expose 2 python interfaces `AsyncReader` and `AsyncWriter`
which mimic rust side implementation and allow users to work on datasets which
are larger than memory in streaming manner.
### Problem and Solution
- Support all read and write options exposed by `reader_with` and
`writer_with`
- Use `into_stream` and `into_sink` instead of `into_future_async*` which is
not zero-cost
- Expose 2 python interfaces `AsyncReader` and `AsyncWriter` to facilitate
this
- `AsyncReader` contains,
- a new method called `read_into(...)` which accepts an `AsyncWriter`
- a new method called `read_chunks(...)` which yields an async generator
allowing streaming chunks instead of returning everything at once.
- `AsyncWriter` contains,
- a new method called `write_from(...)` which accepts an `AsyncReader`
- a new method called `write_chunks(...)` which accepts a generator of
bytes and writes them in parallel to destination
API can look something like,
```
import opendal
import asyncio
async def main():
src = opendal.AsyncOperator("fs", root="/tmp")
dest = opendal.AsyncOperator("fs", ...)
async with (
await src.open("src_path", "rb", ...) as reader,
await dest.open("dest_path", "wb", ...) as writer,
):
async for chunk in reader.read_chunks(...):
# Do whatever
print(f"Received chunk: {len(chunk)} bytes")
# OR
l = writer.write_from(reader)
# OR
l = reader.read_into(writer)
asyncio.run(main())
```
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [x] Yes, I am willing to contribute to the development of this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
