Xuanwo opened a new issue, #5738: URL: https://github.com/apache/arrow-rs/issues/5738
This proposal intends to make `AsyncArrowWriter` accepts a new trait called `AsyncFileWriter` instead like what we do for `ParquetRecordBatchStream`. `AsyncArrowWriter` accepts `AsyncWrite` currently: https://github.com/apache/arrow-rs/blob/19a3bb0d264fdb06294ea8811c7e6514030f01dc/parquet/src/arrow/async_writer/mod.rs#L100-L109 `AsyncWrite` is a low-level, poll-based API. Users with writers that provide `async fn write()` will need to encapsulate it within a manually written future state machine. For example: https://github.com/apache/arrow-rs/blob/08af4710fcd2b56a7624db9b9d97e6715a952cb0/object_store/src/buffered.rs#L306-L313 **Describe the solution you'd like** I propose to make `AsyncArrowWriter` accepts a new trait called `AsyncFileWriter`: ```rust pub struct AsyncArrowWriter<W> { /// Underlying sync writer sync_writer: ArrowWriter<Vec<u8>>, /// Async writer provided by caller async_writer: W, } impl<W: AsyncFileWriter + Unpin + Send> AsyncArrowWriter<W> { ... } pub trait AsyncFileWriter: Send { async fn write(&mut self, bs: Bytes) -> Result<()>; async fn complete(&mut self) -> Result<()>; } impl<T: AsyncWrite> AsyncFileWriter for T { ... } ``` **Describe alternatives you've considered** Not yet. **Additional context** `ParquetRecordBatchStream` accetps `AsyncFileReader`: https://github.com/apache/arrow-rs/blob/f38283b49b29f77e1bb2b0b2af07718724db3285/parquet/src/arrow/async_reader/mod.rs#L123-L147 --- I'm willing help implement this proposal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
