metesynnada commented on issue #3740: URL: https://github.com/apache/arrow-rs/issues/3740#issuecomment-1442893191
I am OK with the separation. The main idea behind adding `AsyncWrite` support was using the object store's `put` and `put_multipart` APIs in Datafusion. https://github.com/apache/arrow-rs/blob/350867436ab3477bafd7008355286378ab37045f/object_store/src/lib.rs#L191-L217 I was planning to add a new API like `put_multipart` which returns only `Box<dyn AsyncWrite + Unpin + Send>` to support single files without multipart support. What do you think about this feature? It would look like ``` async fn put_singlepart( &self, location: &Path, ) -> Result<Box<dyn AsyncWrite + Unpin + Send>>; ``` Consider the `put_multipart` implementation of the local file system, which generates a temporary file and finalizes its name upon reaching the `LocalUploadState::ShuttingDown` stage to complete the writing process. However, when dealing with FIFO cases, what would you recommend? Would it be still appropriate to use the `object_store` for individual files? In Datafusion, my approach would be to verify if the underlying source is unbounded and, if so, utilize the `put_singlepart()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
