metesynnada commented on issue #3740: URL: https://github.com/apache/arrow-rs/issues/3740#issuecomment-1443610520
> I'm not familiar with your use case, but trying to shoehorn streaming through OS-specific filesystem APIs seems a little odd to me. Suppose you read a `ListingTable` in Datafusion. You need to use the `object-store` crate to read the data. More specifically, you would use the `get` API and it would produce a ByteStream (futures::stream) and you can consume it. I was looking for a `put` API similar to this logic, an async writer. However, we will be looking for extending this feature to streaming brokers like Kafka, Kinesis, etc. FIFO was a way to make the ListingTable unifying stream and batch approaches. This is why I was looking for ways for extending the `object-store` crate. Overall, we believe that overfitting the batch solutions is mostly avoidable. However, I understand your concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
