alamb commented on PR #8021: URL: https://github.com/apache/arrow-datafusion/pull/8021#issuecomment-1804291229
It seems to me one of the core tensions here is that `ListingTable` (the built in table source from DataFusion) uses `ObjectStore`to read data. Therefore, if `append` is removed from `ObjectStore` ListingTable can no longer be used directly to append to existing files (e.g. FIFO), though it can make new files in a directory. Thus one API question is probably "should ListingTable" support the ability to modify / append existing files, or should that functionality be delegated to some other structure, possibly even outside the core DataFusion crate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
