alamb commented on issue #7994: URL: https://github.com/apache/arrow-datafusion/issues/7994#issuecomment-1806844489
I agree that starting with a basic `StreamingTable` abstraction, and copying over only API features from the current`ListingTable` that are needed (and not replicating APIs until necessary) makes a lot of sense > I would like to request that we don't overload the file format abstractions for streaming use-cases as that is part of the confusion I am trying to eliminate. Do you mean the [`FileFormat`](https://docs.rs/datafusion/latest/datafusion/datasource/file_format/trait.FileFormat.html) trait? At the moment, it seems like it is tightly tied to ObjectStore both in the `infer` methods and [create_writer_physical_plan](https://docs.rs/datafusion/latest/datafusion/datasource/file_format/trait.FileFormat.html#method.create_writer_physical_plan) via [FileSinkConfig](https://docs.rs/datafusion/latest/datafusion/datasource/physical_plan/struct.FileSinkConfig.html) so I agree it is not clear how to make this easily fit into the streaming design My understanding that the StreamingTable would an analgous trait (like `StreamFormat` perhaps). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
