adriangb commented on PR #17242: URL: https://github.com/apache/datafusion/pull/17242#issuecomment-3243531886
@waynexia thanks so much for the input! I agree with most of your points, let's see what @friendlymatthew thinks > FileFormat takes FilsScanConfig and format config stored in it to produce the associated FileOpener Don't we need a `FileOpener` _per partition_? So we need something that is produced by `FileFormat::make_thing(FileScanConfig)` that can then create a FileOpener per partition, right? > This can likely simplify the logic of FileStream by a lot FWIW I also think we should get rid of the partition value handling in `FileStream`, instead we should just replace references to partition columns with literal values like we now do with filters. Those can then be expanded into an array if the projection demands it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
