alamb commented on issue #7317: URL: https://github.com/apache/arrow-datafusion/issues/7317#issuecomment-1714295732
> It almost seems like the datasource module should live in datafusion-common? I think this would be very challenging as the datasource module has physical plans in it as well > @alamb What do you think of splitting file_format out into its own crate similar to what you're doing with @judahrand -- If you are referring to https://github.com/apache/arrow-datafusion/tree/main/datafusion/core/src/datasource/file_format it may be challenging given that it depends on ExecutionPlan (which is in datafusion-core at the moment) I think the key dependency is that FileScanConfig has embedded `PhysicalExpr` which are not yet created during planning https://github.com/apache/arrow-datafusion/blob/4abae3b4fadeadc8a368155e14186016117529c8/datafusion/core/src/datasource/physical_plan/file_scan_config.rs#L73C1-L108 What I would recommend is updating the FileScanConfig if possible (or making an equivalent in LogicalPlan) so that it represents sort order in terms of `Expr` (the logical expressions) and then converting `Expr` --> `PhysicalExpr` during physical planning -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
