alamb opened a new issue, #14444: URL: https://github.com/apache/datafusion/issues/14444
### Is your feature request related to a problem or challenge? Historically DataFusion was one (very) large crate `datafusion,` and as it grew bigger we extracted various functionality into separate crates. This leads to both faster compile times (as the crates can be compiled in parallel) as well easier to navigate code (as the crates force a cleaner dependency separation) As described by @waynexia the build time of DataFusion has been growing, - https://github.com/apache/datafusion/issues/14256 Some of this is due to the fact there is more code / more features to test. However a non trivial part of the long compile time is the time taken to compile the `datafusion` / core crate in https://github.com/apache/datafusion/tree/main/datafusion/core <img width="2441" alt="Image" src="https://github.com/user-attachments/assets/fde0c93e-77d4-47fb-8daf-25c45c5513c1"; /> While we are pursuing additional ways to reduce compile time, I think we should also move more code out of `datafusion/core` into their own crates. We have successfully done this in the past with other projects such as - https://github.com/apache/datafusion/issues/11502 - https://github.com/apache/datafusion/issues/10782 ### Describe the solution you'd like I would like to split out the https://github.com/apache/datafusion/tree/main/datafusion/core/src/datasource from DataFusion core ### Describe alternatives you've considered I think we will end up with several new crates - `datafusion-catalog-listing`: `ListingTable` and associated types like `PartitionedFile` - `datafusion-datasource-parquet`: `ParquetExec` and file firmat - `datafusion-datasource-avro` `AvroExec` and file formats - `datafusion-datasource-arrow` - `datafusion-datasource-json` - `datafusion-datasource-csv` I think we could start by creating `datafusion-catalog-listing` and trying to pull some of the listing table implementation into there and then trying to move one of the simpler datasources out (`datafusion-datasource-arrow` perhaps) ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
