alamb opened a new issue, #12357: URL: https://github.com/apache/datafusion/issues/12357
### Is your feature request related to a problem or challenge? DataFuson is growing by almost all measures: community, features, and codebase size which is good. However, this growth is causing challenges such as: 1. Lengthy review cycles (especially for new features). For example (TODO correlated subqueries PR) 2. PRs that are written but then not merged as they seem to be too large in scope (e.g. TODO hugging face PR, flightsql source, etc) 3. Uncertaintly on if we should add features (e.g. for example, should we be adding all the (very cool) DuckDB SQL extension features to make the default SQL engine as easy as possible or should those be extension packages) As described in the [Design Goals](https://docs.rs/datafusion/latest/datafusion/index.html#design-goals), it is important for DataFusion to: 1. Work “out of the box”: Provide a very fast, world class query engine with minimal setup or required configuration. 2. Customizable everything: All behavior should be customizable by implementing traits. However, this description doesn't offer any specific criteria about which features should be in the core (to work "out of the box") and which should be implemented as extensions I am worried that if we take all possiblely useful features, the DataFusion core will become unmanageble / unmaintainable. Already we are struggling with review capacity (it takes days / weeks to review new feautre PRs) ### Describe the solution you'd like I would like a clearly articulated set of criteria of when features should be added to the core vs when they should be in downstream projects / crates built with the extension APIs ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
