alamb opened a new issue, #26684: URL: https://github.com/apache/datafusion/issues/26684
- part of https://github.com/apache/datafusion/issues/21036 - Follow on to https://github.com/apache/datafusion/issues/22882 As I have said before, it is still mind boggling 🤯 to me that DataFusion has gotten so far without central planning authority, but the decentralized model has served us (very) well and I don't think we need to change. However, [the rise of AI generated code](https://github.com/apache/datafusion/issues/26257), especially in the last few months, has been challenging this model as it is so easy to generate changes, I think the collective attention of the community has been overwhelmed and we are thrashing. See for example these discussions - #23839 from @jayzhan211 - https://github.com/apache/datafusion/discussions/24413 from @Jefffrey - https://github.com/apache/datafusion/pull/24510 - https://github.com/apache/datafusion/pull/25973 from @2010YOUY01 What I suggest is that a few maintainers agree to help one or more projects along and then prioritize reviews of related PRs. As always, I expect DataFusion priorities to be driven by what the maintainers want to help drive to completion. However, I also personally think the community would benefit from some agreement on higher level "Light House" areas to focus our collective attention on before we wade into the sea of PRs. Since I am just one voice, I don't get to choose what everyone else works on, but I here are some of my suggestions to get the conversation flowing. # Tier 1: Correctness What: Ensure answers are always correct Goal: Have DataFusion be the standard to compare against Tracking Epic: - https://github.com/apache/datafusion/issues/26672 # Tier 1: Performance (TBD) I think it would be great to agree on or two areas to really focus and improve performance. My suggestion is either Parquet IO Patterns and Join benchmark performance, but I think there are other compelling areas to Some options: ### Performance Option: Parquet IO patterns What: Make the I/O pattern of Parquet scans deliberate and configurable (one request per row group, bounded prefetch, or batch-granular fetch) instead of a hard coded trade-off between latency and memory Goal: lower time to first batch and peak memory per partition, overlap I/O with decode, and let the scan adapt to the storage system (object store vs local SSD) Recent Epics: - https://github.com/apache/datafusion/issues/26682 ### Performance Option: Join Benchmark Performance What: Improve performance in Join heavy benchmarks Goal: ensure we keep DataFusion at the top of benchmarks (especially join heavy ones), Recent Epics: - https://github.com/apache/datafusion/issues/26661 - https://github.com/apache/datafusion/issues/26169 ### Performance Option: Filter Pushdown What: Enable Parquet filter pushdown (late materialization) by default and finish the logical filter / leaf expression pushdown work Goal: evaluate filters as early as possible (during the scan where it helps) without performance regressions Recent Epics: - https://github.com/apache/datafusion/issues/22883 - https://github.com/apache/datafusion/issues/20324 - https://github.com/apache/datafusion/issues/25459 - https://github.com/apache/datafusion/issues/15512 ### Performance Option: Nested and Semi-structured Data What: Improve struct field access, nested schema pruning and Variant support in the Parquet scan Goal: make queries over nested columns as efficient as over flat ones Recent Epics: - https://github.com/apache/datafusion/issues/24119 - https://github.com/apache/datafusion/issues/21308 - https://github.com/apache/datafusion/issues/26517 Other potential projects we might want to rally around: - Spilling Joins: https://github.com/apache/datafusion/issues/24768 - blocked / chunked memory in hash aggregation: https://github.com/apache/datafusion/issues/24704 - Optimize range partitioning for joins and distributed pipelines: https://github.com/apache/datafusion/issues/25421 - Subplan materialization: https://github.com/apache/datafusion/issues/22676 - Adaptive predicate evaluation: https://github.com/apache/datafusion/issues/22883 # Things I personally plan to deemphasize * Seemingly random improvements to random parts of the code, especially if it [isn't from a DataFusion user](https://andrew.nerdnetworks.org/blog/2026/10/19/my-plan-for-reviewing-prs-in-open-source-projects-in-the-age-of-ai/) or from a [Prolific Author](https://github.com/apache/datafusion/pulls?q=is%3Apr+state%3Aopen+label%3A%22Prolific+Author%22) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
