Thanks, Szehon! This will help address one of the long-standing limitations in DSv2 that is a common cause of regressions or even blockers for DSv2 adoption. I am looking forward to implementation.
- Anton ср, 18 лют. 2026 р. о 14:30 Szehon Ho <[email protected]> пише: > Hi all, > > I would like to propose enhancements for partition filter pushdown, for > DSV2 data sources that support partitioning (ie, those with partition > stats). > > Some DSV2 data sources, for example table formats like Apache Iceberg, > lack partition filtering in many queries, compared to Spark-native data > sources that directly use Catalyst (like Parquet). This proposal can > bridge that gap while simplifying the data source logic. > > JIRA: https://issues.apache.org/jira/browse/SPARK-55596 > SPIP doc: > https://docs.google.com/document/d/17vcw411PxSRLWoK-BiLI56UiNdokLWtovF8JZUlDTOo > > Look forward to comments and feedback. > > Thanks, > Szehon >
