[ https://issues.apache.org/jira/browse/SPARK-53512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuchuan Huang updated SPARK-53512: ---------------------------------- Description: Currently, the PushDownUtils has following matching cases. {code:java} object PushDownUtils { def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression]) : (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = { scanBuilder match { case r: SupportsPushDownFilters => ... // public interface extends ScanBuilder case r: SupportsPushDownV2Filters => ... // public interface extends ScanBuilder case f: FileScanBuilder => ... // which extends with SupportsPushDownCatalystFilters case _ => (Left(Nil), filters) } {code} As a result, when a new scanBuilderĀ wants to SupportsPushDownCatalystFilters, it will not be picked up by PushDownUtils. To better unify these filter pushdown traits, it would be better if the matching cases are like following: {code:java} scanBuilder match { case r: SupportsPushDownFilters => ... // public interface extends ScanBuilder case r: SupportsPushDownV2Filters => ... // public interface extends ScanBuilder case r: SupportsPushDownCatalystFilters => ... // trait extends ScanBuilder case _ => (Left(Nil), filters) }{code} > Better unification of DSv2 PushDownUtils > ---------------------------------------- > > Key: SPARK-53512 > URL: https://issues.apache.org/jira/browse/SPARK-53512 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 4.1.0 > Reporter: Yuchuan Huang > Priority: Critical > > Currently, the PushDownUtils has following matching cases. > {code:java} > object PushDownUtils { > def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression]) > : (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = { > scanBuilder match { > case r: SupportsPushDownFilters => ... // public interface extends > ScanBuilder > case r: SupportsPushDownV2Filters => ... // public interface extends > ScanBuilder > case f: FileScanBuilder => ... // which extends with > SupportsPushDownCatalystFilters > case _ => (Left(Nil), filters) > } {code} > As a result, when a new scanBuilderĀ wants to > SupportsPushDownCatalystFilters, it will not be picked up by PushDownUtils. > To better unify these filter pushdown traits, it would be better if the > matching cases are like following: > {code:java} > scanBuilder match { > case r: SupportsPushDownFilters => ... // public interface extends > ScanBuilder > case r: SupportsPushDownV2Filters => ... // public interface extends > ScanBuilder > case r: SupportsPushDownCatalystFilters => ... // trait extends > ScanBuilder > case _ => (Left(Nil), filters) > }{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org