[ 
https://issues.apache.org/jira/browse/SPARK-53512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuchuan Huang updated SPARK-53512:
----------------------------------
    Description: 
Currently, the PushDownUtils has following matching cases.
{code:java}
 object PushDownUtils {
  def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression])
    : (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = {
    scanBuilder match {
      case r: SupportsPushDownFilters => ...  // public interface extends 
ScanBuilder
      case r: SupportsPushDownV2Filters => ... // public interface extends 
ScanBuilder
      case f: FileScanBuilder => ... // which extends with 
SupportsPushDownCatalystFilters
      case _ => (Left(Nil), filters)
}  {code}
As a result, when a new scanBuilderĀ  wants to SupportsPushDownCatalystFilters, 
it will not be picked up by PushDownUtils. To better unify these filter 
pushdown traits, it would be better if the matching cases are like following:
{code:java}
scanBuilder match { 
    case r: SupportsPushDownFilters => ... // public interface extends 
ScanBuilder
    case r: SupportsPushDownV2Filters => ... // public interface extends 
ScanBuilder
    case r: SupportsPushDownCatalystFilters => ... // trait extends ScanBuilder
    case _ => (Left(Nil), filters)
}{code}

> Better unification of DSv2 PushDownUtils
> ----------------------------------------
>
>                 Key: SPARK-53512
>                 URL: https://issues.apache.org/jira/browse/SPARK-53512
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.1.0
>            Reporter: Yuchuan Huang
>            Priority: Critical
>
> Currently, the PushDownUtils has following matching cases.
> {code:java}
>  object PushDownUtils {
>   def pushFilters(scanBuilder: ScanBuilder, filters: Seq[Expression])
>     : (Either[Seq[sources.Filter], Seq[Predicate]], Seq[Expression]) = {
>     scanBuilder match {
>       case r: SupportsPushDownFilters => ...  // public interface extends 
> ScanBuilder
>       case r: SupportsPushDownV2Filters => ... // public interface extends 
> ScanBuilder
>       case f: FileScanBuilder => ... // which extends with 
> SupportsPushDownCatalystFilters
>       case _ => (Left(Nil), filters)
> }  {code}
> As a result, when a new scanBuilderĀ  wants to 
> SupportsPushDownCatalystFilters, it will not be picked up by PushDownUtils. 
> To better unify these filter pushdown traits, it would be better if the 
> matching cases are like following:
> {code:java}
> scanBuilder match { 
>     case r: SupportsPushDownFilters => ... // public interface extends 
> ScanBuilder
>     case r: SupportsPushDownV2Filters => ... // public interface extends 
> ScanBuilder
>     case r: SupportsPushDownCatalystFilters => ... // trait extends 
> ScanBuilder
>     case _ => (Left(Nil), filters)
> }{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to