szehon-ho commented on code in PR #59683: URL: https://github.com/apache/spark/pull/59683#discussion_r4343377903
########## docs/sql-v2-data-sources.md: ########## @@ -280,9 +289,31 @@ for mix-in interfaces. The pushdown order is: 5. **Offset** (`SupportsPushDownOffset`) 6. **Column pruning** (`SupportsPushDownRequiredColumns`) -Each mix-in interface has a method that Spark calls with the relevant operators. The -implementation returns the operators it can handle, and Spark applies the remaining operators -itself. +Each mix-in has a method that Spark calls with the candidate operator, and the return value tells +Spark what it must still evaluate itself after the scan: + +- [`SupportsPushDownFilters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.html): + `pushFilters(filters)` receives the filters Spark could translate to + `org.apache.spark.sql.sources.Filter` and returns the ones that still need to be evaluated + **after** the scan. `pushedFilters()` returns the filters the source accepted. A filter may appear + in both: a source that can only skip whole files or row groups on a predicate reports it as pushed + and also returns it, so Spark re-checks it row by row. Spark evaluates only what `pushFilters` + returns (plus anything it could not translate), so a filter that is neither returned nor applied + by the reader is silently dropped from the query. +- [`SupportsPushDownV2Filters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownV2Filters.html) + (since 3.3.0) is the preferred form of the same contract: `pushPredicates(predicates)` returns + the post-scan predicates as [`Predicate`](#expressions) instances and `pushedPredicates()` + reports the accepted ones, + without the conversion to the V1 `Filter` type. Spark checks for `SupportsPushDownFilters` + first, so a `ScanBuilder` should implement one or the other. +- [`SupportsPushDownRequiredColumns`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownRequiredColumns.html): + `pruneColumns(requiredSchema)` passes the narrowest schema the query needs. Pruning may be + partial (for example top-level columns only), and `Scan.readSchema()` must report whatever + pruning was actually applied. +- `pushLimit`, `pushTopN`, `pushOffset`, and `pushAggregation` return `true` when the source Review Comment: should we mention the mixins? ex: SupportsPushDownLimit.pushLimit, SupportsPushDownTopN.pushTopN, SupportsPushDownOffset.pushOffset, and SupportsPushDownAggregates.pushAggregation return true when accepted. For limit and top-N, isPartiallyPushed() indicates whether Spark must retain the operator above the scan. For aggregation, supportCompletePushDown(aggregation) indicates whether the source fully handles the aggregation. ########## docs/sql-v2-data-sources.md: ########## @@ -280,9 +289,31 @@ for mix-in interfaces. The pushdown order is: 5. **Offset** (`SupportsPushDownOffset`) 6. **Column pruning** (`SupportsPushDownRequiredColumns`) -Each mix-in interface has a method that Spark calls with the relevant operators. The -implementation returns the operators it can handle, and Spark applies the remaining operators -itself. +Each mix-in has a method that Spark calls with the candidate operator, and the return value tells +Spark what it must still evaluate itself after the scan: + +- [`SupportsPushDownFilters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.html): + `pushFilters(filters)` receives the filters Spark could translate to + `org.apache.spark.sql.sources.Filter` and returns the ones that still need to be evaluated + **after** the scan. `pushedFilters()` returns the filters the source accepted. A filter may appear + in both: a source that can only skip whole files or row groups on a predicate reports it as pushed + and also returns it, so Spark re-checks it row by row. Spark evaluates only what `pushFilters` + returns (plus anything it could not translate), so a filter that is neither returned nor applied + by the reader is silently dropped from the query. +- [`SupportsPushDownV2Filters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownV2Filters.html) Review Comment: nit: please put this first (as its suggested one now). Maybe put the note on the other one (that it's pre-3.3) ########## docs/sql-v2-data-sources.md: ########## @@ -280,9 +289,31 @@ for mix-in interfaces. The pushdown order is: 5. **Offset** (`SupportsPushDownOffset`) 6. **Column pruning** (`SupportsPushDownRequiredColumns`) -Each mix-in interface has a method that Spark calls with the relevant operators. The -implementation returns the operators it can handle, and Spark applies the remaining operators -itself. +Each mix-in has a method that Spark calls with the candidate operator, and the return value tells +Spark what it must still evaluate itself after the scan: + +- [`SupportsPushDownFilters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.html): Review Comment: Note: we could add the other mixins too then (can be another pr) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
