lisancao commented on code in PR #59683: URL: https://github.com/apache/spark/pull/59683#discussion_r4343692454
########## docs/sql-v2-data-sources.md: ########## @@ -280,9 +289,31 @@ for mix-in interfaces. The pushdown order is: 5. **Offset** (`SupportsPushDownOffset`) 6. **Column pruning** (`SupportsPushDownRequiredColumns`) -Each mix-in interface has a method that Spark calls with the relevant operators. The -implementation returns the operators it can handle, and Spark applies the remaining operators -itself. +Each mix-in has a method that Spark calls with the candidate operator, and the return value tells +Spark what it must still evaluate itself after the scan: + +- [`SupportsPushDownFilters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.html): + `pushFilters(filters)` receives the filters Spark could translate to + `org.apache.spark.sql.sources.Filter` and returns the ones that still need to be evaluated + **after** the scan. `pushedFilters()` returns the filters the source accepted. A filter may appear + in both: a source that can only skip whole files or row groups on a predicate reports it as pushed + and also returns it, so Spark re-checks it row by row. Spark evaluates only what `pushFilters` + returns (plus anything it could not translate), so a filter that is neither returned nor applied + by the reader is silently dropped from the query. +- [`SupportsPushDownV2Filters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownV2Filters.html) Review Comment: Done in adbf8a8: V2 filters lead the entry (and the numbered list), and `SupportsPushDownFilters` is described as the pre-3.3 form over the V1 `Filter` type. ########## docs/sql-v2-data-sources.md: ########## @@ -280,9 +289,31 @@ for mix-in interfaces. The pushdown order is: 5. **Offset** (`SupportsPushDownOffset`) 6. **Column pruning** (`SupportsPushDownRequiredColumns`) -Each mix-in interface has a method that Spark calls with the relevant operators. The -implementation returns the operators it can handle, and Spark applies the remaining operators -itself. +Each mix-in has a method that Spark calls with the candidate operator, and the return value tells +Spark what it must still evaluate itself after the scan: + +- [`SupportsPushDownFilters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownFilters.html): + `pushFilters(filters)` receives the filters Spark could translate to + `org.apache.spark.sql.sources.Filter` and returns the ones that still need to be evaluated + **after** the scan. `pushedFilters()` returns the filters the source accepted. A filter may appear + in both: a source that can only skip whole files or row groups on a predicate reports it as pushed + and also returns it, so Spark re-checks it row by row. Spark evaluates only what `pushFilters` + returns (plus anything it could not translate), so a filter that is neither returned nor applied + by the reader is silently dropped from the query. +- [`SupportsPushDownV2Filters`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownV2Filters.html) + (since 3.3.0) is the preferred form of the same contract: `pushPredicates(predicates)` returns + the post-scan predicates as [`Predicate`](#expressions) instances and `pushedPredicates()` + reports the accepted ones, + without the conversion to the V1 `Filter` type. Spark checks for `SupportsPushDownFilters` + first, so a `ScanBuilder` should implement one or the other. +- [`SupportsPushDownRequiredColumns`](api/java/org/apache/spark/sql/connector/read/SupportsPushDownRequiredColumns.html): + `pruneColumns(requiredSchema)` passes the narrowest schema the query needs. Pruning may be + partial (for example top-level columns only), and `Scan.readSchema()` must report whatever + pruning was actually applied. +- `pushLimit`, `pushTopN`, `pushOffset`, and `pushAggregation` return `true` when the source Review Comment: Took your wording, with each mix-in linked: `SupportsPushDownLimit.pushLimit`, `SupportsPushDownTopN.pushTopN`, `SupportsPushDownOffset.pushOffset` and `SupportsPushDownAggregates.pushAggregation`, then `isPartiallyPushed()` for limit and top-N and `supportCompletePushDown(aggregation)` for aggregates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
