cloud-fan commented on code in PR #57752:
URL: https://github.com/apache/spark/pull/57752#discussion_r3818975817
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/PythonScanBuilder.scala:
##########
@@ -29,25 +30,59 @@ class PythonScanBuilder(
outputSchema: StructType,
options: CaseInsensitiveStringMap)
extends ScanBuilder
- with SupportsPushDownFilters {
+ with SupportsPushDownFilters
+ with SupportsPushDownLimit {
private var supportedFilters: Array[Filter] = Array.empty
+ // All filters handed to `pushFilters`, kept so that `pushLimit` can replay
them and bring the
+ // Python reader back to the same state before calling `pushLimit` on it.
+ private var allFilters: Array[Filter] = Array.empty
+ private var pushedLimit: Option[Int] = None
+ // Read info (partitions + read function) produced as a side effect of
filter/limit pushdown,
+ // carried into the `PythonScan` so it stays scoped to this scan. It must
NOT be stored on the
+ // provider-scoped `PythonDataSourceV2`: a single data source instance is
shared by every scan
+ // built from it -- e.g. a base DataFrame and its `.limit(n)` reuse the same
relation -- so a
+ // pushdown-specific read function stored there would leak into an unrelated
scan and make it
+ // read too few rows.
+ private var readInfo: Option[PythonDataSourceReadInfo] = None
+ // True when a pushdown pass ran but produced no read info, so build() must
plan the read
+ // itself. Two paths set it: (1) the filter-pushdown pass defers planning
while limit pushdown
+ // is enabled and a limit pass might follow; (2) a pushed limit was
rejected, so planning falls
+ // back to a fresh reader. In both, build() plans the read with the filters
only. This is
+ // distinct from `readInfo.isEmpty` when both pushdowns are disabled --
there no pushdown pass
Review Comment:
**Nit:**
This sentence is missing a verb.
```suggestion
// distinct from `readInfo.isEmpty` when both pushdowns are disabled -- in
that case, no pushdown pass
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
