[
https://issues.apache.org/jira/browse/SPARK-58551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-58551.
---------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 57752
[https://github.com/apache/spark/pull/57752]
> Python Data Sources Limit Pushdown API
> --------------------------------------
>
> Key: SPARK-58551
> URL: https://issues.apache.org/jira/browse/SPARK-58551
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Ganesha S
> Assignee: Ganesha S
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Python Data Sources cannot use a query's LIMIT to reduce the work they do.
> A {{DataSourceReader}} always plans its full set of partitions and reads at
> Arrow batch granularity (10,000 rows by default), so a {{LIMIT 5}} over a
> REST or database-backed source can still cost many requests or a full
> extract. The reader has no way to learn that the query only needs a few rows,
> and therefore cannot add a {{LIMIT}} clause, set a page size parameter, or
> plan fewer partitions.
> JVM DSv2 sources already have this capability through
> {{{}SupportsPushDownLimit{}}}; the Python API exposes only {{pushFilters}}
> (SPARK-51271). This is the limit-pushdown counterpart, alongside SPARK-51713
> for column pruning.
> h3. Proposal
> Add an optional {{DataSourceReader.pushLimit(limit) -> bool}} method, called
> once during planning before {{partitions()}} and {{{}read(){}}}, returning
> whether the reader will use the limit to read less data.
> {{PythonScanBuilder}} mixes in {{SupportsPushDownLimit}} to drive it.
> Gated by a new internal config {{spark.sql.python.limitPushdown.enabled}}
> (default false), mirroring {{{}spark.sql.python.filterPushdown.enabled{}}},
> since it costs one additional Python worker invocation during planning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
