[
https://issues.apache.org/jira/browse/SPARK-58551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58551:
-----------------------------------
Labels: pull-request-available (was: )
> Python Data Sources Limit Pushdown API
> --------------------------------------
>
> Key: SPARK-58551
> URL: https://issues.apache.org/jira/browse/SPARK-58551
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Ganesha S
> Priority: Major
> Labels: pull-request-available
>
> Python Data Sources cannot use a query's LIMIT to reduce the work they do.
> A \{{DataSourceReader}} always plans its full set of partitions and reads at
> Arrow batch
> granularity (10,000 rows by default), so a \{{LIMIT 5}} over a REST or
> database-backed source can
> still cost many requests or a full extract. The reader has no way to learn
> that the query only
> needs a few rows, and therefore cannot add a \{{LIMIT}} clause, set a page
> size parameter, or plan
> fewer partitions.
> JVM DSv2 sources already have this capability through
> \{{SupportsPushDownLimit}}; the Python API
> exposes only \{{pushFilters}} (SPARK-51271). This is the limit-pushdown
> counterpart, alongside
> SPARK-51713 for column pruning.
> h3. Proposal
> Add an optional \{{DataSourceReader.pushLimit(limit) -> bool}} method, called
> once during planning
> before \{{partitions()}} and \{{read()}}, returning whether the reader will
> use the limit to read
> less data. \{{PythonScanBuilder}} mixes in \{{SupportsPushDownLimit}} to
> drive it.
> Gated by a new internal config \{{spark.sql.python.limitPushdown.enabled}}
> (default false),
> mirroring \{{spark.sql.python.filterPushdown.enabled}}, since it costs one
> additional Python
> worker invocation during planning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
