Ganesha S created SPARK-58551:
---------------------------------
Summary: Python Data Sources Limit Pushdown API
Key: SPARK-58551
URL: https://issues.apache.org/jira/browse/SPARK-58551
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.2.0
Reporter: Ganesha S
Python Data Sources cannot use a query's LIMIT to reduce the work they do.
A \{{DataSourceReader}} always plans its full set of partitions and reads at
Arrow batch
granularity (10,000 rows by default), so a \{{LIMIT 5}} over a REST or
database-backed source can
still cost many requests or a full extract. The reader has no way to learn that
the query only
needs a few rows, and therefore cannot add a \{{LIMIT}} clause, set a page size
parameter, or plan
fewer partitions.
JVM DSv2 sources already have this capability through
\{{SupportsPushDownLimit}}; the Python API
exposes only \{{pushFilters}} (SPARK-51271). This is the limit-pushdown
counterpart, alongside
SPARK-51713 for column pruning.
h3. Proposal
Add an optional \{{DataSourceReader.pushLimit(limit) -> bool}} method, called
once during planning
before \{{partitions()}} and \{{read()}}, returning whether the reader will use
the limit to read
less data. \{{PythonScanBuilder}} mixes in \{{SupportsPushDownLimit}} to drive
it.
Gated by a new internal config \{{spark.sql.python.limitPushdown.enabled}}
(default false),
mirroring \{{spark.sql.python.filterPushdown.enabled}}, since it costs one
additional Python
worker invocation during planning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
