[
https://issues.apache.org/jira/browse/SPARK-58551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha S updated SPARK-58551:
------------------------------
Description:
Python Data Sources cannot use a query's LIMIT to reduce the work they do.
A {{DataSourceReader}} always plans its full set of partitions and reads at
Arrow batch granularity (10,000 rows by default), so a {{LIMIT 5}} over a REST
or database-backed source can still cost many requests or a full extract. The
reader has no way to learn that the query only needs a few rows, and therefore
cannot add a {{LIMIT}} clause, set a page size parameter, or plan fewer
partitions.
JVM DSv2 sources already have this capability through
{{{}SupportsPushDownLimit{}}}; the Python API exposes only {{pushFilters}}
(SPARK-51271). This is the limit-pushdown counterpart, alongside SPARK-51713
for column pruning.
h3. Proposal
Add an optional {{DataSourceReader.pushLimit(limit) -> bool}} method, called
once during planning before {{partitions()}} and {{{}read(){}}}, returning
whether the reader will use the limit to read less data. {{PythonScanBuilder}}
mixes in {{SupportsPushDownLimit}} to drive it.
Gated by a new internal config {{spark.sql.python.limitPushdown.enabled}}
(default false), mirroring {{{}spark.sql.python.filterPushdown.enabled{}}},
since it costs one additional Python worker invocation during planning.
was:
Python Data Sources cannot use a query's LIMIT to reduce the work they do.
A \{{DataSourceReader}} always plans its full set of partitions and reads at
Arrow batch
granularity (10,000 rows by default), so a \{{LIMIT 5}} over a REST or
database-backed source can
still cost many requests or a full extract. The reader has no way to learn that
the query only
needs a few rows, and therefore cannot add a \{{LIMIT}} clause, set a page size
parameter, or plan
fewer partitions.
JVM DSv2 sources already have this capability through
\{{SupportsPushDownLimit}}; the Python API
exposes only \{{pushFilters}} (SPARK-51271). This is the limit-pushdown
counterpart, alongside
SPARK-51713 for column pruning.
h3. Proposal
Add an optional \{{DataSourceReader.pushLimit(limit) -> bool}} method, called
once during planning
before \{{partitions()}} and \{{read()}}, returning whether the reader will use
the limit to read
less data. \{{PythonScanBuilder}} mixes in \{{SupportsPushDownLimit}} to drive
it.
Gated by a new internal config \{{spark.sql.python.limitPushdown.enabled}}
(default false),
mirroring \{{spark.sql.python.filterPushdown.enabled}}, since it costs one
additional Python
worker invocation during planning.
> Python Data Sources Limit Pushdown API
> --------------------------------------
>
> Key: SPARK-58551
> URL: https://issues.apache.org/jira/browse/SPARK-58551
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Ganesha S
> Priority: Major
> Labels: pull-request-available
>
> Python Data Sources cannot use a query's LIMIT to reduce the work they do.
> A {{DataSourceReader}} always plans its full set of partitions and reads at
> Arrow batch granularity (10,000 rows by default), so a {{LIMIT 5}} over a
> REST or database-backed source can still cost many requests or a full
> extract. The reader has no way to learn that the query only needs a few rows,
> and therefore cannot add a {{LIMIT}} clause, set a page size parameter, or
> plan fewer partitions.
> JVM DSv2 sources already have this capability through
> {{{}SupportsPushDownLimit{}}}; the Python API exposes only {{pushFilters}}
> (SPARK-51271). This is the limit-pushdown counterpart, alongside SPARK-51713
> for column pruning.
> h3. Proposal
> Add an optional {{DataSourceReader.pushLimit(limit) -> bool}} method, called
> once during planning before {{partitions()}} and {{{}read(){}}}, returning
> whether the reader will use the limit to read less data.
> {{PythonScanBuilder}} mixes in {{SupportsPushDownLimit}} to drive it.
> Gated by a new internal config {{spark.sql.python.limitPushdown.enabled}}
> (default false), mirroring {{{}spark.sql.python.filterPushdown.enabled{}}},
> since it costs one additional Python worker invocation during planning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
