allisonwang-db commented on code in PR #57752:
URL: https://github.com/apache/spark/pull/57752#discussion_r3798749239
##########
python/pyspark/sql/worker/data_source_pushdown_filters.py:
##########
@@ -195,14 +204,46 @@ def _main(infile: IO, outfile: IO) -> None:
},
)
+ # Receive the limit to push down. -1 means there is no limit.
+ limit = read_int(infile)
+
# Receive the max arrow batch size.
max_arrow_batch_size = read_int(infile)
assert max_arrow_batch_size > 0, (
"The maximum arrow batch size should be greater than 0, but got "
f"'{max_arrow_batch_size}'"
)
+ enable_limit_pushdown = read_bool(infile)
binary_as_bytes = read_bool(infile)
+ if not enable_limit_pushdown and is_method_overridden(reader,
"pushLimit"):
Review Comment:
Could we also preserve the disabled pushFilters validation in this worker?
With filter pushdown disabled, limit pushdown enabled, and a reader overriding
pushFilters, load().limit(1).collect() succeeds instead of raising
DATA_SOURCE_PUSHDOWN_DISABLED. The limit path caches readInfo, so the
validation in plan_data_source_read never runs. I reproduced this with a
regression test. Please pass the filter-pushdown configuration to this worker,
perform the same two-method validation, and add coverage for this configuration
combination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
