ganeshashree commented on code in PR #57752:
URL: https://github.com/apache/spark/pull/57752#discussion_r3804137264
##########
python/pyspark/sql/worker/data_source_pushdown_filters.py:
##########
@@ -195,14 +204,46 @@ def _main(infile: IO, outfile: IO) -> None:
},
)
+ # Receive the limit to push down. -1 means there is no limit.
+ limit = read_int(infile)
+
# Receive the max arrow batch size.
max_arrow_batch_size = read_int(infile)
assert max_arrow_batch_size > 0, (
"The maximum arrow batch size should be greater than 0, but got "
f"'{max_arrow_batch_size}'"
)
+ enable_limit_pushdown = read_bool(infile)
binary_as_bytes = read_bool(infile)
+ if not enable_limit_pushdown and is_method_overridden(reader,
"pushLimit"):
Review Comment:
Done. This worker now receives the filter-pushdown config alongside the
limit one and runs the same two-method check through a shared
`check_pushdown_not_disabled` helper (used here and in
`plan_data_source_read`), so an overridden `pushFilters` with filter pushdown
disabled raises `DATA_SOURCE_PUSHDOWN_DISABLED` even on the cached limit-only
path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
