cloud-fan commented on code in PR #59170:
URL: https://github.com/apache/spark/pull/59170#discussion_r4249313195
##########
python/pyspark/sql/tests/test_python_datasource.py:
##########
@@ -153,6 +153,16 @@ def main(_infile, _outfile):
(1, {1}, (0, 0), 1),
)
+ def test_deserialize_unsupported_filter(self):
+ with mock.patch.dict(os.environ, {"SPARK_PYTHON_RUNTIME":
"PYTHON_WORKER"}):
+ from pyspark.sql.worker.data_source_pushdown_filters import
deserializeFilter
Review Comment:
**Nit (P3):** This import pulls in `plan_data_source_read`, which does
`import pyarrow as pa` at module top level. `PythonDataSourceWorkerUtilsTests`
has no pyarrow guard, so without pyarrow this test errors with
`ModuleNotFoundError`, while the pyarrow-dependent tests in
`BasePythonDataSourceTestsMixin` are skipped. CI installs pyarrow, so this only
shows up in local or downstream runs without it. Could you add
`@unittest.skipIf(not have_pyarrow, pyarrow_requirement_message)` to this test?
Both helpers are already imported in this file.
**Verification:**
- **Behavior:** With pyarrow available, the unsupported-filter worker test
runs and passes, asserting the UNSUPPORTED_FILTER condition and its name
parameter.
- **Regression:** With pyarrow unavailable, the Python data source test
module imports, and the unsupported-filter worker test is skipped rather than
failing on the pyarrow import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
