Xinyao Zhang created SPARK-60538:
------------------------------------
Summary: Guard optional dependencies in worker-environment and
streaming tester tests
Key: SPARK-60538
URL: https://issues.apache.org/jira/browse/SPARK-60538
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Xinyao Zhang
Parent: SPARK-60531.
The Python 3.15 CI image omits pandas and PyArrow. PythonWorkerEnvTests
nevertheless executes dependent cases and reports 10 errors. test_tws_tester
imports pandas before its class-level skip can apply. Two Arrow-specific worker
tests can also silently fall back to regular Python execution when dependencies
are absent.
Evidence: https://github.com/apache/spark/actions/runs/40032262075.
Proposed fix: add dependency guards to the affected methods in
PythonWorkerEnvMixin, retain regular Python worker coverage, and conditionally
import pandas and pandas-dependent helpers in test_tws_tester.py.
Implementation: https://github.com/apache/spark/pull/59773.
Validation: on Python 3.15.0rc1 without pandas/PyArrow, 12 dependent worker
cases and 60 streaming tester cases skip cleanly. Dependency selection was
checked under all four pandas/PyArrow combinations on Python 3.12. Fork CI
passed 32 Classic worker tests, 32 Connect parity tests, and 60 streaming
tester tests. The full Linux Python 3.15 workflow remains unverified.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
