[
https://issues.apache.org/jira/browse/SPARK-60538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60538:
-----------------------------------
Labels: pull-request-available (was: )
> Guard optional dependencies in worker-environment and streaming tester tests
> ----------------------------------------------------------------------------
>
> Key: SPARK-60538
> URL: https://issues.apache.org/jira/browse/SPARK-60538
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Xinyao Zhang
> Priority: Minor
> Labels: pull-request-available
>
> Parent: SPARK-60531.
> The Python 3.15 CI image omits pandas and PyArrow. PythonWorkerEnvTests
> nevertheless executes dependent cases and reports 10 errors. test_tws_tester
> imports pandas before its class-level skip can apply. Two Arrow-specific
> worker tests can also silently fall back to regular Python execution when
> dependencies are absent.
> Evidence: https://github.com/apache/spark/actions/runs/40032262075.
> Proposed fix: add dependency guards to the affected methods in
> PythonWorkerEnvMixin, retain regular Python worker coverage, and
> conditionally import pandas and pandas-dependent helpers in
> test_tws_tester.py. Implementation:
> https://github.com/apache/spark/pull/59773.
> Validation: on Python 3.15.0rc1 without pandas/PyArrow, 12 dependent worker
> cases and 60 streaming tester cases skip cleanly. Dependency selection was
> checked under all four pandas/PyArrow combinations on Python 3.12. Fork CI
> passed 32 Classic worker tests, 32 Connect parity tests, and 60 streaming
> tester tests. The full Linux Python 3.15 workflow remains unverified.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
