HyukjinKwon opened a new pull request, #56644:
URL: https://github.com/apache/spark/pull/56644
### What changes were proposed in this pull request?
The Spark Connect parity test
`pyspark.sql.tests.connect.test_parity_sql_context` reuses
`SQLContextTestsMixin` from `pyspark.sql.tests.test_sql_context`. That
module imports the
classic SQLContext at module load time:
```python
from pyspark import SQLContext
```
In a remote-only (Spark Connect client-only, i.e. `pyspark-client`) install,
`pyspark/__init__.py`
only exposes `SQLContext` when `not is_remote_only()`. So the import raises
`ImportError` and the
whole module fails to import, aborting the "Python-only, Connect-only"
build's `Run tests (local)`
step with exit code 255 at collection time.
This PR guards that import so the module stays importable in remote-only
mode, falling back to the
Connect `SQLContext` (which the parity test already uses):
```python
try:
from pyspark import SQLContext
except ImportError:
from pyspark.sql.connect.context import SQLContext
```
The same `try/except ImportError` import pattern is already used elsewhere
in the test tree
(e.g. `pyspark/sql/tests/test_session.py`). The change is test-only.
### Why are the changes needed?
The scheduled "Build / Python-only, Connect-only (Python 3.11)" build has
been failing since the
parity test started importing the classic mixin in remote-only mode (it was
last green on
2026-06-11). The `Run tests (local)` step aborts at import:
```
File ".../pyspark/sql/tests/test_sql_context.py", line 21, in <module>
from pyspark import SQLContext
ImportError: cannot import name 'SQLContext' from 'pyspark'
```
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Validated on the fork by running the actual scheduled workflow
(`build_python_connect.yml`) on the branch:
- Fork run: https://github.com/HyukjinKwon/spark/actions/runs/27923450056
With this fix, the `Run tests (local)` step (which previously aborted at
import with exit 255) now
passes and test collection proceeds.
Note: that same run still fails at a **separate, pre-existing** regression
in the later
`Run tests (local-cluster)` step --
`pyspark.resource.tests.test_connect_resources.
ResourceProfileTests.test_profile_before_sc_for_connect` errors with a
server-side
`java.lang.AssertionError` when creating a `ResourceProfile` over Connect in
local-cluster mode.
That failure was masked until now (the job stopped at the failed `local`
step) and is unrelated to
this import fix; it is being tracked/fixed separately.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
---
NOTE FOR COMMITTER: the last commit `[DO NOT MERGE] CI: trigger
build_python_connect on fork for
validation` is validation-only (it lets the scheduled workflow run on the
fork) and must be dropped
before merging. JIRA id TBD -- please assign a SPARK ticket / adjust the
title component as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
