HyukjinKwon commented on PR #56644:
URL: https://github.com/apache/spark/pull/56644#issuecomment-4766440784
### Scope note: a separate, pre-existing local-cluster ResourceProfile
failure (NOT this fix)
This PR is intentionally scoped to the SQLContext import guard in
`test_sql_context.py`, which is a real, validated fix: the connect-only build
(`build_python_connect`) used to abort at the "Run tests (local)" collection
step with `ImportError: cannot import name 'SQLContext' from 'pyspark'` (exit
255) in the remote-only client install. With this change that step passes.
After the import is fixed, the connect-only build re-exposes a separate,
pre-existing failure that this PR does not touch:
- Test:
`pyspark.resource.tests.test_connect_resources.ResourceProfileTests.test_profile_before_sc_for_connect`,
in the "Run tests (local-cluster)" step.
- Symptom: a fast (~0.025s) server-side `java.lang.AssertionError`
("assertion failed", no message) surfaced to the client as `StatusCode.UNKNOWN:
"Application error processing RPC"`, triggered at the test's first explicit
`rp.id` call (test line 49), which fires a synchronous
`CreateResourceProfileCommand` RPC immediately after `getOrCreate()` on a
freshly started `local-cluster[2,4,1024]` Connect server.
Why this is pre-existing and out of scope for the import fix:
1. The test and its explicit early `rp.id` line are unchanged for months
(predate the current commit history window). It is not a new test and not
modified here.
2. It never ran on apache's scheduled connect-only build: that job always
died earlier at the same SQLContext import this PR fixes, so the local-cluster
step has been masked since the import regression landed. The import fix merely
re-exposes it.
3. The ResourceProfile-over-Connect feature itself works: in the same build,
`pyspark.sql.tests.connect.test_parity_resources.ResourceProfileParityTests` (5
tests) passes against the same local-cluster server, building a
`TaskResourceProfile` and running `mapInPandas`/`mapInArrow`. The classic
analogue `test_profile_before_sc_for_sql` also passes; it never calls `rp.id`
explicitly (it resolves the profile lazily via `mapInPandas`). The unique
trigger here is the synchronous cold-start `rp.id` probe.
4. The failure is non-deterministic across runs (passes on some attempts,
fails on others) with a ~25ms fast-fail, which is the signature of a cold-start
race rather than a deterministic logic bug. The driver-side
`addResourceProfile` -> `isSupported` -> `limitingResource` path for a
task-only profile in local-cluster contains no message-less `assert` that can
fire (the only message-less assert, `assert(master != null)` in
`ResourceProfileManager`, cannot fire since `master` is an `Option`), and
`ResourceProfile*.scala` is unchanged for months. So the bare assert originates
outside the resource package, on the server/scheduler init path raced by the
very-early RPC.
This is being investigated separately and should be filed as its own JIRA
(Connect / ResourceProfile cold-start over local-cluster). It should not block
this import-guard fix, which is independently correct and was confirmed working
("Run tests (local)" green) on the fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
