This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 5b3ee7503d5e [SPARK-59985][4.X][ML][PYTHON][FOLLOWUP] Drop the
log_streaming_auth_secret assertion that branch-4.x does not have
5b3ee7503d5e is described below
commit 5b3ee7503d5e9e3b53f0526c033f9fdd1f723c14
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Wed Oct 21 19:59:51 2026 -0700
[SPARK-59985][4.X][ML][PYTHON][FOLLOWUP] Drop the log_streaming_auth_secret
assertion that branch-4.x does not have
### What changes were proposed in this pull request?
This is a branch-4.x follow-up of SPARK-59985 (#59242). It removes the
`self.assertIsNone(dist.log_streaming_auth_secret)` assertion from
`test_log_streaming_server_start_failure_warns` in
`python/pyspark/ml/torch/tests/test_distributor.py`.
`TorchDistributor.log_streaming_auth_secret` exists only on master. It was
added by 77f1e05768f ("[PYTHON][MINOR] Improve TorchDistributor log server
provisioning"), which is not on branch-4.x. When #59242 was cherry-picked onto
branch-4.x as b628a945a24, the test was taken as written for master, including
this assertion. The manual backports for 4.3/4.2/4.1/4.0 (#59381, #59382,
#59383, #59384) already leave it out. This PR makes branch-4.x match them; the
rest of the test is unchanged.
### Why are the changes needed?
On branch-4.x the test fails with:
```
AttributeError: 'TorchDistributor' object has no attribute
'log_streaming_auth_secret'
```
This breaks the PySpark CI for PRs against branch-4.x, for example #59676,
which does not touch `pyspark.ml.torch`.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Ran
`TorchDistributorBaselineUnitTests.test_log_streaming_server_start_failure_warns`
locally. Before this change it fails with the `AttributeError` above; after
it, it passes. Also checked that branch-4.x has no other reference to
`log_streaming_auth_secret` under `python/`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59691 from viirya/SPARK-59985-4.x-followup.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
python/pyspark/ml/torch/tests/test_distributor.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/python/pyspark/ml/torch/tests/test_distributor.py
b/python/pyspark/ml/torch/tests/test_distributor.py
index af994c81af40..1b9af7a7a3f1 100644
--- a/python/pyspark/ml/torch/tests/test_distributor.py
+++ b/python/pyspark/ml/torch/tests/test_distributor.py
@@ -294,7 +294,6 @@ class TorchDistributorBaselineUnitTestsMixin:
dist._run_distributed_training(MagicMock(), MagicMock(), None,
None)
self.assertEqual(dist.log_streaming_server_port, -1)
- self.assertIsNone(dist.log_streaming_auth_secret)
self.assertEqual(len(logs.records), 1)
message = logs.records[0].getMessage()
self.assertIn("Start torch distributor log streaming server failed",
message)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]