This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 631c824544e1 [SPARK-58636][PYTHON] Remove unnecessary type:
ignore[union-attr] in streaming MLlib trainers
631c824544e1 is described below
commit 631c824544e1d16f9f5e850b5f310b1804259cdf
Author: Spenser Sun <[email protected]>
AuthorDate: Mon Aug 10 05:53:07 2026 -0700
[SPARK-58636][PYTHON] Remove unnecessary type: ignore[union-attr] in
streaming MLlib trainers
### What changes were proposed in this pull request?
Part of the ongoing effort to remove unnecessary `# type: ignore` comments
in PySpark by fixing the underlying type issue rather than suppressing it.
The three streaming MLlib trainers each define a `trainOn` method with a
nested `update(rdd)` closure passed to `dstream.foreachRDD`, operating on an
`Optional` `self._model` that the preceding `self._validate(dstream)` call
guarantees is non-`None`. mypy cannot connect `_validate` to the attribute, nor
narrow an instance attribute across the closure boundary, so it reports
`union-attr` on `self._model`. Two of the three trainers silenced this with `#
type: ignore[union-attr]`, while [...]
This standardizes the other two to that existing `assert` idiom, removing
the two `# type: ignore[union-attr]` comments:
- `StreamingKMeans.trainOn` in `python/pyspark/mllib/clustering.py`
- `StreamingLogisticRegressionWithSGD.trainOn` in
`python/pyspark/mllib/classification.py`
### Why are the changes needed?
Consistency across the three near-identical streaming trainer closures. The
`assert` documents the invariant that `_validate` enforces in a form the type
checker understands, replacing an opaque suppression comment, and matches the
pattern already present in `python/pyspark/mllib/regression.py`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing `python/pyspark/mllib/tests/test_streaming_algorithms.py`
(passing; the one pre-existing `SPARK-10086` skip is unrelated). Full-scope
`mypy --namespace-packages --config-file python/mypy.ini python/pyspark` passes
clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57843 from Spenserrrr/mllib-streaming-model.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 0e63f0160f428d8021561fa617f30b00c132664c)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/mllib/classification.py | 3 ++-
python/pyspark/mllib/clustering.py | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/mllib/classification.py
b/python/pyspark/mllib/classification.py
index c8b66947e07c..bc661e9c6bb6 100644
--- a/python/pyspark/mllib/classification.py
+++ b/python/pyspark/mllib/classification.py
@@ -950,12 +950,13 @@ class
StreamingLogisticRegressionWithSGD(StreamingLinearAlgorithm):
def update(rdd: RDD[LabeledPoint]) -> None:
# LogisticRegressionWithSGD.train raises an error for an empty RDD.
if not rdd.isEmpty():
+ assert self._model is not None
self._model = LogisticRegressionWithSGD.train(
rdd,
self.numIterations,
self.stepSize,
self.miniBatchFraction,
- self._model.weights, # type: ignore[union-attr]
+ self._model.weights,
regParam=self.regParam,
convergenceTol=self.convergenceTol,
)
diff --git a/python/pyspark/mllib/clustering.py
b/python/pyspark/mllib/clustering.py
index ca7335999a83..c74c51859648 100644
--- a/python/pyspark/mllib/clustering.py
+++ b/python/pyspark/mllib/clustering.py
@@ -1075,7 +1075,8 @@ class StreamingKMeans:
self._validate(dstream)
def update(rdd: RDD["VectorLike"]) -> None:
- self._model.update(rdd, self._decayFactor, self._timeUnit) #
type: ignore[union-attr]
+ assert self._model is not None
+ self._model.update(rdd, self._decayFactor, self._timeUnit)
dstream.foreachRDD(update)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
