Spenserrrr opened a new pull request, #57843: URL: https://github.com/apache/spark/pull/57843
### What changes were proposed in this pull request? Part of the ongoing effort to remove unnecessary `# type: ignore` comments in PySpark by fixing the underlying type issue rather than suppressing it. The three streaming MLlib trainers each define a `trainOn` method with a nested `update(rdd)` closure passed to `dstream.foreachRDD`, operating on an `Optional` `self._model` that the preceding `self._validate(dstream)` call guarantees is non-`None`. mypy cannot connect `_validate` to the attribute, nor narrow an instance attribute across the closure boundary, so it reports `union-attr` on `self._model`. Two of the three trainers silenced this with `# type: ignore[union-attr]`, while `StreamingLinearRegressionWithSGD` already used `assert self._model is not None` in the same closure (see `python/pyspark/mllib/regression.py`). This standardizes the other two to that existing `assert` idiom, removing the two `# type: ignore[union-attr]` comments: - `StreamingKMeans.trainOn` in `python/pyspark/mllib/clustering.py` - `StreamingLogisticRegressionWithSGD.trainOn` in `python/pyspark/mllib/classification.py` ### Why are the changes needed? Consistency across the three near-identical streaming trainer closures. The `assert` documents the invariant that `_validate` enforces in a form the type checker understands, replacing an opaque suppression comment, and matches the pattern already present in `python/pyspark/mllib/regression.py`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing `python/pyspark/mllib/tests/test_streaming_algorithms.py` (passing; the one pre-existing `SPARK-10086` skip is unrelated). Full-scope `mypy --namespace-packages --config-file python/mypy.ini python/pyspark` passes clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
