Haotian Sun created SPARK-58636:
-----------------------------------
Summary: Remove unnecessary type: ignore[union-attr] in streaming
MLlib trainers
Key: SPARK-58636
URL: https://issues.apache.org/jira/browse/SPARK-58636
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
Part of the ongoing effort to remove unnecessary `# type: ignore` comments in
PySpark by fixing the underlying type issue rather than suppressing it.
The three streaming MLlib trainers each define a `trainOn` method with a nested
`update(rdd)` closure passed to `dstream.foreachRDD`. Each operates on an
`Optional` `self._model` that the preceding `self._validate(dstream)` call
guarantees is non-`None`. mypy cannot connect `_validate` to the attribute, nor
narrow an instance attribute across the closure boundary, so it reports
`union-attr` on `self._model`. Two of the three trainers silenced this with `#
type: ignore[union-attr]`, while `StreamingLinearRegressionWithSGD` already
used `assert self._model is not None` in the same closure.
This standardizes the other two to the existing `assert` idiom, removing the
two `# type: ignore[union-attr]` comments:
- `StreamingKMeans.trainOn` in `pyspark/mllib/clustering.py`
- `StreamingLogisticRegressionWithSGD.trainOn` in
`pyspark/mllib/classification.py`
The `assert` documents the invariant `_validate` enforces in a form the type
checker understands, and matches the pattern already present in
`pyspark/mllib/regression.py`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
