[
https://issues.apache.org/jira/browse/SPARK-58636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haotian Sun updated SPARK-58636:
--------------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> Remove unnecessary type: ignore[union-attr] in streaming MLlib trainers
> -----------------------------------------------------------------------
>
> Key: SPARK-58636
> URL: https://issues.apache.org/jira/browse/SPARK-58636
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Haotian Sun
> Priority: Major
>
> Part of the ongoing effort to remove unnecessary `# type: ignore` comments in
> PySpark by fixing the underlying type issue rather than suppressing it.
> The three streaming MLlib trainers each define a `trainOn` method with a
> nested `update(rdd)` closure passed to `dstream.foreachRDD`. Each operates on
> an `Optional` `self._model` that the preceding `self._validate(dstream)` call
> guarantees is non-`None`. mypy cannot connect `_validate` to the attribute,
> nor narrow an instance attribute across the closure boundary, so it reports
> `union-attr` on `self._model`. Two of the three trainers silenced this with
> `# type: ignore[union-attr]`, while `StreamingLinearRegressionWithSGD`
> already used `assert self._model is not None` in the same closure.
> This standardizes the other two to the existing `assert` idiom, removing the
> two `# type: ignore[union-attr]` comments:
> - `StreamingKMeans.trainOn` in `pyspark/mllib/clustering.py`
> - `StreamingLogisticRegressionWithSGD.trainOn` in
> `pyspark/mllib/classification.py`
> The `assert` documents the invariant `_validate` enforces in a form the type
> checker understands, and matches the pattern already present in
> `pyspark/mllib/regression.py`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
