[
https://issues.apache.org/jira/browse/SPARK-60173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-60173:
----------------------------------
Affects Version/s: 4.4.0
> Remove unnecessary type: ignore[union-attr] in pyspark.ml
> ---------------------------------------------------------
>
> Key: SPARK-60173
> URL: https://issues.apache.org/jira/browse/SPARK-60173
> Project: Spark
> Issue Type: Improvement
> Components: ML, PySpark
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Vito Lin
> Assignee: Vito Lin
> Priority: Major
> Labels: pull-request-available
>
> pyspark.ml module has five "type: ignore[union-attr]" comments that can be
> replaced with proper type narrowing or corrected annotations:
> 1. ml/torch/torch_run_process_wrapper.py: task.stdin and task.stdout are
> created with subprocess.PIPE so they are never None; narrow with an assert.
> 2. ml/connect/io_utils.py: narrow sparkContext._gateway with an assert.
> 3. ml/wrapper.py: check SparkContext._active_spark_context._gateway is not
> None before calling detach, instead of relying on the surrounding except.
> 4. ml/connect/tuning.py: param_map comes from Sequence[ParamMap], so annotate
> it as ParamMap instead of Any.
> 5. ml/connect/base.py: Estimator.fit only accepts a single ParamMap and never
> returns a list, but its return type is declared as Union[M, List[M]]. Correct
> it to M. This is what caused the ignore in tuning.py.
> No behavior change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
