Vito Lin created SPARK-60173:
--------------------------------
Summary: Remove unnecessary type: ignore[union-attr] in pyspark.ml
Key: SPARK-60173
URL: https://issues.apache.org/jira/browse/SPARK-60173
Project: Spark
Issue Type: Improvement
Components: ML, PySpark
Affects Versions: 5.0.0
Reporter: Vito Lin
pyspark.ml module has five "type: ignore[union-attr]" comments that can be
replaced with proper type narrowing or corrected annotations:
1. ml/torch/torch_run_process_wrapper.py: task.stdin and task.stdout are
created with subprocess.PIPE so they are never None; narrow with an assert.
2. ml/connect/io_utils.py: narrow sparkContext._gateway with an assert.
3. ml/wrapper.py: check SparkContext._active_spark_context._gateway is not None
before calling detach, instead of relying on the surrounding except.
4. ml/connect/tuning.py: param_map comes from Sequence[ParamMap], so annotate
it as ParamMap instead of Any.
5. ml/connect/base.py: Estimator.fit only accepts a single ParamMap and never
returns a list, but its return type is declared as Union[M, List[M]]. Correct
it to M. This is what caused the ignore in tuning.py.
No behavior change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
