This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 059d5219d0bc [SPARK-60171][PYTHON] Declare validator parameter
dependency in Connect persistence
059d5219d0bc is described below
commit 059d5219d0bc3b912800071bd3972bf5f0286cac
Author: Tim Hsiung <[email protected]>
AuthorDate: Thu Nov 5 11:00:49 2026 -0800
[SPARK-60171][PYTHON] Declare validator parameter dependency in Connect
persistence
### What changes were proposed in this pull request?
Make `_CrossValidatorReadWrite` inherit `_ValidatorParams` and remove four
redundant `type: ignore[attr-defined]` comments.
### Why are the changes needed?
The persistence mixin unconditionally uses validator parameters and getters
supplied by its sibling base, `_CrossValidatorParams`, when combined in
`CrossValidator` and `CrossValidatorModel`. This implicit sibling dependency
works at runtime but is invisible when mypy checks the mixin against its own
bases. Inheriting `_ValidatorParams` makes that requirement explicit.
This is an incremental cleanup: both concrete classes already inherit
`_ValidatorParams`, so it reuses their existing parameter definitions. A future
refactor could simplify the overlapping inheritance paths by consolidating
shared validator parameters and persistence in a common base. Typing the
estimator/evaluator persistence interfaces could also remove the remaining
suppressions. Both are outside this PR's scope.
Related code in `python/pyspark/ml/connect/tuning.py`:
- [Parameter definitions and getters; `_CrossValidatorParams`
inheritance](https://github.com/bearomorphism/spark/blob/a0c6b88ceae9dc472aef5e4c61fe18f63ac6cf9e/python/pyspark/ml/connect/tuning.py#L52-L91).
- [Persistence mixin and its parameter
accesses](https://github.com/bearomorphism/spark/blob/a0c6b88ceae9dc472aef5e4c61fe18f63ac6cf9e/python/pyspark/ml/connect/tuning.py#L196-L253).
- Concrete classes combining the sibling bases:
[CrossValidator](https://github.com/bearomorphism/spark/blob/a0c6b88ceae9dc472aef5e4c61fe18f63ac6cf9e/python/pyspark/ml/connect/tuning.py#L264-L269)
and
[CrossValidatorModel](https://github.com/bearomorphism/spark/blob/a0c6b88ceae9dc472aef5e4c61fe18f63ac6cf9e/python/pyspark/ml/connect/tuning.py#L519).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Reproduced the missing-attribute error with mypy before the fix; focused
mypy and Ruff checks pass afterward.
- Checked the MRO change and ran local save/load and copy smoke checks for
both `CrossValidator` and `CrossValidatorModel`.
- No new tests added; Spark-backed suites were not run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Pi 1.1.0
Closes #59369 from
bearomorphism/bearomorphism/fix-estimator-param-maps-typing.
Authored-by: Tim Hsiung <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit f45fbe9e5c6bc6c96c9a15fab022c283eba672a1)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/ml/connect/tuning.py | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/ml/connect/tuning.py
b/python/pyspark/ml/connect/tuning.py
index d6b1b6117508..d43c486fa7e2 100644
--- a/python/pyspark/ml/connect/tuning.py
+++ b/python/pyspark/ml/connect/tuning.py
@@ -191,7 +191,7 @@ def _parallelFitTasks(
return [get_single_task(index, param_map) for index, param_map in
enumerate(epm)]
-class _CrossValidatorReadWrite(MetaAlgorithmReadWrite):
+class _CrossValidatorReadWrite(_ValidatorParams, MetaAlgorithmReadWrite):
def _get_skip_saving_params(self) -> List[str]:
"""
Returns params to be skipped when saving metadata.
@@ -211,7 +211,7 @@ class _CrossValidatorReadWrite(MetaAlgorithmReadWrite):
{"parent": param.parent, "name": param.name, "value": value}
for param, value in param_map.items()
]
- for param_map in self.getEstimatorParamMaps() # type:
ignore[attr-defined]
+ for param_map in self.getEstimatorParamMaps()
]
if isinstance(self, CrossValidatorModel):
@@ -227,12 +227,12 @@ class _CrossValidatorReadWrite(MetaAlgorithmReadWrite):
estimator = ParamsReadWrite._load_instance_from_metadata(
node_metadata["estimator"], root_path
)
- self.set(self.estimator, estimator) # type: ignore[attr-defined]
+ self.set(self.estimator, estimator)
evaluator = ParamsReadWrite._load_instance_from_metadata(
node_metadata["evaluator"], root_path
)
- self.set(self.evaluator, evaluator) # type: ignore[attr-defined]
+ self.set(self.evaluator, evaluator)
json_epm = node_metadata["estimator_param_maps"]
@@ -248,7 +248,7 @@ class _CrossValidatorReadWrite(MetaAlgorithmReadWrite):
param_map[param] = value
epm.append(param_map)
- self.set(self.estimatorParamMaps, epm) # type: ignore[attr-defined]
+ self.set(self.estimatorParamMaps, epm)
if isinstance(self, CrossValidatorModel):
self.avgMetrics = node_metadata["avg_metrics"]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]