WeichenXu123 commented on a change in pull request #30471:
URL: https://github.com/apache/spark/pull/30471#discussion_r530060435
##########
File path: python/pyspark/ml/tuning.py
##########
@@ -207,6 +210,205 @@ def _to_java_impl(self):
return java_estimator, java_epms, java_evaluator
+class _ValidatorSharedReadWrite:
+
+ @staticmethod
+ def saveImpl(path, instance, sc, extraMetadata=None):
+ from pyspark.ml.classification import OneVsRest
+ numParamsNotJson = 0
+ jsonEstimatorParamMaps = []
+ for paramMap in instance.getEstimatorParamMaps():
+ jsonParamMap = []
+ for p, v in paramMap.items():
+ jsonParam = {'parent': p.parent, 'name': p.name}
+ if (isinstance(v, Estimator) and not (
+ isinstance(v, _ValidatorParams) or
+ isinstance(v, OneVsRest))
+ ) or isinstance(v, Transformer) or \
Review comment:
> The Validators class will directly take Estimator and Evaluator, and
the Transformer will be part of the pipeline Estimator. Should the Transformer
params be part of the pipeline params?
A pyspark param value can be estimator/transformer/evaluator. They're all
legal.
Although currently pyspark does not have the case "transformer" to be a
param value,
but, allow it here is to provide extensibility.
```
if (isinstance(v, Estimator) and not (
isinstance(v, _ValidatorParams) or
isinstance(v, OneVsRest))
) or isinstance(v, Transformer) or \
isinstance(Evaluator):
```
The logic try to keep equivalent with jvm side logic
`isInstanceOf[DefaultParamsWritable] && !isInstanceOf[MLWritable]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
