[
https://issues.apache.org/jira/browse/SPARK-60151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60151:
-----------------------------------
Labels: pull-request-available (was: )
> CrossValidatorModel.load() returns the last fold's sub-models for every fold
> ----------------------------------------------------------------------------
>
> Key: SPARK-60151
> URL: https://issues.apache.org/jira/browse/SPARK-60151
> Project: Spark
> Issue Type: Bug
> Components: ML, PySpark
> Affects Versions: 3.5.8, 4.2.0, 5.0.0, 4.1.3, 4.0.4
> Reporter: HUANG GUAN LUN
> Priority: Minor
> Labels: pull-request-available
>
> When a CrossValidatorModel is saved with sub-models and loaded back in
> PySpark, every fold's entry of \{{subModels}} refers to the same list, which
> holds the last fold's models.
> The cause is in \{{CrossValidatorModelReader.load}}
> (python/pyspark/ml/tuning.py):
> {code:python}
> subModels = [[None] * len(estimatorParamMaps)] * numFolds
> {code}
> {\{[x] * n}} repeats a reference to the same inner list, so each fold's
> assignment overwrites the previous one. \{{CrossValidator._fit}},
> \{{CrossValidatorModel.copy}} and \{{_from_java}} all build an independent
> list per fold, as does the Scala reader (\{{Array.fill(numFolds)(...)}}).
> The files on disk are correct. Only the PySpark reader is affected.
> \{{bestModel}} and \{{avgMetrics}} are not affected. However, saving a loaded
> model again writes the last fold's models into every fold directory, so the
> original sub-models are lost.
> *Reproduce*
> {code:python}
> from pyspark.ml.classification import LogisticRegression
> from pyspark.ml.evaluation import BinaryClassificationEvaluator
> from pyspark.ml.linalg import Vectors
> from pyspark.ml.tuning import CrossValidator, CrossValidatorModel,
> ParamGridBuilder
> df = spark.createDataFrame(
> [(Vectors.dense([float(i)]), float(i % 2)) for i in range(30)],
> ["features", "label"])
> lr = LogisticRegression(maxIter=2)
> grid = ParamGridBuilder().addGrid(lr.regParam, [0.0, 0.1]).build()
> cv = CrossValidator(estimator=lr, estimatorParamMaps=grid,
> evaluator=BinaryClassificationEvaluator(),
> numFolds=3, collectSubModels=True, seed=1)
> model = cv.fit(df)
> model.write().save("/tmp/cv")
> loaded = CrossValidatorModel.load("/tmp/cv")
> print([m.coefficients for m in model.subModels[0]]) # fold 0
> print([m.coefficients for m in loaded.subModels[0]]) # same as fold 2
> print(loaded.subModels[0] is loaded.subModels[1]) # True
> {code}
> First coefficient of each sub-model:
> || ||fold0||fold1||fold2||
> |after fit|[0.0705, 0.0439]|[0.0067, 0.0047]|[-0.0155, -0.0108]|
> |on disk after save|[0.0705, 0.0439]|[0.0067, 0.0047]|[-0.0155, -0.0108]|
> |after load|[-0.0155, -0.0108]|[-0.0155, -0.0108]|[-0.0155, -0.0108]|
> The existing test in python/pyspark/ml/tests/test_tuning.py does not catch
> this. It compares \{{model.subModels[i][j].uid}} with itself, and after
> loading it only checks the list lengths. All sub-models share the estimator's
> uid, so a uid comparison would not detect it either.
> The line was introduced by SPARK-33520.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
