Ruifeng Zheng created SPARK-61456:
-------------------------------------
Summary: Support configurable intermediate storage levels in
regression estimators
Key: SPARK-61456
URL: https://issues.apache.org/jira/browse/SPARK-61456
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
LinearRegression, AFTSurvivalRegression, FMRegressor, and IsotonicRegression
currently persist internally created training data at a hard-coded
MEMORY_AND_DISK storage level. Caching the input DataFrame does not configure
these intermediate datasets.
Expose the existing intermediateStorageLevel shared parameter on these four
regressors in Scala and PySpark. Apply it to their persistence calls while
retaining MEMORY_AND_DISK as the default. For LinearRegression, the setting
applies to the block-based iterative solver.
Include PySpark constructor and setParams keyword support, setters, model
parameter propagation, and checks in the existing estimator suites.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]