Ruifeng Zheng created SPARK-61361:
-------------------------------------

             Summary: Support configurable intermediate storage levels in tree 
estimators
                 Key: SPARK-61361
                 URL: https://issues.apache.org/jira/browse/SPARK-61361
             Project: Spark
          Issue Type: Improvement
          Components: ML
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


Decision tree, random forest, and gradient-boosted tree classifiers and 
regressors cache intermediate training datasets at hard-coded storage levels. 
Expose the existing intermediateStorageLevel shared parameter on all six 
spark.ml estimators in Scala and PySpark.

Pass the configured level through the shared RandomForest and 
GradientBoostedTrees implementations, including bagged data, cached node IDs, 
binned training and validation data, sampling counts, and prediction/error 
caches. Keep MEMORY_AND_DISK as the default for existing callers.

Include PySpark constructor and setParams keyword support, and extend the 
existing estimator tests.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to