Ruifeng Zheng created SPARK-61954:
-------------------------------------
Summary: Support configurable intermediate storage levels in
BisectingKMeans
Key: SPARK-61954
URL: https://issues.apache.org/jira/browse/SPARK-61954
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
BisectingKMeans currently hard-codes MEMORY_AND_DISK for its intermediate
vectors, norms, and cluster-assignment RDDs.
Expose the shared intermediateStorageLevel parameter in the Scala and PySpark
ML APIs, including the Python constructor, setParams, and setter. Pass the
configured level to the shared training implementation and apply it to all
three intermediate cache sites. Preserve the default MEMORY_AND_DISK storage
level and the existing choice between caching vectors for uncached input and
norms for cached input.
Extend the existing Scala and Python test suites. New API annotations use 4.5.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]