Ruifeng Zheng created SPARK-61844:
-------------------------------------
Summary: Support configurable intermediate storage levels in LDA
Key: SPARK-61844
URL: https://issues.apache.org/jira/browse/SPARK-61844
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
LDA currently persists intermediate training instances with a hard-coded
MEMORY_AND_DISK storage level when the online optimizer is used and the input
dataset is not already cached.
Expose the shared intermediateStorageLevel parameter in the Scala and PySpark
LDA APIs, including the Python constructor, setParams, and setter. Keep
MEMORY_AND_DISK as the default and preserve the existing cached-input guard.
This parameter applies to the online optimizer; EM's graph caching is outside
this change.
Extend the existing Scala and Python test suites. New API annotations use
4.5.0, following the branch-4.4 cut.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]