Ruifeng Zheng created SPARK-61844:
-------------------------------------

             Summary: Support configurable intermediate storage levels in LDA
                 Key: SPARK-61844
                 URL: https://issues.apache.org/jira/browse/SPARK-61844
             Project: Spark
          Issue Type: Improvement
          Components: ML
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


LDA currently persists intermediate training instances with a hard-coded 
MEMORY_AND_DISK storage level when the online optimizer is used and the input 
dataset is not already cached.

Expose the shared intermediateStorageLevel parameter in the Scala and PySpark 
LDA APIs, including the Python constructor, setParams, and setter. Keep 
MEMORY_AND_DISK as the default and preserve the existing cached-input guard. 
This parameter applies to the online optimizer; EM's graph caching is outside 
this change.

Extend the existing Scala and Python test suites. New API annotations use 
4.5.0, following the branch-4.4 cut.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to