Ruifeng Zheng created SPARK-61633:
-------------------------------------
Summary: Support the remaining native clustering algorithms: LDA
and PowerIterationClustering
Key: SPARK-61633
URL: https://issues.apache.org/jira/browse/SPARK-61633
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Ruifeng Zheng
Complete native clustering algorithm execution by adding the remaining PySpark
4.3.0 algorithms: LDA and PowerIterationClustering.
LDA should support both online and EM optimizers, all 14 constructor parameters
and setters, cached fitted models and DataFrame transforms.
PowerIterationClustering should expose its six parameters and assignClusters,
using Spark Connect's powerIterationClusteringAssignClusters helper instead of
an estimator fit API. Implement both the Rust API and native Python bindings,
preserving the Spark-DataFrame-only input contract.
Validate parameter serialization and helper requests against official PySpark
fixtures, and run Rust and native Python live tests for inference, graph
assignments, parameter validation and model lifetime. This completes algorithm
execution coverage; richer model inspection and persistence remain separate
work.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]