Ruifeng Zheng created SPARK-59535:
-------------------------------------
Summary: Run PCA covariance decomposition on an executor
Key: SPARK-59535
URL: https://issues.apache.org/jira/browse/SPARK-59535
Project: Spark
Issue Type: Improvement
Components: MLlib
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
For PCA with at most 65535 features, RowMatrix currently aggregates a packed
covariance matrix
across executors, returns that large aggregate to the driver, expands it into a
dense matrix, and
runs the local Breeze SVD on the driver. The driver must hold the packed
matrix, dense covariance,
and SVD workspace even though only the smaller principal-component result needs
to be returned.
Add an internal tree-aggregation variant that preserves the final aggregate as
a single-partition
RDD. Use it in RowMatrix so covariance expansion and PCA decomposition run in
the final executor
task, and return only the public method result to the driver. Public APIs and
numerical behavior
remain unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
