[
https://issues.apache.org/jira/browse/SPARK-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng updated SPARK-2361:
---------------------------------
Assignee: Xiangrui Meng
> Decide whether to broadcast or serialize the weights directly in MLlib
> algorithms
> ---------------------------------------------------------------------------------
>
> Key: SPARK-2361
> URL: https://issues.apache.org/jira/browse/SPARK-2361
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: Xiangrui Meng
> Assignee: Xiangrui Meng
>
> In the current implementation, MLlib serializes weights directly into
> closure. This is okay for small feature dimension, but not efficient for
> feature dimensions beyond 1M. Especially the default akka.frameSize is 10m.
> We should use broadcast when the size of the serialized task is going to be
> large.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
