[
https://issues.apache.org/jira/browse/SPARK-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134831#comment-14134831
]
Xiangrui Meng commented on SPARK-3366:
--------------------------------------
It is more about communication than computation. For random forest, even at
shallow levels, the size of each stats array will be close to the limit, e.g,
256MB or more. The driver needs to aggregate them. Even with tree aggregation,
the driver is still a bottleneck on communication. That's why most of the time
is spent on aggregation. Shuffling the stats arrays by node index and then
computing the best split for each node distributively may help reduce the
driver load. In that case, the driver only need to collect the final result.
> Compute best splits distributively in decision tree
> ---------------------------------------------------
>
> Key: SPARK-3366
> URL: https://issues.apache.org/jira/browse/SPARK-3366
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: Xiangrui Meng
>
> The current implementation computes all best splits locally on the driver,
> which makes the driver a bottleneck for both communication and computation.
> It would be nice if we can compute the best splits distributively.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
