Apache Spark commented on SPARK-18471:

User 'AnthonyTruchet' has created a pull request for this issue:

> In treeAggregate, generate (big) zeros instead of sending them.
> ---------------------------------------------------------------
>                 Key: SPARK-18471
>                 URL: https://issues.apache.org/jira/browse/SPARK-18471
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib, Spark Core
>            Reporter: Anthony Truchet
>            Priority: Minor
> When using optimization routine like LBFGS, treeAggregate curently sends the 
> zero vector as part of the closure. This zero can be huge (e.g. ML vectors 
> with millions of zeros) but can be easily generated.
> Several option are possible (upcoming patches to come soon for some of them).
> On is to provide a treeAggregateWithZeroGenerator method (either in core on 
> in MLlib) which wrap treeAggregate in an option and generate the zero if None.
> Another one is to rewrite treeAggregate to wrap an underlying implementation 
> which use a zero generator directly.
> There might be other better alternative we have not spotted...

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to