[
https://issues.apache.org/jira/browse/SPARK-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377200#comment-14377200
]
Xiangrui Meng commented on SPARK-3735:
--------------------------------------
The proposal is actually something different. For example, in a single block if
we have 1000 items that are rated by all users. We need to send those 1000 item
factors to all user in-blocks. The shuffle size is 1000*k*numUserBlocks.
However, if we only send the partial AtA and Atb generated by those items, the
shuffle size is k*(k+1)*numUserBlocks, which is much smaller if k << 1000. This
is orthogonal to explicit/implicit feedback models.
> Sending the factor directly or AtA based on the cost in ALS
> -----------------------------------------------------------
>
> Key: SPARK-3735
> URL: https://issues.apache.org/jira/browse/SPARK-3735
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: Xiangrui Meng
> Assignee: Xiangrui Meng
>
> It is common to have some super popular products in the dataset. In this
> case, sending many user factors to the target product block could be more
> expensive than sending the normal equation `\sum_i u_i u_i^T` and `\sum_i u_i
> r_ij` to the product block. The cost of sending a single factor is `k`, while
> the cost of sending a normal equation is much more expensive, `k * (k + 3) /
> 2`. However, if we use normal equation for all products associated with a
> user, we don't need to send this user factor.
> Determining the optimal assignment is hard. But we could use a simple
> heuristic. Inside any rating block,
> 1) order the product ids by the number of user ids associated with them in
> desc order
> 2) starting from the most popular product, mark popular products as "use
> normal eq" and calculate the cost
> Remember the best assignment that comes with the lowest cost and use it for
> computation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
