Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/493#discussion_r12080780
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -708,6 +709,123 @@ object ALS {
trainImplicit(ratings, rank, iterations, 0.01, -1, 1.0)
}
+ @DeveloperApi
+ case class IterationCost(inboundBytes: Double, computation: Double,
outboundBytes: Double)
--- End diff --
From our previous discussion:
> The return format is not very easy to understand. For each user block, we
want to know three estimates:
>
> how much incoming data in a user iteration.
> how much computation (YtY and LS) in a user iteration.
> how much outgoing data in a product iteration.
> We can create a case class Cost(index, n, numRatings, dataOut, dataIn)
where n is number of users/products > in this block. Then the output type
becomes (Seq[Cost], Seq[Cost]) -> user block costs and product block > costs.
Instead of estimating bytes and flops directly, the cost may be more
interpretable with counts. For example, `numUsers/numProducts` determines the
number of LS subproblems to solve, `numRatings` gives a good estimate about the
storage cost and the cost of assembling `YtY`, `numFactorsToSend` and
`numFactorsToReceive` decide the size of shuffle writes. And those estimates
becomes independent of rank.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
