viirya commented on issue #25576: [SPARK-28866][ML] Persist item factors RDD when checkpointing in ALS URL: https://github.com/apache/spark/pull/25576#issuecomment-525512692 Because for non-implicit case, we don't call computeYtY to materialize the RDD of factor blocks. So there is no any RDD materialization in this loop. To perform checkpoint actually, I think we need to run a job (.count()) to trigger it. The implicit case has persist and unpersist on userFactors and itemFactors RDDs. For non-implicit case, in this loop, it basically builds a RDD chain like userFactors_(t - 1) -> itemFactors_(t) ->userFactors_(t) -> itemFactors_(t + 1) -> ..., the RDDs involved are referred only once in the chain. I think that is why we don't see persist/unpersist there. The only exception is when we do checkpoint itemFactors, we need persist/unpersist it to avoid recomputation, this is also this PR wants to do.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
