Repository: spark Updated Branches: refs/heads/master fe854f2e4 -> c5a64d760
[SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun ## What changes were proposed in this pull request? Fix a broadcasted variable leak occurring at each invocation of CostFun in L-BFGS. ## How was this patch tested? UTests + check that fixed fatal memory consumption on Criteo's use cases. This contribution is made on behalf of Criteo S.A. (http://labs.criteo.com/) under the terms of the Apache v2 License. Author: Anthony Truchet <a.truc...@criteo.com> Closes #16040 from AnthonyTruchet/SPARK-18612-lbfgs-cost-fun. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c5a64d76 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c5a64d76 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c5a64d76 Branch: refs/heads/master Commit: c5a64d760600ff430899e401751c41dc6b27cee6 Parents: fe854f2 Author: Anthony Truchet <a.truc...@criteo.com> Authored: Wed Nov 30 10:04:47 2016 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Wed Nov 30 10:04:47 2016 +0000 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/mllib/optimization/LBFGS.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c5a64d76/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala ---------------------------------------------------------------------- diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala b/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala index 900eec1..e0e41f7 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala @@ -252,6 +252,9 @@ object LBFGS extends Logging { (grad1, loss1 + loss2) }) + // broadcasted model is not needed anymore + bcW.destroy() + /** * regVal is sum of weight squares if it's L2 updater; * for other updater, the same logic is followed. --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
