Repository: spark Updated Branches: refs/heads/branch-2.1 7043c6b69 -> 05ba5eed7
[SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun ## What changes were proposed in this pull request? Fix a broadcasted variable leak occurring at each invocation of CostFun in L-BFGS. ## How was this patch tested? UTests + check that fixed fatal memory consumption on Criteo's use cases. This contribution is made on behalf of Criteo S.A. (http://labs.criteo.com/) under the terms of the Apache v2 License. Author: Anthony Truchet <[email protected]> Closes #16040 from AnthonyTruchet/SPARK-18612-lbfgs-cost-fun. (cherry picked from commit c5a64d760600ff430899e401751c41dc6b27cee6) Signed-off-by: Sean Owen <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/05ba5eed Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/05ba5eed Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/05ba5eed Branch: refs/heads/branch-2.1 Commit: 05ba5eed71309e104feb1951aa8197e4336cdb2a Parents: 7043c6b Author: Anthony Truchet <[email protected]> Authored: Wed Nov 30 10:04:47 2016 +0000 Committer: Sean Owen <[email protected]> Committed: Wed Nov 30 10:04:57 2016 +0000 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/mllib/optimization/LBFGS.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/05ba5eed/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala ---------------------------------------------------------------------- diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala b/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala index 900eec1..e0e41f7 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala @@ -252,6 +252,9 @@ object LBFGS extends Logging { (grad1, loss1 + loss2) }) + // broadcasted model is not needed anymore + bcW.destroy() + /** * regVal is sum of weight squares if it's L2 updater; * for other updater, the same logic is followed. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
