GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3259
[SPARK-4393] Fix memory leak in ConnectionManager ACK timeout TimerTasks;
use HashedWheelTimer
This patch is intended to fix a subtle memory leak in ConnectionManager's
ACK timeout TimerTasks: in the old code, each TimerTask held a reference to the
message being sent and a cancelled TimerTask won't necessarily be
garbage-collected until it's scheduled to run, so this caused huge buildups of
messages that weren't garbage collected until their timeouts expired, leading
to OOMs.
This patch addresses this problem by capturing only the message ID in the
TimerTask instead of the whole message. I've also modified this code to use
Netty's HashedWheelTimer, whose performance characteristics should be better
for this use-case.
Thanks to @cristianopris for narrowing down this issue!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
connection-manager-timeout-bugfix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3259.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3259
----
commit f847dd4a4a8e7f92de879de9d5c9eb31743f8a26
Author: Josh Rosen <[email protected]>
Date: 2014-11-14T00:13:15Z
Don't capture entire message in ACK timeout task.
The old code caused memory leaks.
commit 3200c33363d8daed187ecd10f7b5fc370d44f349
Author: Josh Rosen <[email protected]>
Date: 2014-11-14T00:45:41Z
Use Netty HashedWheelTimer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
