jenkins-bot has submitted this change and it was merged.
Change subject: Add logging to track down OOM
......................................................................
Add logging to track down OOM
The WMF job runners have been OOM'ing fairly recently lately.
Watching the memory for the jobrunners we can see they don't grow
their memory over time like a leak, instead they sit at 80M for
some time until quickly growing to >300M and dieing. The logs
all point to line 666, which issues commands to redis. I'm not
sure whats going on yet, but I think if we log the commands so we
know what it did just before OOM we can figure that out.
Change-Id: Ie927b49b0b21a4478e7ea8c207d1e65b9c63c729
---
M redisJobRunnerService
1 file changed, 3 insertions(+), 0 deletions(-)
Approvals:
Aaron Schulz: Looks good to me, approved
jenkins-bot: Verified
diff --git a/redisJobRunnerService b/redisJobRunnerService
index 894c32c..69ec126 100755
--- a/redisJobRunnerService
+++ b/redisJobRunnerService
@@ -662,6 +662,9 @@
*/
protected function redisCmd( Redis $conn, $cmd, array $args = array() )
{
$conn->clearLastError();
+ // we had some job runners oom'ing on this call, log what we are
+ // doing so there is relevant information next to the oom
+ $this->debug( "Redis cmd: $cmd " . json_encode( $args ) );
$res = call_user_func_array( array( $conn, $cmd ), $args );
if ( $conn->getLastError() ) {
// Make all errors be exceptions instead of "most but
not all".
--
To view, visit https://gerrit.wikimedia.org/r/172011
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie927b49b0b21a4478e7ea8c207d1e65b9c63c729
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/jobrunner
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <[email protected]>
Gerrit-Reviewer: Aaron Schulz <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
