EBernhardson has uploaded a new change for review.
https://gerrit.wikimedia.org/r/172011
Change subject: Add logging to track down OOM
......................................................................
Add logging to track down OOM
The WMF job runners have been OOM'ing fairly recently lately.
Watching the memory for the jobrunners we can see they don't grow
their memory over time like a leak, instead they sit at 80M for
some time until quickly growing to >300M and dieing. The logs
all point to line 666, which issues commands to redis. I'm not
sure whats going on yet, but I think if we log the commands so we
know what it did just before OOM we can figure that out.
Change-Id: Ie927b49b0b21a4478e7ea8c207d1e65b9c63c729
---
M redisJobRunnerService
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/mediawiki/services/jobrunner
refs/changes/11/172011/1
diff --git a/redisJobRunnerService b/redisJobRunnerService
index 894c32c..69ec126 100755
--- a/redisJobRunnerService
+++ b/redisJobRunnerService
@@ -662,6 +662,9 @@
*/
protected function redisCmd( Redis $conn, $cmd, array $args = array() )
{
$conn->clearLastError();
+ // we had some job runners oom'ing on this call, log what we are
+ // doing so there is relevant information next to the oom
+ $this->debug( "Redis cmd: $cmd " . json_encode( $args ) );
$res = call_user_func_array( array( $conn, $cmd ), $args );
if ( $conn->getLastError() ) {
// Make all errors be exceptions instead of "most but
not all".
--
To view, visit https://gerrit.wikimedia.org/r/172011
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie927b49b0b21a4478e7ea8c207d1e65b9c63c729
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/services/jobrunner
Gerrit-Branch: master
Gerrit-Owner: EBernhardson <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
