hashar added a comment.

@Addshore wrote https://logstash.wikimedia.org/goto/259821dc32242eb3fde0cd02755685f6

There are a few failing wfShellExec in the exec log bucket:

$ grep -c getConfiguration archive/exec.log-20160916
248

That is from 19:12:01 till 19:59:54 and they all come from terbium for the wikidatawiki. The failing message being:

Probably exited with signal 25:
/bin/bash '/srv/mediawiki/php-1.28.0-wmf.19/includes/limit.sh' ''\''/usr/bin/php'\'' '\''/srv/mediawiki/multiversion/MWScript.php'\'' '\''maintenance/getConfiguration.php'\'' '\''--wiki'\''  SOMEWIKI_NAME_HERE '\''--settings'\'' '\''wgJobClasses'\'' '\''--format'\'' '\''PHP'\''' 'MW_INCLUDE_STDERR=;MW_CPU_LIMIT=50; MW_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; MW_MEM_LIMIT=0; MW_FILE_SIZE_LIMIT=524288; MW_WALL_CLOCK_LIMIT=180; MW_USE_LOG_PIPE=yes'

Or in a human friendly way:

MW_INCLUDE_STDERR=;
MW_CPU_LIMIT=50;
MW_CGROUP='/sys/fs/cgroup/memory/mediawiki/job';
MW_MEM_LIMIT=0;
MW_FILE_SIZE_LIMIT=524288;
MW_WALL_CLOCK_LIMIT=180;
MW_USE_LOG_PIPE=yes
mwscript maintenance/getConfiguration.php --wiki=WIKI_NAME --settings wgJobClasses --format PHP

The "signal 25" is extracted by wfShellExec. Looks like the exit code was 153 or 128 + 25. From man 7 signal that would be SIGXFSZ:

SIGXFSZ     25,25,31    Core    File size limit exceeded (4.2BSD)

MW_FILE_SIZE_LIMIT=524288 is passed as ulimit -f:

-f        the maximum size of files written by the shell and its children

And 524288 == 512 * 1024 bytes

That value seems to come from $wgMaxShellFileSize. Default to 102400 in MediaWiki and on Wikimedia config it is at 512 * 1024:

wmf-config/CommonSettings.php:1564:$wgMaxShellFileSize = 512 * 1024;

That has been the case for as long as 2015-10-31 and f85267e35fed0d38bb58b181e3382a63f1796266 that made the value the default (previously it was only applied when /etc/wikimedia-image-scaler file existed.


Anyway. Gotta:

  • reproduce the file limit and see whether it is areason.
  • figure out why it overflow
  • is that really the root cause?

TASK DETAIL
https://phabricator.wikimedia.org/T145819

EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: hashar
Cc: Lydia_Pintscher, daniel, aude, Addshore, Aklapper, greg, Legoktm, demon, gerritbot, Stashbot, hashar, Lewizho99, Maathavan, D3r1ck01, Liudvikas, Izno, Luke081515, Wikidata-bugs, JanZerebecki, zeljkofilipin, Mbch331, Jay8g
_______________________________________________
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

Reply via email to