| EBernhardson created this task. EBernhardson added projects: Dumps-Generation, Wikidata. |
TASK DESCRIPTION
Was looking into why a number of cirrus dumps this week came out with 0 size (well, 20 but thats just the compression header). From snapshot1007.eqiad.wmnet:/var/log/cirrusdump/cirrusdump-alswiki-20170327-cirrussearch-content.log:
could not allocate 218103807 bytes for translation cache [Mon Mar 27 16:16:46 2017] [hphp] [24518:7f1008f43100:0:000001] [] Lost parent, LightProcess exiting [Mon Mar 27 16:16:46 2017] [hphp] [24515:7f1008f43100:0:000001] [] Lost parent, LightProcess exiting
dmesg doesn't have that specific entry, but i see dmesg logs for killing hhvm, php5 and python throughout the day.
Pastie of one such event: P5145The dmesg output unfortunately only says the program, and not the various command line flags used to run the script. Looking at the machine right now there are a few processes using whats seems like an outsized amount of memory:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 10045 datasets 20 0 8191576 4.729g 3948 R 91.9 7.5 57:09.34 php 10038 datasets 20 0 8162064 4.726g 3960 R 90.1 7.5 57:19.45 php 10047 datasets 20 0 8146932 4.714g 3960 R 86.4 7.5 56:59.44 php 10036 datasets 20 0 8144216 4.714g 3948 S 93.8 7.5 57:33.15 php 10042 datasets 20 0 8081348 4.645g 3952 S 80.9 7.4 57:15.63 phpThose all belong to this process tree:
root 10021 0.0 0.0 59760 460 ? S Mar27 0:00 \_ CRON datasets 10022 0.0 0.0 4440 548 ? Ss Mar27 0:00 \_ /bin/sh -c /usr/local/bin/dumpwikidatattl.sh datasets 10023 0.0 0.0 12436 920 ? S Mar27 0:00 \_ /bin/bash /usr/local/bin/dumpwikidatattl.sh datasets 10034 0.0 0.0 12432 320 ? S Mar27 0:00 \_ /bin/bash /usr/local/bin/dumpwikidatattl.sh datasets 10038 87.0 7.7 8334096 5127396 ? R Mar27 58:41 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 5 --format ttl datasets 10056 0.0 0.0 381892 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 5 --format ttl datasets 10063 0.0 0.0 381892 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 5 --format ttl datasets 10069 0.0 0.0 381892 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 5 --format ttl datasets 10074 0.0 0.0 381892 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 5 --format ttl datasets 10078 0.0 0.0 381892 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 0 --sharding-factor 5 --format ttl datasets 10040 3.6 0.0 4596 788 ? S Mar27 2:29 | \_ gzip datasets 10035 0.0 0.0 12432 320 ? S Mar27 0:00 \_ /bin/bash /usr/local/bin/dumpwikidatattl.sh datasets 10036 87.4 7.7 8316248 5115320 ? R Mar27 58:55 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 5 --format ttl datasets 10054 0.0 0.0 382448 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 5 --format ttl datasets 10060 0.0 0.0 382448 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 5 --format ttl datasets 10065 0.0 0.0 382448 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 5 --format ttl datasets 10071 0.0 0.0 382448 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 5 --format ttl datasets 10077 0.0 0.0 382448 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 1 --sharding-factor 5 --format ttl datasets 10039 3.7 0.0 4596 788 ? S Mar27 2:29 | \_ gzip datasets 10037 0.0 0.0 12432 320 ? S Mar27 0:00 \_ /bin/bash /usr/local/bin/dumpwikidatattl.sh datasets 10042 86.9 7.6 8253380 5043028 ? R Mar27 58:38 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 2 --sharding-factor 5 --format ttl datasets 10058 0.0 0.0 381024 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 2 --sharding-factor 5 --format ttl datasets 10062 0.0 0.0 381024 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 2 --sharding-factor 5 --format ttl datasets 10066 0.0 0.0 381024 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 2 --sharding-factor 5 --format ttl datasets 10072 0.0 0.0 381024 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 2 --sharding-factor 5 --format ttl datasets 10075 0.0 0.0 381024 43760 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 2 --sharding-factor 5 --format ttl datasets 10043 3.6 0.0 4596 784 ? S Mar27 2:29 | \_ gzip datasets 10041 0.0 0.0 12432 320 ? S Mar27 0:00 \_ /bin/bash /usr/local/bin/dumpwikidatattl.sh datasets 10047 86.5 7.7 8314868 5111292 ? R Mar27 58:20 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 3 --sharding-factor 5 --format ttl datasets 10055 0.0 0.0 381068 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 3 --sharding-factor 5 --format ttl datasets 10059 0.0 0.0 381068 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 3 --sharding-factor 5 --format ttl datasets 10064 0.0 0.0 381068 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 3 --sharding-factor 5 --format ttl datasets 10068 0.0 0.0 381068 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 3 --sharding-factor 5 --format ttl datasets 10070 0.0 0.0 381068 43764 ? Ss Mar27 0:00 | | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 3 --sharding-factor 5 --format ttl datasets 10048 3.7 0.0 4596 788 ? S Mar27 2:29 | \_ gzip datasets 10044 0.0 0.0 12432 320 ? S Mar27 0:00 \_ /bin/bash /usr/local/bin/dumpwikidatattl.sh datasets 10045 86.8 7.7 8359512 5129128 ? R Mar27 58:30 \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 4 --sharding-factor 5 --format ttl datasets 10057 0.0 0.0 380660 41780 ? Ss Mar27 0:00 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 4 --sharding-factor 5 --format ttl datasets 10061 0.0 0.0 380660 41780 ? Ss Mar27 0:00 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 4 --sharding-factor 5 --format ttl datasets 10067 0.0 0.0 380660 41780 ? Ss Mar27 0:00 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 4 --sharding-factor 5 --format ttl datasets 10073 0.0 0.0 380660 41780 ? Ss Mar27 0:00 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 4 --sharding-factor 5 --format ttl datasets 10076 0.0 0.0 380660 41780 ? Ss Mar27 0:00 | \_ php /srv/mediawiki/multiversion/MWScript.php extensions/Wikidata/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --shard 4 --sharding-factor 5 --format ttl datasets 10046 3.7 0.0 4596 784 ? S Mar27 2:30 \_ gzipI don't know for sure its the rdf dumping causing this, but it seems a likely culprit
TASK DETAIL
EMAIL PREFERENCES
To: EBernhardson
Cc: ArielGlenn, Gehel, EBernhardson, Lunewa, QZanden, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, aude, Mbch331
Cc: ArielGlenn, Gehel, EBernhardson, Lunewa, QZanden, D3r1ck01, Izno, gnosygnu, Wikidata-bugs, aude, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
