ArthurTaylor added a comment.
I had a poke around at this the last days. Quick status update from my side.
Running Quibble locally it's possible to reproduce the issue:
mkdir -p src && mkdir -p cache && mkdir -p log && mkdir -p ref
docker run --entrypoint=quibble-with-supervisord \
--tmpfs /workspace/db:size=320M \
--volume "$(pwd)"/src:/workspace/src \
--volume "$(pwd)"/cache:/cache \
--volume "$(pwd)"/log:/workspace/log \
--volume "$(pwd)"/ref:/srv/git:ro \
--security-opt seccomp=unconfined \
--env-file=.env --init --rm \
docker-registry.wikimedia.org/releng/quibble-buster-php74:1.6.0-s6 \
--packages-source composer \
--db mysql --db-dir /workspace/db \
--git-parallel=8 \
--run qunit
`.env`:
ZUUL_PROJECT=mediawiki/extensions/Wikibase
ZUUL_BRANCH=REL1_40
EXT_NAME=Wikibase
SKIN_DEPENDENCIES=mediawiki/skins/MinervaNeue\nmediawiki/skins/Vector
EXT_DEPENDENCIES=mediawiki/extensions/CirrusSearch\nmediawiki/extensions/VisualEditor\nmediawiki/extensions/Wikibase\nmediawiki/extensions/Scribunto\nmediawiki/extensions/Elastica\nmediawiki/extensions/Echo\nmediawiki/extensions/DiscussionTools\nmediawiki/extensions/Linter
I created a local version of the Quibble container with XDebug enabled and
was able to get this stack trace for the failed request: F42443515:
load-php-request-failed-trace.php.html
<https://phabricator.wikimedia.org/F42443515>
I've also been able to generate profiler output for failed and successful
requests, but they are ~400MB in size so I will not include them here.
Running the QUnit tests hits the 128MB memory limit for the container when a
request like this needs to be served:
http://127.0.0.1:9413/load.php?modules=mediawiki.language%2CmessagePoster%2Cpulsatingdot%2Cqunit-testrunner%2Crouter%2Cstorage%2Ctemplate%2Ctoc%2Cuser%2Cutil%2CvisibleTimeout%2Cwidgets%7Cmediawiki.language.months%2Cnames%2CspecialCharacters%2Ctestdata%7Cmediawiki.libs.guiders%2Cjpegmeta%2Cpluralruleparser%7Cmediawiki.page.gallery.styles%7Cmediawiki.page.ready%7Cmediawiki.page.watch.ajax%7Cmediawiki.rcfilters.filters.ui%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.template.handlebars%2Cmustache%2Cvue%7Cmediawiki.ui.button%2Cicon%7Cmediawiki.widgets.AbandonEditDialog%2CCategoryMultiselectWidget%2CDateInputWidget%2CMediaSearch%2CTable%2CUserInputWidget%2Cstyles%7Cmediawiki.widgets.DateInputWidget.styles%7Cmobile.ooui.icons%7Cmobile.pagelist.styles%7Cmobile.pagesummary.styles%7Cmobile.placeholder.images%7Cmobile.startup%7Cmobile.startup.images%7Cmoment%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Cpapaparse%2Crangefix%2Csinonjs%2Cspark-md5%2CtreeDiffer%2Cunicodejs%2CvalueFormatters%2CvalueParsers%7Cmw.config.values.wbDataTypes%2CwbRefTabsEnabled%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.icons%2Cstyles%7Coojs-ui-toolbars.icons%7Coojs-ui-widgets.icons%7Coojs-ui-windows.icons%7Coojs-ui.styles.icons-accessibility%2Cicons-alerts%2Cicons-content%2Cicons-editing-advanced%2Cicons-editing-citation%2Cicons-editing-core%2Cicons-editing-list%2Cicons-editing-styling%2Cicons-interactions%2Cicons-layout%2Cicons-media%2Cicons-moderation%2Cicons-movement%2Cicons-user%2Cindicators%7Cskins.minerva.messageBox.styles%7Csocket.io%7Ctest.ArticlePlaceholder%2CCentralAuth%2CCite%2CDiscussionTools%2CEcho%2CEventLogging%2CFlow%2CGuidedTour%2CMediaWiki%2CMinervaNeue%2CTemplateData%2CThanks%2CUniversalLanguageSelector%2CVisualEditor%2CWikiEditor%2CWikibaseClient%2CWikibaseLexeme%2CWikibaseRepository%2CWikimediaEvents%7Cutil.ContentLanguages%2CExtendable%2CMessageProvider%2CMessageProviders%2CNotifier%2ChighlightSubstring%2Cinherit%7CvalueParsers.ValueParserStore%2Cparsers&version=1a741http://127.0.0.1:9413/load.php?modules=mediawiki.language%2CmessagePoster%2Cpulsatingdot%2Cqunit-testrunner%2Crouter%2Cstorage%2Ctemplate%2Ctoc%2Cuser%2Cutil%2CvisibleTimeout%2Cwidgets%7Cmediawiki.language.months%2Cnames%2CspecialCharacters%2Ctestdata%7Cmediawiki.libs.guiders%2Cjpegmeta%2Cpluralruleparser%7Cmediawiki.page.gallery.styles%7Cmediawiki.page.ready%7Cmediawiki.page.watch.ajax%7Cmediawiki.rcfilters.filters.ui%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.template.handlebars%2Cmustache%2Cvue%7Cmediawiki.ui.button%2Cicon%7Cmediawiki.widgets.AbandonEditDialog%2CCategoryMultiselectWidget%2CDateInputWidget%2CMediaSearch%2CTable%2CUserInputWidget%2Cstyles%7Cmediawiki.widgets.DateInputWidget.styles%7Cmobile.ooui.icons%7Cmobile.pagelist.styles%7Cmobile.pagesummary.styles%7Cmobile.placeholder.images%7Cmobile.startup%7Cmobile.startup.images%7Cmoment%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Cpapaparse%2Crangefix%2Csinonjs%2Cspark-md5%2CtreeDiffer%2Cunicodejs%2CvalueFormatters%2CvalueParsers%7Cmw.config.values.wbDataTypes%2CwbRefTabsEnabled%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.icons%2Cstyles%7Coojs-ui-toolbars.icons%7Coojs-ui-widgets.icons%7Coojs-ui-windows.icons%7Coojs-ui.styles.icons-accessibility%2Cicons-alerts%2Cicons-content%2Cicons-editing-advanced%2Cicons-editing-citation%2Cicons-editing-core%2Cicons-editing-list%2Cicons-editing-styling%2Cicons-interactions%2Cicons-layout%2Cicons-media%2Cicons-moderation%2Cicons-movement%2Cicons-user%2Cindicators%7Cskins.minerva.messageBox.styles%7Csocket.io%7Ctest.ArticlePlaceholder%2CCentralAuth%2CCite%2CDiscussionTools%2CEcho%2CEventLogging%2CFlow%2CGuidedTour%2CMediaWiki%2CMinervaNeue%2CTemplateData%2CThanks%2CUniversalLanguageSelector%2CVisualEditor%2CWikiEditor%2CWikibaseClient%2CWikibaseLexeme%2CWikibaseRepository%2CWikimediaEvents%7Cutil.ContentLanguages%2CExtendable%2CMessageProvider%2CMessageProviders%2CNotifier%2ChighlightSubstring%2Cinherit%7CvalueParsers.ValueParserStore%2Cparsers&version=1a741
In particular, loading the modules and dependencies for the
`test.DiscussionTools` QUnit tests loads in about 150 files, some of which are
many megabytes in size. The ResourceLoader loads these from disk into memory,
duplicates the strings in memory while it wraps and unwraps them in `XmlJsCode`
objects (`HtmlJsCode` on master), and it's eventually a call to `implode` just
before the output is sent to the client which causes the process to ask for too
much memory. The memory usage is not helped by the fact that the `Module`
objects keep a cache of the content, and the `ResourceLoader` keeps a cache of
modules. So my working theory is that in the first pass, where we have to warm
those caches and manipulate all the strings in memory, for particularly long
module lists, there are just too many bytes involved.
My plan for next week is to attach `gdb` and get a heap dump to see where all
the memory is going. There might be some low-hanging fruit for optimising the
way we use memory.
Also worth noting:
- Running Quibble without php-fpm doesn't have this issue. I'm not currently
sure why this is - it could be an implementation quirk, or it could be that
we're right at the boundary of the 128MB and `php-fpm` just has a little bit
extra overhead. I haven't looked deeply into the memory accounting there
- Making a second request to the same URL works fine. Presumably serving the
modules from the heap-cached `Module` objects uses less memory than streaming
them in from disk and generating the memory representation and caching that
before serving the module.
TASK DETAIL
https://phabricator.wikimedia.org/T356896
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: ArthurTaylor
Cc: karapayneWMDE, Lucas_Werkmeister_WMDE, Aklapper, Michael,
Danny_Benjafield_WMDE, Astuthiodit_1, Invadibot, maantietaja, ItamarWMDE,
Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, cmadeo, KimKelting,
LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, jayvdb,
Ricordisamoa, Mbch331
_______________________________________________
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
