TimedMediaHandler is particularly bad at this right now because it's still in the middle of a switch from an old set of frontend modules to a new one, so we have to register both. :) I'll make sure we chop the old ones out soon...
-- brion On Mon, Sep 16, 2019 at 2:11 PM Amir Sarabadani <ladsgr...@gmail.com> wrote: > Hello, > Startup module, is the Javascript code [1] that is being served at almost > every request to Wikimedia as part of <head> (non-blocking though) to load > other RL modules. It does important things for example it checks if the > requested modules already exist with the given hash in the local storage > cache. > > Because of the given reasons, the startup module needs and includes list of > all registered modules with their version hash in the code. As the result > if you register a module, even if you don't load it, it adds a rather small > overhead (around 19 bytes after minification and compression) to every > request to any wiki the code is enabled. So if you register a new module in > core or one of the extensions that is deployed in all wikis (CX, > WikibaseClient, etc.), it's going to be added to every page request, > meaning 30 GBs more traffic every day to our users. Even if you don't use > it anywhere. > > If you're adding a feature, 30GB/day is acceptable but sometimes developers > use Resource loader for dependency management or class loading (yours truly > used to do that) and introduce 20 modules instead of one and each one > causing an extra 600 GB/day. The big overhead for our users is bad for > three reasons: 1- RL has to handle the dependency management, making every > page view slightly slower and worse UX for our users 2- The extra network > is bad with places without access to broadband connection, where Wikipedia > is the most likely place that people learn and grow [2] 3- The scale we are > talking is so massive (petabytes a year) that It has environmental impact. > > Let me give you an example, a couple of weeks ago we dropped 137 modules > from WikibaseClient. After deployment, it dropped 4TB/day from our network > (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms > from every response time [3]. > > We now have a dashboard to track size of size of RL registry [4] plus a > weekly metrics for changes [5][6] > > If you're looking for ways to help. I wrote a tool [7] to do some graph > analysis and it gives list of extensions that has modules that can be > merged. The extensions that according to the analysis (that can have false > positives) can get better are TimedMediaHandler, PageTriage, Graph, > RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict, > Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more > including some bits and pieces in core. I put the graphs of modules that > can be merged at [8] and I invite you to have fun with those modules. > Modules can be merged using package modules [9] > > Most of the is work done by the performance team [10] and volunteers and > developers in lots of teams. I joined the party later as volunteer/WMDE > staff and I'm sharing mostly the results and writing long emails. Big kudos > to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan, > Alexandros Kosiaris, and so many others that helped and I forgot. > > [1] An example from English Wikipedia: > > https://en.wikipedia.org/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector&debug=true > [2] https://arxiv.org/abs/1812.00474 > [3] https://phabricator.wikimedia.org/T203696#5387672 > [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1 > [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07 > [6] > > https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZb4U > [7] https://phabricator.wikimedia.org/T232728 > [8] https://phabricator.wikimedia.org/T233048 > [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules > [10] https://phabricator.wikimedia.org/T202154 > > Best > -- > Amir (he/him) > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
