Re: [Wikitech-l] Startup module size and ResourceLoader registry overhead

[Greg Grossmeier]

On Thu, Sep 19, 2019 at 05:09:59PM +0200, Daniel Kinzler wrote:
> My friend @wetterfrosch points out on twitter that reducing our daily 
> bandwidth
> usage by 4.3TB reduces CO2 emission by 30 tons PER DAY. That's roughly
> equivalent to flying 20 people from Berlin to San Francisco and back. Every 
> day.

More events/conferences then, got it! :)

Congrats and thanks Performance and others!

Greg

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| Release Team ManagerA18D 1138 8E47 FAC8 1C7D |

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Startup module size and ResourceLoader registry overhead

[Daniel Kinzler]

My friend @wetterfrosch points out on twitter that reducing our daily bandwidth
usage by 4.3TB reduces CO2 emission by 30 tons PER DAY. That's roughly
equivalent to flying 20 people from Berlin to San Francisco and back. Every day.

Thank you all!

Sources:
https://twitter.com/wetterfrosch/status/1174675018795638785
https://alistapart.com/article/sustainable-web-design/
https://en.wikipedia.org/wiki/Environmental_impact_of_transport
https://www.carbonfootprint.com/calculator.aspx

Am 16.09.19 um 23:10 schrieb Amir Sarabadani:
> Hello,
> Startup module, is the Javascript code [1] that is being served at almost
> every request to Wikimedia as part of  (non-blocking though) to load
> other RL modules. It does important things for example it checks if the
> requested modules already exist with the given hash in the local storage
> cache.
> 
> Because of the given reasons, the startup module needs and includes list of
> all registered modules with their version hash in the code. As the result
> if you register a module, even if you don't load it, it adds a rather small
> overhead (around 19 bytes after minification and compression) to every
> request to any wiki the code is enabled. So if you register a new module in
> core or one of the extensions that is deployed in all wikis (CX,
> WikibaseClient, etc.), it's going to be added to every page request,
> meaning 30 GBs more traffic every day to our users. Even if you don't use
> it anywhere.
> 
> If you're adding a feature, 30GB/day is acceptable but sometimes developers
> use Resource loader for dependency management or class loading (yours truly
> used to do that) and introduce 20 modules instead of one and each one
> causing an extra 600 GB/day. The big overhead for our users is bad for
> three reasons: 1- RL has to handle the dependency management, making every
> page view slightly slower and worse UX for our users 2- The extra network
> is bad with places without access to broadband connection, where Wikipedia
> is the most likely place that people learn and grow [2] 3- The scale we are
> talking is so massive (petabytes a year) that It has environmental impact.
> 
> Let me give you an example, a couple of weeks ago we dropped 137 modules
> from WikibaseClient. After deployment, it dropped 4TB/day from our network
> (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms
> from every response time [3].
> 
> We now have a dashboard to track size of size of RL registry [4] plus a
> weekly metrics for changes [5][6]
> 
> If you're looking for ways to help. I wrote a tool [7] to do some graph
> analysis and it gives list of extensions that has modules that can be
> merged. The extensions that according to the analysis (that can have false
> positives) can get better are TimedMediaHandler, PageTriage, Graph,
> RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict,
> Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more
> including some bits and pieces in core. I put the graphs of modules that
> can be merged at [8] and I invite you to have fun with those modules.
> Modules can be merged using package modules [9]
> 
> Most of the is work done by the performance team [10] and volunteers and
> developers in lots of teams. I joined the party later as volunteer/WMDE
> staff and I'm sharing mostly the results and writing long emails. Big kudos
> to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan,
> Alexandros Kosiaris, and so many others that helped and I forgot.
> 
> [1] An example from English Wikipedia:
> https://en.wikipedia.org/w/load.php?lang=en=startup=scripts=1=vector=true
> [2] https://arxiv.org/abs/1812.00474
> [3] https://phabricator.wikimedia.org/T203696#5387672
> [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1
> [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07
> [6]
> https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZb4U
> [7] https://phabricator.wikimedia.org/T232728
> [8] https://phabricator.wikimedia.org/T233048
> [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules
> [10] https://phabricator.wikimedia.org/T202154
> 
> Best
> 

-- 
Daniel Kinzler
Principal Software Engineer, Core Platform
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Startup module size and ResourceLoader registry overhead

[Kosta Harlan]

Amazing work!

See also 
https://phabricator.wikimedia.org/phame/post/view/175/wikipedia_s_javascript_initialisation_on_a_budget/
 

 

Kosta

> On Sep 16, 2019, at 11:10 PM, Amir Sarabadani  wrote:
> 
> Hello,
> Startup module, is the Javascript code [1] that is being served at almost
> every request to Wikimedia as part of  (non-blocking though) to load
> other RL modules. It does important things for example it checks if the
> requested modules already exist with the given hash in the local storage
> cache.
> 
> Because of the given reasons, the startup module needs and includes list of
> all registered modules with their version hash in the code. As the result
> if you register a module, even if you don't load it, it adds a rather small
> overhead (around 19 bytes after minification and compression) to every
> request to any wiki the code is enabled. So if you register a new module in
> core or one of the extensions that is deployed in all wikis (CX,
> WikibaseClient, etc.), it's going to be added to every page request,
> meaning 30 GBs more traffic every day to our users. Even if you don't use
> it anywhere.
> 
> If you're adding a feature, 30GB/day is acceptable but sometimes developers
> use Resource loader for dependency management or class loading (yours truly
> used to do that) and introduce 20 modules instead of one and each one
> causing an extra 600 GB/day. The big overhead for our users is bad for
> three reasons: 1- RL has to handle the dependency management, making every
> page view slightly slower and worse UX for our users 2- The extra network
> is bad with places without access to broadband connection, where Wikipedia
> is the most likely place that people learn and grow [2] 3- The scale we are
> talking is so massive (petabytes a year) that It has environmental impact.
> 
> Let me give you an example, a couple of weeks ago we dropped 137 modules
> from WikibaseClient. After deployment, it dropped 4TB/day from our network
> (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms
> from every response time [3].
> 
> We now have a dashboard to track size of size of RL registry [4] plus a
> weekly metrics for changes [5][6]
> 
> If you're looking for ways to help. I wrote a tool [7] to do some graph
> analysis and it gives list of extensions that has modules that can be
> merged. The extensions that according to the analysis (that can have false
> positives) can get better are TimedMediaHandler, PageTriage, Graph,
> RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict,
> Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more
> including some bits and pieces in core. I put the graphs of modules that
> can be merged at [8] and I invite you to have fun with those modules.
> Modules can be merged using package modules [9]
> 
> Most of the is work done by the performance team [10] and volunteers and
> developers in lots of teams. I joined the party later as volunteer/WMDE
> staff and I'm sharing mostly the results and writing long emails. Big kudos
> to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan,
> Alexandros Kosiaris, and so many others that helped and I forgot.
> 
> [1] An example from English Wikipedia:
> https://en.wikipedia.org/w/load.php?lang=en=startup=scripts=1=vector=true
> [2] https://arxiv.org/abs/1812.00474
> [3] https://phabricator.wikimedia.org/T203696#5387672
> [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1
> [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07
> [6]
> https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZb4U
> [7] https://phabricator.wikimedia.org/T232728
> [8] https://phabricator.wikimedia.org/T233048
> [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules
> [10] https://phabricator.wikimedia.org/T202154
> 
> Best
> -- 
> Amir (he/him)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Startup module size and ResourceLoader registry overhead

[Brion Vibber]

TimedMediaHandler is particularly bad at this right now because it's still
in the middle of a switch from an old set of frontend modules to a new one,
so we have to register both. :) I'll make sure we chop the old ones out
soon...

-- brion

On Mon, Sep 16, 2019 at 2:11 PM Amir Sarabadani  wrote:

> Hello,
> Startup module, is the Javascript code [1] that is being served at almost
> every request to Wikimedia as part of  (non-blocking though) to load
> other RL modules. It does important things for example it checks if the
> requested modules already exist with the given hash in the local storage
> cache.
>
> Because of the given reasons, the startup module needs and includes list of
> all registered modules with their version hash in the code. As the result
> if you register a module, even if you don't load it, it adds a rather small
> overhead (around 19 bytes after minification and compression) to every
> request to any wiki the code is enabled. So if you register a new module in
> core or one of the extensions that is deployed in all wikis (CX,
> WikibaseClient, etc.), it's going to be added to every page request,
> meaning 30 GBs more traffic every day to our users. Even if you don't use
> it anywhere.
>
> If you're adding a feature, 30GB/day is acceptable but sometimes developers
> use Resource loader for dependency management or class loading (yours truly
> used to do that) and introduce 20 modules instead of one and each one
> causing an extra 600 GB/day. The big overhead for our users is bad for
> three reasons: 1- RL has to handle the dependency management, making every
> page view slightly slower and worse UX for our users 2- The extra network
> is bad with places without access to broadband connection, where Wikipedia
> is the most likely place that people learn and grow [2] 3- The scale we are
> talking is so massive (petabytes a year) that It has environmental impact.
>
> Let me give you an example, a couple of weeks ago we dropped 137 modules
> from WikibaseClient. After deployment, it dropped 4TB/day from our network
> (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms
> from every response time [3].
>
> We now have a dashboard to track size of size of RL registry [4] plus a
> weekly metrics for changes [5][6]
>
> If you're looking for ways to help. I wrote a tool [7] to do some graph
> analysis and it gives list of extensions that has modules that can be
> merged. The extensions that according to the analysis (that can have false
> positives) can get better are TimedMediaHandler, PageTriage, Graph,
> RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict,
> Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more
> including some bits and pieces in core. I put the graphs of modules that
> can be merged at [8] and I invite you to have fun with those modules.
> Modules can be merged using package modules [9]
>
> Most of the is work done by the performance team [10] and volunteers and
> developers in lots of teams. I joined the party later as volunteer/WMDE
> staff and I'm sharing mostly the results and writing long emails. Big kudos
> to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan,
> Alexandros Kosiaris, and so many others that helped and I forgot.
>
> [1] An example from English Wikipedia:
>
> https://en.wikipedia.org/w/load.php?lang=en=startup=scripts=1=vector=true
> [2] https://arxiv.org/abs/1812.00474
> [3] https://phabricator.wikimedia.org/T203696#5387672
> [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1
> [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07
> [6]
>
> https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZb4U
> [7] https://phabricator.wikimedia.org/T232728
> [8] https://phabricator.wikimedia.org/T233048
> [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules
> [10] https://phabricator.wikimedia.org/T202154
>
> Best
> --
> Amir (he/him)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Startup module size and ResourceLoader registry overhead

[Amir Sarabadani]

Hello,
Startup module, is the Javascript code [1] that is being served at almost
every request to Wikimedia as part of  (non-blocking though) to load
other RL modules. It does important things for example it checks if the
requested modules already exist with the given hash in the local storage
cache.

Because of the given reasons, the startup module needs and includes list of
all registered modules with their version hash in the code. As the result
if you register a module, even if you don't load it, it adds a rather small
overhead (around 19 bytes after minification and compression) to every
request to any wiki the code is enabled. So if you register a new module in
core or one of the extensions that is deployed in all wikis (CX,
WikibaseClient, etc.), it's going to be added to every page request,
meaning 30 GBs more traffic every day to our users. Even if you don't use
it anywhere.

If you're adding a feature, 30GB/day is acceptable but sometimes developers
use Resource loader for dependency management or class loading (yours truly
used to do that) and introduce 20 modules instead of one and each one
causing an extra 600 GB/day. The big overhead for our users is bad for
three reasons: 1- RL has to handle the dependency management, making every
page view slightly slower and worse UX for our users 2- The extra network
is bad with places without access to broadband connection, where Wikipedia
is the most likely place that people learn and grow [2] 3- The scale we are
talking is so massive (petabytes a year) that It has environmental impact.

Let me give you an example, a couple of weeks ago we dropped 137 modules
from WikibaseClient. After deployment, it dropped 4TB/day from our network
(= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms
from every response time [3].

We now have a dashboard to track size of size of RL registry [4] plus a
weekly metrics for changes [5][6]

If you're looking for ways to help. I wrote a tool [7] to do some graph
analysis and it gives list of extensions that has modules that can be
merged. The extensions that according to the analysis (that can have false
positives) can get better are TimedMediaHandler, PageTriage, Graph,
RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict,
Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more
including some bits and pieces in core. I put the graphs of modules that
can be merged at [8] and I invite you to have fun with those modules.
Modules can be merged using package modules [9]

Most of the is work done by the performance team [10] and volunteers and
developers in lots of teams. I joined the party later as volunteer/WMDE
staff and I'm sharing mostly the results and writing long emails. Big kudos
to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan,
Alexandros Kosiaris, and so many others that helped and I forgot.

[1] An example from English Wikipedia:
https://en.wikipedia.org/w/load.php?lang=en=startup=scripts=1=vector=true
[2] https://arxiv.org/abs/1812.00474
[3] https://phabricator.wikimedia.org/T203696#5387672
[4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1
[5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07
[6]
https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZb4U
[7] https://phabricator.wikimedia.org/T232728
[8] https://phabricator.wikimedia.org/T233048
[9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules
[10] https://phabricator.wikimedia.org/T202154

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l