[Wikitech-l] Re: 3D models in Commons

[Brion Vibber]

Based on my prior research I'm pretty sure GLTF support -- a well-supported
modern format that allows textures -- is not a huge amount of work to add.
Mostly needs cleanup of the old code & updating dependencies, and adding
appropriate validation.

I am planning to get to that this year, after we wrap up some more
improvements to audio/video making sure we can update the thumbnailing
system; but I don't know if we'll be able to squeeze any other 3d-specific
features in yet.

-- brion

On Sun, Jul 3, 2022 at 3:52 AM Strainu  wrote:

>
>
> Pe duminică, 3 iulie 2022, Derk-Jan Hartman 
> a scris:
> > You mean 3d models besides the type we already support ?
> > https://www.mediawiki.org/wiki/Extension:3D
>
> Yes, specifically the formats that support textures. There is a ticket
> list in phab:
> https://phabricator.wikimedia.org/maniphest/query/ZgJlL4Jm.OCj/#R
>
> Strainu
> >
> > On 3 Jul 2022, at 11:49, Strainu  wrote:
> > Hey folks,
> >
> > I know it's a bit early in the fiscal year and that's probably why I
> can't find anything on wiki, but I understood that the new yearly plan puts
> a lot of emphasis on the multimedia features. Does that include making 3D
> models usable in our wikis? If there are planned projects related to that
> in this fiscal year, would it be possible to get a link to the project page?
> >
> > I'm asking because there are a lot of cool, freely licensed models out
> there just waiting to be imported...
> >
> > Thank you,
> >  Strainu ___
> > Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> > To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> >
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
> > ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re:  Amir Sarabadani receives Web Perf Hero award!

[Brion Vibber]

On Thu, May 26, 2022 at 7:22 AM Krinkle  wrote:

> I'm happy to share that the first Web Perf Hero award of 2022 goes to Amir
> Sarabadani!
>
> This award is in recognition of Amir's work (@Ladsgroup) over the past six
> months, in which he demonstrated deep expertise of the MediaWiki platform
> and significantly sped up the process of saving edits in MediaWiki. This
> improved both the potential of MediaWiki core, and as experienced
> concretely on WMF wikis, especially on Wikidata.org.
>

Woohoo! Major congrats to Amir. :)

-- brion
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Commons-l] Uplifting the multimedia stack (was: Community Wishlist Survery)

[Brion Vibber]

(Anyway I'm just grumping. I hear positive things about plans for this year
and I'm heartened to see more folks involved in planning the next stages!)

-- brion

On Mon, Jan 3, 2022, 6:10 AM Brion Vibber  wrote:

> On Thu, Dec 30, 2021, 10:27 AM Samuel Klein  wrote:
>
>> Separate thread.  I'm not sure which list is appropriate.
>> *... but not all the way to sentience
>> <https://en.wikipedia.org/wiki/The_Uplift_War>.*
>>
>> The annual community wishlist survey (implemented by a small team,
>> possibly in isolation?) may not be the mechanism for prioritizing large
>> changes, but the latter also deserves a community-curated priority queue.
>> To complement the staff-maintained priorities in phab ~
>>
>> For core challenges (like Commons stability and capacity), I'd be
>> surprised if the bottleneck were people or budget.
>>
>
> Currently there are zero people and no budget for multimedia, aside from
> whatever work I and others manage to get done here there. And I'm afraid I
> don't scale.
>
> It's Wikimedia Foundation's job to assign budget and people here. I've
> been hoping for years that this will happen, and continue to hope.
>
>
> -- brion
>
> We do need a shared understanding of what issues are most important and
>> most urgent, and how to solve them. For instance, a way to turn Amir's
>> recent email about the problem (and related phab tickets) into a family of
>> persistent, implementable specs and proposals and their articulated
>> obstacles.
>>
>> An issue tracker like phab is good for tracking the progress and
>> dependencies of agreed-upon tasks, but weak for discussing what is
>> important, what we know about it, how to address it. And weak for
>> discussing ecosystem-design issues that are important and need persistent
>> updating but don't have a simple checklist of steps.
>>
>> So where is the best current place to discuss scaling Commons, and all
>> that entails?  Some examples from recent discussions (most from the wm-l
>> thread below):
>> - *Uploads*: Support for large file uploads / Keeping bulk upload tools
>> online
>> - *Video*: Debugging + rolling out the videojs
>> <https://phabricator.wikimedia.org/T248418> player
>> - *Formats*: Adding support for CML
>> <https://phabricator.wikimedia.org/T18491> and dozens of other
>> <https://phabricator.wikimedia.org/T297514> common high-demand file
>> formats
>> - *Thumbs*: Updating thumbor <https://phabricator.wikimedia.org/T216815>
>> and librsvg <https://phabricator.wikimedia.org/T193352>
>> - *Search*: WCQS still <https://phabricator.wikimedia.org/T297454> down
>> <https://phabricator.wikimedia.org/T297454>, noauth option
>> <https://phabricator.wikimedia.org/T297995> wanted for tools
>> - *General*: Finish implementing redesign
>> <https://phabricator.wikimedia.org/T28741> of the image table
>>
>> SJ
>>
>> On Wed, Dec 29, 2021 at 6:26 AM Amir Sarabadani 
>> wrote:
>>
>>> I'm not debating your note. It is very valid that we lack proper support
>>> for multimedia stack. I myself wrote a detailed rant on how broken it is
>>> [1] but three notes:
>>>  - Fixing something like this takes time, you need to assign the budget
>>> for it (which means it has to be done during the annual planning) and if
>>> gets approved, you need to start it with the fiscal year (meaning July
>>> 2022) and then hire (meaning, write JD, do recruitment, interview lots of
>>> people, get them hired) which can take from several months to years. Once
>>> they are hired, you need to onboard them and let them learn about our
>>> technical infrastructure which takes at least two good months. Software
>>> engineering is not magic, it takes time, blood and sweat. [2]
>>>  - Making another team focus on multimedia requires changes in planning,
>>> budget, OKR, etc. etc. Are we sure moving the focus of teams is a good
>>> idea? Most teams are already focusing on vital parts of wikimedia and
>>> changing the focus will turn this into a whack-a-mole game where we fix
>>> multimedia but now we have critical issues in security or performance.
>>>  - Voting Wishlist survey is a good band-aid in the meantime. To at
>>> least address the worst parts for now.
>>>
>>> I don't understand your point tbh, either you think it's a good idea to
>>> make requests for improvements in multimedia in the wishlist survey or you
>>> think it's not. If you think it's not, then it's offtopic to this thread.
>>>
&g

[Wikitech-l] Re: [Commons-l] Uplifting the multimedia stack (was: Community Wishlist Survery)

[Brion Vibber]

On Thu, Dec 30, 2021, 10:27 AM Samuel Klein  wrote:

> Separate thread.  I'm not sure which list is appropriate.
> *... but not all the way to sentience
> .*
>
> The annual community wishlist survey (implemented by a small team,
> possibly in isolation?) may not be the mechanism for prioritizing large
> changes, but the latter also deserves a community-curated priority queue.
> To complement the staff-maintained priorities in phab ~
>
> For core challenges (like Commons stability and capacity), I'd be
> surprised if the bottleneck were people or budget.
>

Currently there are zero people and no budget for multimedia, aside from
whatever work I and others manage to get done here there. And I'm afraid I
don't scale.

It's Wikimedia Foundation's job to assign budget and people here. I've been
hoping for years that this will happen, and continue to hope.


-- brion

We do need a shared understanding of what issues are most important and
> most urgent, and how to solve them. For instance, a way to turn Amir's
> recent email about the problem (and related phab tickets) into a family of
> persistent, implementable specs and proposals and their articulated
> obstacles.
>
> An issue tracker like phab is good for tracking the progress and
> dependencies of agreed-upon tasks, but weak for discussing what is
> important, what we know about it, how to address it. And weak for
> discussing ecosystem-design issues that are important and need persistent
> updating but don't have a simple checklist of steps.
>
> So where is the best current place to discuss scaling Commons, and all
> that entails?  Some examples from recent discussions (most from the wm-l
> thread below):
> - *Uploads*: Support for large file uploads / Keeping bulk upload tools
> online
> - *Video*: Debugging + rolling out the videojs
>  player
> - *Formats*: Adding support for CML
>  and dozens of other
>  common high-demand file
> formats
> - *Thumbs*: Updating thumbor 
> and librsvg 
> - *Search*: WCQS still  down
> , noauth option
>  wanted for tools
> - *General*: Finish implementing redesign
>  of the image table
>
> SJ
>
> On Wed, Dec 29, 2021 at 6:26 AM Amir Sarabadani 
> wrote:
>
>> I'm not debating your note. It is very valid that we lack proper support
>> for multimedia stack. I myself wrote a detailed rant on how broken it is
>> [1] but three notes:
>>  - Fixing something like this takes time, you need to assign the budget
>> for it (which means it has to be done during the annual planning) and if
>> gets approved, you need to start it with the fiscal year (meaning July
>> 2022) and then hire (meaning, write JD, do recruitment, interview lots of
>> people, get them hired) which can take from several months to years. Once
>> they are hired, you need to onboard them and let them learn about our
>> technical infrastructure which takes at least two good months. Software
>> engineering is not magic, it takes time, blood and sweat. [2]
>>  - Making another team focus on multimedia requires changes in planning,
>> budget, OKR, etc. etc. Are we sure moving the focus of teams is a good
>> idea? Most teams are already focusing on vital parts of wikimedia and
>> changing the focus will turn this into a whack-a-mole game where we fix
>> multimedia but now we have critical issues in security or performance.
>>  - Voting Wishlist survey is a good band-aid in the meantime. To at least
>> address the worst parts for now.
>>
>> I don't understand your point tbh, either you think it's a good idea to
>> make requests for improvements in multimedia in the wishlist survey or you
>> think it's not. If you think it's not, then it's offtopic to this thread.
>>
>> [1]
>> https://lists.wikimedia.org/hyperkitty/list/wikimedi...@lists.wikimedia.org/message/WMPZHMXSLQJ6GONAVTFLDFFMPNJDVORS/
>> [2] There is a classic book in this topic called "The Mythical Man-month"
>>
>> On Wed, Dec 29, 2021 at 11:41 AM Gnangarra  wrote:
>>
>>> we have to vote for regular maintenance and support for
>>> essential functions like uploading files which is the core mission of
>>> Wikimedia Commons
>>>
>> ___
> Commons-l mailing list -- common...@lists.wikimedia.org
> To unsubscribe send an email to commons-l-le...@lists.wikimedia.org
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Re: [Wikitech-l] Startup module size and ResourceLoader registry overhead

[Brion Vibber]

TimedMediaHandler is particularly bad at this right now because it's still
in the middle of a switch from an old set of frontend modules to a new one,
so we have to register both. :) I'll make sure we chop the old ones out
soon...

-- brion

On Mon, Sep 16, 2019 at 2:11 PM Amir Sarabadani  wrote:

> Hello,
> Startup module, is the Javascript code [1] that is being served at almost
> every request to Wikimedia as part of  (non-blocking though) to load
> other RL modules. It does important things for example it checks if the
> requested modules already exist with the given hash in the local storage
> cache.
>
> Because of the given reasons, the startup module needs and includes list of
> all registered modules with their version hash in the code. As the result
> if you register a module, even if you don't load it, it adds a rather small
> overhead (around 19 bytes after minification and compression) to every
> request to any wiki the code is enabled. So if you register a new module in
> core or one of the extensions that is deployed in all wikis (CX,
> WikibaseClient, etc.), it's going to be added to every page request,
> meaning 30 GBs more traffic every day to our users. Even if you don't use
> it anywhere.
>
> If you're adding a feature, 30GB/day is acceptable but sometimes developers
> use Resource loader for dependency management or class loading (yours truly
> used to do that) and introduce 20 modules instead of one and each one
> causing an extra 600 GB/day. The big overhead for our users is bad for
> three reasons: 1- RL has to handle the dependency management, making every
> page view slightly slower and worse UX for our users 2- The extra network
> is bad with places without access to broadband connection, where Wikipedia
> is the most likely place that people learn and grow [2] 3- The scale we are
> talking is so massive (petabytes a year) that It has environmental impact.
>
> Let me give you an example, a couple of weeks ago we dropped 137 modules
> from WikibaseClient. After deployment, it dropped 4TB/day from our network
> (= 1.5 PB/year) and synthetic data shows, in an average case we drop 40 ms
> from every response time [3].
>
> We now have a dashboard to track size of size of RL registry [4] plus a
> weekly metrics for changes [5][6]
>
> If you're looking for ways to help. I wrote a tool [7] to do some graph
> analysis and it gives list of extensions that has modules that can be
> merged. The extensions that according to the analysis (that can have false
> positives) can get better are TimedMediaHandler, PageTriage, Graph,
> RevisionSlider, CodeMirror, Citoid, TemplateData, TwoColConflict,
> Collection, CentralNotice, AdvancedSearch, 3D, MobileFrontend and many more
> including some bits and pieces in core. I put the graphs of modules that
> can be merged at [8] and I invite you to have fun with those modules.
> Modules can be merged using package modules [9]
>
> Most of the is work done by the performance team [10] and volunteers and
> developers in lots of teams. I joined the party later as volunteer/WMDE
> staff and I'm sharing mostly the results and writing long emails. Big kudos
> to Krinkle, James Forrester, Santhosh Thottingal, Jon Robson, Alaa Sarhaan,
> Alexandros Kosiaris, and so many others that helped and I forgot.
>
> [1] An example from English Wikipedia:
>
> https://en.wikipedia.org/w/load.php?lang=en=startup=scripts=1=vector=true
> [2] https://arxiv.org/abs/1812.00474
> [3] https://phabricator.wikimedia.org/T203696#5387672
> [4] https://grafana.wikimedia.org/d/BvWJlaDWk/startup-module-size?orgId=1
> [5] https://gist.github.com/Krinkle/f76229f512fead79fb4868824b5bee07
> [6]
>
> https://docs.google.com/document/d/1SESOADAH9phJTeLo4lqipAjYUMaLpGsQTAUqdgyZb4U
> [7] https://phabricator.wikimedia.org/T232728
> [8] https://phabricator.wikimedia.org/T233048
> [9] https://www.mediawiki.org/wiki/ResourceLoader/Package_modules
> [10] https://phabricator.wikimedia.org/T202154
>
> Best
> --
> Amir (he/him)
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit Vandalism

[Brion Vibber]

Just a quick note: incident response is active and ongoing; this is being
worked on.

-- brion

On Mon, Mar 18, 2019, 10:54 PM Eran Rosenthal  wrote:

> I couldn't catch anyone in the IRC so this is also tracked in:
> https://phabricator.wikimedia.org/T218636
>
> On Tue, Mar 19, 2019 at 7:40 AM Jay prakash <0freerunn...@gmail.com>
> wrote:
>
> > Hi please stop User Mill on Gerrit. He/She vandalised my and my Tech's
> > patches.
> >
> > User:Jayprakash12345
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Commons-l] Sunsetting VP8 version of WebM for video playback

[Brion Vibber]

On Mon, Feb 4, 2019, 2:26 PM Gergo Tisza  On Mon, Jan 28, 2019 at 12:22 PM Brion Vibber via Commons-l <
> common...@lists.wikimedia.org> wrote:
>
>> IE 11 users will receive low-resolution, slow JavaScript-based video
>> playback instead. If you find this is troublesome, the recommended solution
>> is to switch to any other browser.
>>
>
> Will they get a warning that they should try another browser? This seems
> like a sensible limitation, as long as the user has a chance of figuring
> out why they are being limited.
>

Yeah, I could have it detect sub-par JS playback performance and display an
explanation in an overlay recommending a more modern browser. Will file a
task to improve this experience...

-- brion

>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] IE 6/7 MIME type sniffing checks on uploads - is it time to retire them?

[Brion Vibber]

On Mon, Jan 28, 2019 at 10:58 PM Kunal Mehta  wrote:

> Tim wrote a nice blog post about how he reverse-engineered this:
> .
>
> I don't have any comments on whether it's still needed, but if it's
> determined that MediaWiki can drop the checks, I'd like to see it
> turned into a PHP library...mostly because it's some neat code.
>

Heck yes. :)

My provisional patch is still keeping the code, but using it more
conservatively:
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/487527

The exact IE-based heuristics are still checked but will trip only on files
matching the exact conditions that would affect old IE versions. Most
uploaded JPEG or PNG files with HTML-ish links in EXIF metadata should not
trigger it, as the tag strings won't appear in the first 256 bytes of the
file.

The less-exact heuristics (held over from before Tim's addition of the
reverse-engineered exact checks) are now less overbearing, and should no
longer trigger on https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] IE 6/7 MIME type sniffing checks on uploads - is it time to retire them?

[Brion Vibber]

There's been some comments on some old tasks such as T27707
 about problems with uploading
files that include text metadata that looks like HTML elements.

Years ago, we added security checks for IE 5/6/7 to work around IE's mime
type sniffing: if you went to view a .png file directly in IE (as opposed
to in an ) the browser would check the first few bytes of the file to
detect its type, overriding the HTTP Content-Type header. HTML would be
detected with a higher priority than the actual image formats, making it
possible to create an actual .png image which when viewed as an image
looked like an image, but when viewed as a web page was interpreted as
HTML, including any embedded JavaScript.

(This was defense in depth in addition to hosting files on a separate
domain; among other things, we sometimes serve files out from the main
domain when dealing with archived (deleted) versions, and third-party
installs are not guaranteed to have a second domain.)

Browsers have moved on, but the code remains and it trips up legitimate
files containing links in metadata, or sometimes just random compressed
data that looks like an element!

I've done a quick research check on feasibility:
* IE 6 and earlier can no longer access Wikimedia sites due to lack of SNI
and TLS 1.0 or later
* IE 7 on Windows XP can no longer access Wikimedia sites due to lack of SNI
* IE 7 on Windows Vista **can** access Wikimedia sites.
* IE 8 and higher support X-Content-Options: nosniff to disable sniffing,
which we already use on all MediaWiki requests.

At some point Microsoft dropped the sniffing, but I'm not sure if it was a
later IE version or an Edge version. No other browsers in reasonably
current versions seem to have this problem.

So the only remaining browser version that might be affected is IE 7 on
Windows Vista, which supports SNI and TLS 1.0. It might or might not still
work once we drop TLS 1.0 some time in the future. (Per our TLS dashboard

about
1.2% of our connections still use TLS 1.0, but this isn't broken down
between logged-in-user views and anon views.)

Open questions:
* Should we drop the anti-sniff checks on upload?
* If we do, should we forbid logins with IE 7, or something else to protect
the occasional IE 7 logged-in user from a hypothetical targeted drive-by
attack? (Is it actually worth doing work and testing it for this?)
* Should we add X-Content-Options: nosniff on files served from
upload.wikimedia.org too?

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Sunsetting VP8 version of WebM for video playback

[Brion Vibber]

A few months ago I switched our TimedMediaHandler's config to support the
newer, more bandwidth-efficient VP9/Opus variant of WebM and to use these
preferably over the older VP8/Vorbis version when creating scaled,
playback-ready derivatives.

(This does not affect upload support -- you may continue to upload video
files in WebM VP9, WebM VP8, or Ogg Theora formats, with either Vorbis or
Opus audio.)

Conversions of existing files on Commons ran in the background for some
weeks, finishing in November. I'm now running a final pass for
high-resolution files and any files on other wikis that didn't get a
conversion yet, in preparation for removing the VP8 derivatives in the next
couple of weeks to free storage space.

This should have relatively little visible effect for users, unless someone
is relying on the particular derivative files with extensions like
".360p.webm"; the new versions are named like ".360p.vp9.webm".

Note that IE 11 users using the "WebM Media Foundation Components for IE
" will not be able to play back the new
VP9/Opus files natively, as this driver has never been updated for VP9 or
Opus.  IE 11 users will receive low-resolution, slow JavaScript-based video
playback instead. If you find this is troublesome, the recommended solution
is to switch to any other browser.

Third-party MediaWiki + TimedMediaHandler users should be aware that the
defaults are changing, and in future the VP8-specific support and code
paths may be removed. TimedMediaHandler will probably change a lot in the
coming months with upcoming WebVTT subtitle format, a streamlined
videojs-based player, and hopefully more!

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Reviving 'EmbedScript' extension for sandboxed JavaScript widgets

[Brion Vibber]

Hey all!

I'm reviving an old project to embed sandboxed HTML/JavaScript "widgets"
into wiki pages as a click-to-play media type, using modern browsers'
 sandbox and Content-Security-Policy restrictions.

Intro and detail notes which I'll keep updating:
https://www.mediawiki.org/wiki/User:Brion_VIBBER/EmbedScript_2019

I hope to extend it with a headless "plugin" mode which allows less-trusted
user-written code to interact safely with fully-trusted host APIs, and a
dependency system to let common library modules, string localizations,
image files from Commons, and data from Wikidata be bundled up and used
safely, without cross-site data exposure.

I'm hoping to solicit some more feedback while I'm in the prototyping
stage, with an eye towards issues we'll need to resolve before it reaches a
productizable stage we could seriously deploy.

Open questions include:

* Can we really replace some user scripts and gadgets with a split-trust
model, and which ones are good ones to start experimenting with?
* What should a user-permissions UX look like for plugins? What threat
models are not examined yet?
* What kind of comment / code review system is needed?
* What about patches, and forks, and copies and centralization? what's the
best Commons-centric or alternate model that will prevent fragmentation of
code?
* How should libraries / dependencies work?
* How should localization work?
* How much coupling to MediaWiki is desired/required?
* How to implement mobile app and offline support?

Feel free to poke me directly or on the wiki talk page with
questions/comments/ideas. Love it? Hate it? Great! Let me know. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Just an update -- background transcoding has been restarted now that we're
switched back to the Virginia data center where the majority of encoders
are.

Converting the lat 1/3 of videos to VP9 should be finished by the end of
November, approximately. At that point we'll probably start removing the
old VP8 versions to free up space.

-- brion

On Mon, Sep 10, 2018 at 12:05 PM Brion Vibber  wrote:

> These conversions have been running in the background, at a moderate load
> to ensure we don't slow down newer uploads much, and are up to the "P"s
> alphabetically so we're on schedule. :)
>
> I've temporarily stopped the batch process pending the upcoming datacenter
> switchover test, and will continue once it's switched.
>
> -- brion
>
> On Wed, Aug 8, 2018 at 2:06 PM Brion Vibber  wrote:
>
>> Quick update... no major problems noticed so far. One configuration
>> change I made was to re-enable VP8 transcodes for new files, since
>> disabling them completely lead to the existing ones not being used. Later
>> in the transition, or afterwards, we'll clean up the now-unused VP8
>> transcodes.
>>
>> If I'm reading numbers in
>> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
>> transition/reports right, almost 5% of files by count or 12% by duration
>> have been converted so far.
>>
>> That leads to a remaining duration somewhere between 9 weeks (assuming 9
>> days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
>> actual factors like whether the remaining files are the same mix of
>> resolutions, and how we adjust the load balance on the job queue.
>>
>> -- brion
>>
>> On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber 
>> wrote:
>>
>>> Batch process is continuing... over 2200 source videos have been
>>> compressed to VP9 so far, of the 120k+ total in the system.
>>>
>>> Seeing big improvements in overall compression, though it's hard to tell
>>> how representative the subset of conversions done so far is. I'll post
>>> occasional updates to the transcode report charts at:
>>>
>>> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports
>>>
>>> In the histograms you can see that not only is the average bitrate down
>>> for VP9 vs VP8, but there's a larger "spread" between low and high
>>> bandwidth in the VP9 versions thanks to the constrained-quality
>>> configuration vs the old fixed bitrate target. This allows files that
>>> compress well to use fewer bits, while those that have high frame rates or
>>> lots of detail/motion use more bits to encode higher quality than they did
>>> before.
>>>
>>> It will take at least a few weeks to recompress everything, and we may
>>> or may not want to increase the number of simultaneous jobs (so far the
>>> servers are running beautifully, but are a bit under-loaded).
>>>
>>> -- brion
>>>
>>>
>>> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber 
>>> wrote:
>>>
>>>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>>>> videos should start encoding as VP9 starting now.
>>>>
>>>> I'll start the backfill batch process tonight or tomorrow, and will
>>>> likely stop and restart it a few times over the coming weeks if anything
>>>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>>>> IRC if any issues come up with the new conversions, or with old files!
>>>> (Existing VP8 files will be left as-is for now until we're sure
>>>> everything's up to snuff.)
>>>>
>>>> Big thanks to everybody who's helped prepping this, with libvpx and
>>>> ffmpeg deployments, with the patches and review, and with final deployment
>>>> which always takes longer than you expect. :) This'll be a nice improvement
>>>> to our video output for now, and lays a lot of groundwork for next steps.
>>>>
>>>> -- brion
>>>>
>>>>
>>>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber 
>>>> wrote:
>>>>
>>>>> Oh and one more thing!
>>>>>
>>>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>>>> resolutions, which people can manually bump up to when watching videos 
>>>>> with
>>>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>>>> only a small fraction of input files are 4K so should 

Re: [Wikitech-l] problematic use of "Declined" in Phabricator

[Brion Vibber]

*nods* I think the root problem is that the phabricator task system does
double duty as both an *issue reporting system* for users and a *task
tracker* for devs.

An issue reporting system should capture all actual problems and all actual
suggestions, and is meant to provide visibility for the devs into the world
of users. A task tracker should capture only things that are, or are
planned to be, worked on and is a work planning tool for the devs.
Secondarily if open, the task tracker provides visibility for the users
into the world of devs.

This mixup of concerns is endemic in open source software development,
unfortunately, and leads to exactly the sorts of conflicts you mention.

One way to handle this in a mixed single tracker environment is to use a
state marker such as a backlog column in a workboard -- don't decline, move
it to the "backlog" or "someday" column. Another is to use separate project
tags for general issues and specific work efforts. Put it in the general
project for issues, copy it to the work project if it's being tracked in
your work group.

-- brion







On Tue, Oct 2, 2018, 9:31 AM Amir E. Aharoni 
wrote:

> Hi,
>
> I sometimes see WMF developers and product managers marking tasks as
> "Declined" with comments such as these:
> * "No resources for it in (team name)"
> * "We won't have the resources to work on this anytime soon."
> * "I do not plan to work on this any time soon."
>
> Can we perhaps agree that the "Declined" status shouldn't be used like
> this?
>
> "Declined" should be valid when:
> * The component is no longer maintained (this is often done as
> mass-declining).
> * A product manager, a developer, or any other sensible stakeholder thinks
> that doing the task as proposed is a bad idea. There are also variants of
> this:
> * The person who filed the tasks misunderstood what the software component
> is supposed to do and had wrong expectations.
> * The person who filed the tasks identified a real problem, but another
> task proposes a better solution.
>
> It's quite possible that some people will disagree with the decision to
> mark a particular task as "Declined", but the reasons above are legitimate
> explanations.
>
> However, if the task suggests a valid idea, but the reason for declining is
> that a team or a person doesn't plan to work on it because of lack of
> resources or different near-term priorities, it's quite problematic to mark
> it as Declined.
>
> It's possible to reopen tasks, of course, but nevertheless "Declined" gives
> a somewhat permanent feeling, and may cause good ideas to get lost.
>
> So can we perhaps decide that such tasks should just remain Open? Maybe
> with a Lowest priority, maybe in something like a "Freezer" or "Long term"
> or "Volunteer needed" column on a project workboard, but nevertheless Open?
>
> --
> Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
> http://aharoni.wordpress.com
> ‪“We're living in pieces,
> I want to live in peace.” – T. Moore‬
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

These conversions have been running in the background, at a moderate load
to ensure we don't slow down newer uploads much, and are up to the "P"s
alphabetically so we're on schedule. :)

I've temporarily stopped the batch process pending the upcoming datacenter
switchover test, and will continue once it's switched.

-- brion

On Wed, Aug 8, 2018 at 2:06 PM Brion Vibber  wrote:

> Quick update... no major problems noticed so far. One configuration change
> I made was to re-enable VP8 transcodes for new files, since disabling them
> completely lead to the existing ones not being used. Later in the
> transition, or afterwards, we'll clean up the now-unused VP8 transcodes.
>
> If I'm reading numbers in
> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
> transition/reports right, almost 5% of files by count or 12% by duration
> have been converted so far.
>
> That leads to a remaining duration somewhere between 9 weeks (assuming 9
> days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
> actual factors like whether the remaining files are the same mix of
> resolutions, and how we adjust the load balance on the job queue.
>
> -- brion
>
> On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber 
> wrote:
>
>> Batch process is continuing... over 2200 source videos have been
>> compressed to VP9 so far, of the 120k+ total in the system.
>>
>> Seeing big improvements in overall compression, though it's hard to tell
>> how representative the subset of conversions done so far is. I'll post
>> occasional updates to the transcode report charts at:
>>
>> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports
>>
>> In the histograms you can see that not only is the average bitrate down
>> for VP9 vs VP8, but there's a larger "spread" between low and high
>> bandwidth in the VP9 versions thanks to the constrained-quality
>> configuration vs the old fixed bitrate target. This allows files that
>> compress well to use fewer bits, while those that have high frame rates or
>> lots of detail/motion use more bits to encode higher quality than they did
>> before.
>>
>> It will take at least a few weeks to recompress everything, and we may or
>> may not want to increase the number of simultaneous jobs (so far the
>> servers are running beautifully, but are a bit under-loaded).
>>
>> -- brion
>>
>>
>> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber 
>> wrote:
>>
>>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>>> videos should start encoding as VP9 starting now.
>>>
>>> I'll start the backfill batch process tonight or tomorrow, and will
>>> likely stop and restart it a few times over the coming weeks if anything
>>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>>> IRC if any issues come up with the new conversions, or with old files!
>>> (Existing VP8 files will be left as-is for now until we're sure
>>> everything's up to snuff.)
>>>
>>> Big thanks to everybody who's helped prepping this, with libvpx and
>>> ffmpeg deployments, with the patches and review, and with final deployment
>>> which always takes longer than you expect. :) This'll be a nice improvement
>>> to our video output for now, and lays a lot of groundwork for next steps.
>>>
>>> -- brion
>>>
>>>
>>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber 
>>> wrote:
>>>
>>>> Oh and one more thing!
>>>>
>>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>>> resolutions, which people can manually bump up to when watching videos with
>>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>>> only a small fraction of input files are 4K so should not significantly
>>>> increase disk space projections for now.
>>>>
>>>> These can take a long time to compress, so if we find it's problematic
>>>> we'll turn them back off until the jobs can be split into tiny chunks
>>>> (future work planned!), but it works in my testing and shouldn't clog the
>>>> servers now that we have more available.
>>>>
>>>> (Note that the ogv.js player shim for Safari will not handle
>>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>>> Chromium-based browser.)
>>>>
>>>> -- brion
>>>>
>

[Wikitech-l] MediaWiki-Vagrant users beware: vagrant 2.1.4 breakage

[Brion Vibber]

HashiCorp has released a 2.1.4 update to Vagrant -- beware that this
version changes the order of plugin loading in a way that breaks on
MediaWiki-Vagrant's configuration file.

It should be fixable on our end, but for now either stick with the version
of Vagrant you have installed, or downgrade back to 2.1.2 if you already
updated.

https://phabricator.wikimedia.org/T203289

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ForkableMaintenance: thoughts and help with tests/code coverage

[Brion Vibber]

On Tue, Aug 14, 2018 at 8:34 AM Brion Vibber  wrote:

> On Tue, Aug 14, 2018, 3:39 AM Aleksey Bekh-Ivanov <
> aleksey.bekh-iva...@wikimedia.de> wrote:
>
>> Hi Brion.
>>
>> I might be a bit late, but I just wonder why did you decide to go with
>> forking instead of Pthreads PHP extension?
>>
>
> ForkController and OrderedStreamingForkController were already present and
> using pcntl_fork; also I didn't know PHP has a pthreads extension now. :)
> Worth looking at...
>
>
>> AFAIK, Mediawiki should support not only *nix platforms, but Windows as
>> well, and `pcntl_fork()` does not work on Windows.
>>
>
> If no threads are specified it'll run in-process which will still work,
> but of course you get no additional processes then.
>

I took a quick look at the PHP pthreads extension; it looks interesting,
but looks like it might be trickier to integrate.

Not all of the global environment is available to threads -- only classes
that extend a special Threaded type -- so I think threads would either end
up without the MediaWiki environment, severely limiting what they can do,
or they would have to reinitialize MediaWiki locally.

Also it doesn't seem to be packaged in Debian, so it would be extra work
for us to adopt and package it for production use, and it's less likely to
be present by default in other peoples' operating environments.

If re-initializing the application state is necessary in the threads anyway
without retooling the entire application, it might be easier to use a
separate-process model where the maintenance script is re-invoked with
wfShellExec() with a special child-process mode, and data routed over
stdin/stdout.

In which case it might be more consistent to do that than using
pcntl_fork() to begin with... but I'm not sure how to integrate that into
phpunit tests. :)


>
> PS: Couldn't find related ticket. Can you post a link to it?
>>
>
> No ticket currently, though I can open one. :)
>

Opened as https://phabricator.wikimedia.org/T201970 -- if there's more
interest in getting this right I can make this an RFC. (I had an inquiry
about thinking about ways to do multiprocess work from web requests too,
which might not work with pcntl_fork().)

Additional thoughts welcome on pcntl_fork() vs separate process launch vs
something else. :D

-- brion

>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ForkableMaintenance: thoughts and help with tests/code coverage

[Brion Vibber]

On Tue, Aug 14, 2018, 3:39 AM Aleksey Bekh-Ivanov <
aleksey.bekh-iva...@wikimedia.de> wrote:

> Hi Brion.
>
> I might be a bit late, but I just wonder why did you decide to go with
> forking instead of Pthreads PHP extension?
>

ForkController and OrderedStreamingForkController were already present and
using pcntl_fork; also I didn't know PHP has a pthreads extension now. :)
Worth looking at...


> AFAIK, Mediawiki should support not only *nix platforms, but Windows as
> well, and `pcntl_fork()` does not work on Windows.
>

If no threads are specified it'll run in-process which will still work, but
of course you get no additional processes then.


> PS: Couldn't find related ticket. Can you post a link to it?
>

No ticket currently, though I can open one. :)

-- brion


> Links:
> https://secure.php.net/manual/en/book.pthreads.php
> https://github.com/krakjoe/pthreads
>
> Aleksey, Wikimedia Deutschland
>
> On 13 August 2018 at 02:51, Brion Vibber  wrote:
>
> > Thanks, looks like I misinterpreted the report output. :)
> >
> > I think I can add a test case for ParallelMaintenance which should make
> the
> > warning go away.
> >
> > -- brion
> >
> > On Sun, Aug 12, 2018, 1:51 PM Kunal Mehta 
> wrote:
> >
> > > -BEGIN PGP SIGNED MESSAGE-
> > > Hash: SHA512
> > >
> > > Hi,
> > >
> > > On 08/11/2018 06:48 PM, Brion Vibber wrote:
> > > > Second, probably related to that I'm seeing a failure in the code
> > > > coverage calculations -- it's seeing some increased coverage on the
> > > > parent process at least but seems to think it's returning a
> > > > non-zero exit code somewhere, which marks the whole operation as a
> > > > failure:
> > > >
> > > >
> https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-pa
> > > tch-docker/460/console
> > > <https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-
> > patch-docker/460/console>
> > >
> > > I
> > > >
> > > think your test and the tool are working properly, there just is an
> > > actual coverage drop:
> > >
> > > +-+++
> > > | Filename| Old %  | New %  |
> > > +-+++
> > > | includes/QueueingForkController.php | 0  |  07.92 |
> > > | maintenance/Maintenance.php |  20.51 |  19.83 |
> > > +-+++
> > >
> > > If you look at the HTML report[1], all the new lines added to
> > > Maintenance.php are not covered by tests, which is decreasing coverage.
> > >
> > > The tool currently reports a coverage drop if it drops in any file,
> > > which isn't necessary ideal, see [2].
> > >
> > > [1]
> > >
> https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-patc
> > > h-docker/460/artifact/log/coverage.html
> > > <https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-
> > patch-docker/460/artifact/log/coverage.html>
> > > [2] https://phabricator.wikimedia.org/T188687
> > >
> > > - -- Legoktm
> > > -BEGIN PGP SIGNATURE-
> > >
> > > iQIzBAEBCgAdFiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAltwnZ4ACgkQUvyOe+23
> > > /KIm8w//WZVvuCCxHlW2CmbKtw/hjiBCxTUsmFAdb4QCco2nJe1qcSKtU9tBWq0n
> > > HrT32rEK06sYSPFrcHE6KYCHYtaLAGn8zcpXTCnB15mq1c/yrkwNucXwpBbbs28b
> > > 776EjNzTnU8UbTP0y9qt+Z3g1rRAjFjbXSqbh/3Vi9nQDlgS+cCgMwudZ+INzCeV
> > > L0O+JKZKmfAswcSZbSVkWFDBQMZulhlP4ztS0hTYyixnGTl5z2Cc3c7F2OvjlcIx
> > > lyGZkh1544X6hG7t9t5o35Tjbwt/Y5de617QiiN6dvFO6OrxD/mNs17kp302WJS7
> > > FjcPjFXvSsOdobG0Ff/cg8/cy1m5Ek6fctw1cCMWnYruy7rcYvt9QYz8NRxaSIES
> > > PdYmL8AiuY0173AKTTMgmOxjg0phY6Mrf7d8eo81zRwkENBjzwut2gEF6s4xeR6I
> > > jnKdqpHta6HWs3wF7+dTNxH8v5f7TRGVkz1PwacdbHZBj8PEUAge7789f2qqzByb
> > > V2P5tr/nMCTrIoc+iPsjif1AbQsbLk+dKs1BDHxymChnZa0gTIbUGnTriqKn4xIo
> > > qkZyflm66OHa+R6C3hcs5+OTfVnx7Sqqcmk3b7vC3N5ydlEwUXJSI9PrOC6xr77s
> > > ltUPh/8hsA3TLJ6CHyCwgnZtyZOL6XczysOHiWpBeH1wWhl+gwQ=
> > > =DSVw
> > > -END PGP SIGNATURE-
> > >
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ForkableMaintenance: thoughts and help with tests/code coverage

[Brion Vibber]

Thanks, looks like I misinterpreted the report output. :)

I think I can add a test case for ParallelMaintenance which should make the
warning go away.

-- brion

On Sun, Aug 12, 2018, 1:51 PM Kunal Mehta  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hi,
>
> On 08/11/2018 06:48 PM, Brion Vibber wrote:
> > Second, probably related to that I'm seeing a failure in the code
> > coverage calculations -- it's seeing some increased coverage on the
> > parent process at least but seems to think it's returning a
> > non-zero exit code somewhere, which marks the whole operation as a
> > failure:
> >
> > https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-pa
> tch-docker/460/console
> <https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-patch-docker/460/console>
>
> I
> >
> think your test and the tool are working properly, there just is an
> actual coverage drop:
>
> +-+++
> | Filename| Old %  | New %  |
> +-+++
> | includes/QueueingForkController.php | 0  |  07.92 |
> | maintenance/Maintenance.php |  20.51 |  19.83 |
> +-+++
>
> If you look at the HTML report[1], all the new lines added to
> Maintenance.php are not covered by tests, which is decreasing coverage.
>
> The tool currently reports a coverage drop if it drops in any file,
> which isn't necessary ideal, see [2].
>
> [1]
> https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-patc
> h-docker/460/artifact/log/coverage.html
> <https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-patch-docker/460/artifact/log/coverage.html>
> [2] https://phabricator.wikimedia.org/T188687
>
> - -- Legoktm
> -BEGIN PGP SIGNATURE-
>
> iQIzBAEBCgAdFiEE+h6fmkHn9DUCyl1jUvyOe+23/KIFAltwnZ4ACgkQUvyOe+23
> /KIm8w//WZVvuCCxHlW2CmbKtw/hjiBCxTUsmFAdb4QCco2nJe1qcSKtU9tBWq0n
> HrT32rEK06sYSPFrcHE6KYCHYtaLAGn8zcpXTCnB15mq1c/yrkwNucXwpBbbs28b
> 776EjNzTnU8UbTP0y9qt+Z3g1rRAjFjbXSqbh/3Vi9nQDlgS+cCgMwudZ+INzCeV
> L0O+JKZKmfAswcSZbSVkWFDBQMZulhlP4ztS0hTYyixnGTl5z2Cc3c7F2OvjlcIx
> lyGZkh1544X6hG7t9t5o35Tjbwt/Y5de617QiiN6dvFO6OrxD/mNs17kp302WJS7
> FjcPjFXvSsOdobG0Ff/cg8/cy1m5Ek6fctw1cCMWnYruy7rcYvt9QYz8NRxaSIES
> PdYmL8AiuY0173AKTTMgmOxjg0phY6Mrf7d8eo81zRwkENBjzwut2gEF6s4xeR6I
> jnKdqpHta6HWs3wF7+dTNxH8v5f7TRGVkz1PwacdbHZBj8PEUAge7789f2qqzByb
> V2P5tr/nMCTrIoc+iPsjif1AbQsbLk+dKs1BDHxymChnZa0gTIbUGnTriqKn4xIo
> qkZyflm66OHa+R6C3hcs5+OTfVnx7Sqqcmk3b7vC3N5ydlEwUXJSI9PrOC6xr77s
> ltUPh/8hsA3TLJ6CHyCwgnZtyZOL6XczysOHiWpBeH1wWhl+gwQ=
> =DSVw
> -END PGP SIGNATURE-
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ForkableMaintenance: thoughts and help with tests/code coverage

[Brion Vibber]

(I've made both changes on the PR.)

-- brion

On Sun, Aug 12, 2018 at 7:54 AM Brion Vibber  wrote:

>
> On Sun, Aug 12, 2018, 2:49 AM Aryeh Gregor  wrote:
>
>>
>> For what it's worth, when I saw ForkableMaintenance I thought of
>> forking an open-source project, not Unix fork().  Something like
>> ParallelMaintenance or ParallelizableMaintenance would better suggest
>> the desired meaning for me.
>>
>
> I like it... I'll reserve the Fork terminology for the low level
> controllers which deal with the processes and change the maintenance
> wrapper to ParallelMaintenance.
>
> Any preferences on --fork=N vs --threads=N for the cli option? I'm leaning
> to changing it to --threads.
>
> -- brion
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ForkableMaintenance: thoughts and help with tests/code coverage

[Brion Vibber]

On Sun, Aug 12, 2018, 2:49 AM Aryeh Gregor  wrote:

>
> For what it's worth, when I saw ForkableMaintenance I thought of
> forking an open-source project, not Unix fork().  Something like
> ParallelMaintenance or ParallelizableMaintenance would better suggest
> the desired meaning for me.
>

I like it... I'll reserve the Fork terminology for the low level
controllers which deal with the processes and change the maintenance
wrapper to ParallelMaintenance.

Any preferences on --fork=N vs --threads=N for the cli option? I'm leaning
to changing it to --threads.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] ForkableMaintenance: thoughts and help with tests/code coverage

[Brion Vibber]

Hey all!

While working on some maintenance scripts for TimedMediaHandler I've been
trying to make it easier to do scripts that use multiple parallel processes
to run through a large input set faster.

My proposal is a ForkableMaintenance class, with an underlying
QueueingForkController which is a refactoring of the
OrderedStreamingForkController used by (at least) some CirrusSearch
maintenance scripts.

Patch in progress:
https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/451099/

The expected use case is a relatively long loop of reading input data
interleaved with running CPU-intensive or DB-intensive jobs, where the
individual jobs are independent and order of input is not strongly coupled.
(Ordering of _running_ of items on the child processes is not guaranteed,
but the order of result processing is guaranteed to be in the same order as
input, making output across runs predictable for idempotent processing.)

A simple ForkableMaintenance script might look like:

class Foo extends ForkableMaintenance {
  // Handle any input on the parent thread, and
  // pass any data as JSON-serializable form into
  // the queue() method, where it gets funneled into
  // a child process.
  public function loop() {
 for ( $i = 0; $i < 1000; $i++) {
   $this->queue( $i );
 }
  }

  // On the child process, receives the queued value
  // via JSON encode/decode. Here it's a number.
  public function work( $count ) {
return str_repeat( '*', $count );
  }

  // On the parent thread, receives the work() return value
  // via JSON encode/decode. Here it's a string.
  public function result( $data ) {
$this->output( $data . "\n" );
  }
}

Because data is serialized as JSON and sent over a pipe, you can't send
live objects like Titles or Pages or Files, but you can send arrays or
associative arrays with fairly complex data.

There is a little per-job overhead and multiple threads can cause more
contention on the database server etc, but it scales well on the subtitle
format conversion script I'm testing with, which is a little DB loading and
some CPU work. On an 8-core/16-thread test server:

threads runtime (s) speedup
0 320 n/a
1 324 0.987654321
2 183 1.74863388
4 105 3.047619048
8 66 4.848484848
16 58 5.517241379

I've added a phpunit test case for OrderedStreamingForkController to make
sure I don't regress something used by other teams, but noticed a couple
problems with using this fork stuff in the test runners.

First, doing pcntl_fork() inside a phpunit test case has some potential
side effects, since there's a lot of tear-down work done in destructors
even if you call exit() after processing completes. As a workaround, when
I'm having the child procs send a SIGKILL to themselves to terminate
immediately without running test-case destructors.

Second, probably related to that I'm seeing a failure in the code coverage
calculations -- it's seeing some increased coverage on the parent process
at least but seems to think it's returning a non-zero exit code somewhere,
which marks the whole operation as a failure:

https://integration.wikimedia.org/ci/job/mediawiki-phpunit-coverage-patch-docker/460/console

Worst case, can probably exclude these from some automated tests but that
always seems like a bad idea. :D

If anybody else is using, or thinking about using, ForkController and its
friends and wants to help polish this up, give a shout!

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] (no subject)

[Brion Vibber]

I wanted to comment quickly on one thing that was called out.

Closing "valid tasks" may be appropriate depending on the task and the
context. Closed is a valid state for a task and may well be most
appropriate.

There is a reason for this, and I want to be clear:

Tasks are not isolated platonic constructs; they are tools for people to
use in their work on software. If a task and the discussion on it is
unconstructive, then closing it sounds fine.


Now that's just off the top of my head; I'm unfamiliar with the particular
case you presumably are citing.

But again I have to stress that this is not a hobby project, this is a
working environment for dozens of people building and maintaining the
software that the Wikimedia community has entrusted them with.

That's not to say volunteers aren't welcome: rather, that's to say that
volunteers are expected to behave themselves just as we are when they
choose to work alongside us.

Not sure about the language references; they don't seem relevant to the
patterns of behavior under discussion.

-- brion

On Wed, Aug 8, 2018, 4:18 PM MZMcBride  wrote:

> Brion Vibber wrote:
> >I would advise you generally to treat wikitech-l like a professional
> >workspace, which it is for those of us who are employees of WMF or WMDE.
>
> I think there's a big schism that you're pointing to here: why do you
> think it's appropriate for you or anyone else to impose your particular
> U.S. workplace standards on a global community of volunteers? For many
> users, wikitech-l, Phabricator, and other venues are specifically not
> workplaces, they're places for technical work on hobby projects.


> >If your corporate HR department would frown at you making a statement
> >about people's character or motivations with derogatory language, think
> >twice about it. Treat people with respect.
>
> Sure, treat people with respect. As a colleague of Greg Varnum, have you
> privately messaged him to let him know that closing valid tasks is
> inappropriate? Have you told him that gaslighting users into thinking that
> an obvious bug is an intentional design choice is unacceptable behavior?
> Have you discussed with Greg V. that un-assigning yourself from a task and
> attempting to orphan it is bad practice?
>
> Or are you simply focused on trying to shame and silence volunteers who
> are critical of Wikimedia Foundation Inc. employees?
>
> Regarding the word fuck generally, I've been here long enough to remember
> commits such as <https://github.com/wikimedia/mediawiki/commit/32936ec8>.
> There are also many commits and tasks that use similar language. As the
> English Wiktionary notes, "what the fuck" is a common interjection:
> <https://en.wiktionary.org/wiki/what_the_fuck#Interjection>. I do not
> think it's a phrase that should be commonly used on Phabricator, but at
> times it can be appropriate, _as your code commit from 2008 notes_, to
> underscore the severity of a particular issue. What Greg did was really
> bad and is compounded, in my opinion, by his continued silence and the
> lack of resolution to the issue of German text appearing on an English
> landing page. Saying what Greg V. did was confusing and bad, even
> forcefully, is not the real issue here.
>
> For what it's worth, here's Daniel using the same language in 2016:
> <https://phabricator.wikimedia.org/T110728#2227182>. And MatmaRex using
> the same language in the same year:
> <https://phabricator.wikimedia.org/T130478>. A quick search of Phabricator
> for "what the fuck", "fuck", or "wtf" shows that none of them are rare.
>
> MZMcBride
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Quick update... no major problems noticed so far. One configuration change
I made was to re-enable VP8 transcodes for new files, since disabling them
completely lead to the existing ones not being used. Later in the
transition, or afterwards, we'll clean up the now-unused VP8 transcodes.

If I'm reading numbers in
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_
transition/reports right, almost 5% of files by count or 12% by duration
have been converted so far.

That leads to a remaining duration somewhere between 9 weeks (assuming 9
days did 12% of stuff) and 24 weeks (9 days did 5% of stuff), depending on
actual factors like whether the remaining files are the same mix of
resolutions, and how we adjust the load balance on the job queue.

-- brion

On Fri, Aug 3, 2018 at 11:22 AM Brion Vibber  wrote:

> Batch process is continuing... over 2200 source videos have been
> compressed to VP9 so far, of the 120k+ total in the system.
>
> Seeing big improvements in overall compression, though it's hard to tell
> how representative the subset of conversions done so far is. I'll post
> occasional updates to the transcode report charts at:
>
> https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports
>
> In the histograms you can see that not only is the average bitrate down
> for VP9 vs VP8, but there's a larger "spread" between low and high
> bandwidth in the VP9 versions thanks to the constrained-quality
> configuration vs the old fixed bitrate target. This allows files that
> compress well to use fewer bits, while those that have high frame rates or
> lots of detail/motion use more bits to encode higher quality than they did
> before.
>
> It will take at least a few weeks to recompress everything, and we may or
> may not want to increase the number of simultaneous jobs (so far the
> servers are running beautifully, but are a bit under-loaded).
>
> -- brion
>
>
> On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber 
> wrote:
>
>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>> videos should start encoding as VP9 starting now.
>>
>> I'll start the backfill batch process tonight or tomorrow, and will
>> likely stop and restart it a few times over the coming weeks if anything
>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>> IRC if any issues come up with the new conversions, or with old files!
>> (Existing VP8 files will be left as-is for now until we're sure
>> everything's up to snuff.)
>>
>> Big thanks to everybody who's helped prepping this, with libvpx and
>> ffmpeg deployments, with the patches and review, and with final deployment
>> which always takes longer than you expect. :) This'll be a nice improvement
>> to our video output for now, and lays a lot of groundwork for next steps.
>>
>> -- brion
>>
>>
>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber 
>> wrote:
>>
>>> Oh and one more thing!
>>>
>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>> resolutions, which people can manually bump up to when watching videos with
>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>> only a small fraction of input files are 4K so should not significantly
>>> increase disk space projections for now.
>>>
>>> These can take a long time to compress, so if we find it's problematic
>>> we'll turn them back off until the jobs can be split into tiny chunks
>>> (future work planned!), but it works in my testing and shouldn't clog the
>>> servers now that we have more available.
>>>
>>> (Note that the ogv.js player shim for Safari will not handle
>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>> Chromium-based browser.)
>>>
>>> -- brion
>>>
>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
>>> wrote:
>>>
>>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>>> quality when needed, and pulling in some other improvements to the config
>>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>>
>>>> If all goes well with the general deployments in the next few days,
>>>> expect the beginning of VP9 rollout starting next week.
>>>>
>>>> Changes since the earlier announcement:
>>>> * the new row-multithreading will be available, which allows higher
>>>> threading usage at all resolutions; encoding times

Re: [Wikitech-l] coc ban

[Brion Vibber]

Oleg -- I interpret that suggestion as "employees of WMF and WMDE have to
accept all ongoing abuse they are given without complaint"; that may not be
what you intended but that's how I read it, and I'd like to unequivocally
*reject that notion*.

WMF and WMDE employees are people performing a job, and deserve a safe work
environment. When it's suggested that long-term abusive behavior against
employees be tolerated because it's not a big deal or it's just volunteers
letting off steam, I have to counter that it *is* a big deal. It promotes
burn-out and harms people both personally and professionally.

Please don't be part of that cycle of abuse and burn-out, all. It's not
cool, it's not productive, it's not funny, and it doesn't help you get what
you wanted done.

-- brion

On Wed, Aug 8, 2018 at 10:43 AM Saint Johann  wrote:

> Sure.
>
> Wikimedia Foundation employees inherently have more privilege and weight
> in MediaWiki developer community than the volunteers do, especially less
> participating ones. Power dynamics of the discussion between a volunteer
> and an employee (and, sometimes even more generally on Phabricator) are
> structured in a way in that more than frequently an end decision will be
> taken not by volunteers or all Wikimedia community, but by employees or
> people that are more well-versed in MediaWiki development spaces (who
> also can happen to be employees).
>
> Code of conduct is important to be enforced, but, in my opinion, there
> should be a difference in how it’s enforced. To volunteers that help the
> movement, there should be no unacceptable language, as it is a way (and
> a purpose of something like code of conduct) to make MediaWiki
> development spaces more welcoming to future volunteers.
>
> However, employees, while in their capacity, should be (in reasonable
> amounts) less guarded against non-constructive criticism, because at
> many times all you can provide to someone’s work decisions could only be
> non-constructive because you know that no minds and hearts will be
> changed by any amount of constructive criticism. I am, of course, not
> talking about any kinds of serious stuff (Jimmy Wales language), but
> more about ‘WTF language’.
>
> Oleg
>
> On 08/08/2018 19:20, Arlo Breault wrote:
> >
> >> On Aug 8, 2018, at 9:42 AM, Saint Johann  wrote:
> >>
> >> especially when said to Wikimedia employees as opposed to volunteers.)
> > Can you elaborate on that?
> >
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] (no subject)

[Brion Vibber]

Hi Dennis.

I would advise you generally to treat wikitech-l like a professional
workspace, which it is for those of us who are employees of WMF or WMDE.

If your corporate HR department would frown at you making a statement about
people's character or motivations with derogatory language, think twice
about it. Treat people with respect.

Thanks!

-- brion


On Wed, Aug 8, 2018 at 7:42 AM Dennis During  wrote:

> What kind of snowflake environment is this? Or do we use the alleged
> presence of snowflakes as a weapon against opposing views?
>
>
> >
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Batch process is continuing... over 2200 source videos have been compressed
to VP9 so far, of the 120k+ total in the system.

Seeing big improvements in overall compression, though it's hard to tell
how representative the subset of conversions done so far is. I'll post
occasional updates to the transcode report charts at:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition/reports

In the histograms you can see that not only is the average bitrate down for
VP9 vs VP8, but there's a larger "spread" between low and high bandwidth in
the VP9 versions thanks to the constrained-quality configuration vs the old
fixed bitrate target. This allows files that compress well to use fewer
bits, while those that have high frame rates or lots of detail/motion use
more bits to encode higher quality than they did before.

It will take at least a few weeks to recompress everything, and we may or
may not want to increase the number of simultaneous jobs (so far the
servers are running beautifully, but are a bit under-loaded).

-- brion


On Mon, Jul 30, 2018 at 5:51 PM Brion Vibber  wrote:

> Configuration has been updated and VP9 mode swapped in. Newly uploaded
> videos should start encoding as VP9 starting now.
>
> I'll start the backfill batch process tonight or tomorrow, and will likely
> stop and restart it a few times over the coming weeks if anything needs
> adjustment. Please file tasks in phabricator and/or give me a ping on IRC
> if any issues come up with the new conversions, or with old files!
> (Existing VP8 files will be left as-is for now until we're sure
> everything's up to snuff.)
>
> Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
> deployments, with the patches and review, and with final deployment which
> always takes longer than you expect. :) This'll be a nice improvement to
> our video output for now, and lays a lot of groundwork for next steps.
>
> -- brion
>
>
> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber 
> wrote:
>
>> Oh and one more thing!
>>
>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>> resolutions, which people can manually bump up to when watching videos with
>> a suitable 4K source on a high-res screen. They use higher data rates, but
>> only a small fraction of input files are 4K so should not significantly
>> increase disk space projections for now.
>>
>> These can take a long time to compress, so if we find it's problematic
>> we'll turn them back off until the jobs can be split into tiny chunks
>> (future work planned!), but it works in my testing and shouldn't clog the
>> servers now that we have more available.
>>
>> (Note that the ogv.js player shim for Safari will not handle
>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>> Chromium-based browser.)
>>
>> -- brion
>>
>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
>> wrote:
>>
>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>> quality when needed, and pulling in some other improvements to the config
>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>
>>> If all goes well with the general deployments in the next few days,
>>> expect the beginning of VP9 rollout starting next week.
>>>
>>> Changes since the earlier announcement:
>>> * the new row-multithreading will be available, which allows higher
>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>> slower instead of 3-4x slower.
>>> * switch to constrained quality with a larger max bitrate: many files
>>> will become significantly smaller in their VP9 versions, but some will
>>> actually increase in exchange for a huge increase in quality -- this is
>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>> didn't compress well at the default low data rates.
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
>>> wrote:
>>>
>>>> Awesome sauce. Thanks Moritz!
>>>>
>>>> -- brion
>>>>
>>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>>> mmuhlenh...@wikimedia.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>>> > Current state on this:
>>>>> >
>>>>> > * still hoping to deploy the libvpx+ffmpeg backport firs

Re: [Wikitech-l] [Multimedia] phone cannot play mobile site video, but standard site video

[Brion Vibber]

Our old video player frontend doesn't quite work right on mobile so we've
been migrating towards a modern one based on video.js, which does work on
mobile and can integrate with our Safari compatibility shim.

It's been blocked on some issues with subtitle compatibility, which I'll be
working on in the next couple weeks as the VP9 conversion is now underway.

Once that's resolved we should be able to deploy the update and have
working video on iOS and better video on Android, with subtitles when
available.

-- brion

On Fri, Aug 3, 2018, 3:56 AM rupert THURNER 
wrote:

> why is it that the excellent player is available on the standard wikipedia
> view, and not on mobile view? is this just my configuration? as eyample:
> * https://en.wikipedia.org/wiki/Polar_orbit
> * https://en.m.wikipedia.org/wiki/Polar_orbit
> the effect is that my mobile can play the standard view video, but not the
> mobile view video.
>
> rupert
>
> ___
> Multimedia mailing list
> multime...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/multimedia
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Configuration has been updated and VP9 mode swapped in. Newly uploaded
videos should start encoding as VP9 starting now.

I'll start the backfill batch process tonight or tomorrow, and will likely
stop and restart it a few times over the coming weeks if anything needs
adjustment. Please file tasks in phabricator and/or give me a ping on IRC
if any issues come up with the new conversions, or with old files!
(Existing VP8 files will be left as-is for now until we're sure
everything's up to snuff.)

Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
deployments, with the patches and review, and with final deployment which
always takes longer than you expect. :) This'll be a nice improvement to
our video output for now, and lays a lot of groundwork for next steps.

-- brion


On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber  wrote:

> Oh and one more thing!
>
> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
> resolutions, which people can manually bump up to when watching videos with
> a suitable 4K source on a high-res screen. They use higher data rates, but
> only a small fraction of input files are 4K so should not significantly
> increase disk space projections for now.
>
> These can take a long time to compress, so if we find it's problematic
> we'll turn them back off until the jobs can be split into tiny chunks
> (future work planned!), but it works in my testing and shouldn't clog the
> servers now that we have more available.
>
> (Note that the ogv.js player shim for Safari will not handle
> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
> iPad; for best results for 4K playback use Firefox, Chrome, or a
> Chromium-based browser.)
>
> -- brion
>
> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
> wrote:
>
>> Ok, after some delay for re-tweaking the encoding settings for higher
>> quality when needed, and pulling in some other improvements to the config
>> system, all related updates to TimedMediaHandler have been merged. :D
>>
>> If all goes well with the general deployments in the next few days,
>> expect the beginning of VP9 rollout starting next week.
>>
>> Changes since the earlier announcement:
>> * the new row-multithreading will be available, which allows higher
>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>> slower instead of 3-4x slower.
>> * switch to constrained quality with a larger max bitrate: many files
>> will become significantly smaller in their VP9 versions, but some will
>> actually increase in exchange for a huge increase in quality -- this is
>> mostly 60fps high-rate files, and those with lots of motion and detail that
>> didn't compress well at the default low data rates.
>>
>> -- brion
>>
>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
>> wrote:
>>
>>> Awesome sauce. Thanks Moritz!
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>> mmuhlenh...@wikimedia.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>>> > Current state on this:
>>>> >
>>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>>>> with
>>>> > best performance; Moritz made a start on libvpx but we still have to
>>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>>> way to
>>>> > 3.4)
>>>>
>>>> I've completed this today. We now have a separate repository component
>>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>>>> (thus allowing us to follow the ffmpeg security updates released in
>>>> Debian
>>>> with a local rebuild) with backported row-mt support and linked against
>>>> libvpx 1.7.0.
>>>>
>>>> I tested re-encoding
>>>>
>>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>>>> a size reduction from 48M to 31M.
>>>>
>>>> When using eight CPU cores on one of our video scaler servers, enabling
>>>> row-mt
>>>> gives a significant performance boost; encoding time went down from
>>>> 5:31 mins
>>>> to 3:36 mins.
>>>>
>>>> All the details can be found at
>>>> https://phabricator.wikimedia.org/T190333#4324995
>>>>
>>>> Cheers,
>>>> Moritz
>>>>
>>>> ___
>>>> Wikitech-l mailing list
>>>> Wikitech-l@lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>
>>>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

The quality/bandwidth for the output in this command will be determined by
the qmin/qmax settings you provided; the value of 19 is a bit conservative
and will likely produce a relatively large, but good quality file. Try
various different settings to dial in on desired quality; larger settings
for qmin/qmax will be lower quality / smaller bandwidth.

The settings we'll be using for derivatives are based on the constrained
quality mode, using the -crf option plus a relatively high maximum bitrate
as a constraint for high-motion files, and also a conservative -qmin to
ensure that bits aren't wasted on details that are actually source
compression artifacts.


In general, you should optimize the files you upload for quality, not for
size -- higher source quality ensures better quality for the viewable
derivatives, and for future re-use possibilities. Note that you can keep
using VP8 for uploaded files; there's no immediate need to change
configurations for uploads unless the files themselves are too big to
upload and need higher compression.

-- brion

On Thu, Jul 26, 2018 at 9:38 PM rupert THURNER 
wrote:

> i tried VP9 uploading a video with a spinning wheel made with a
> samsung galaxy s5, mp4, original size 99.5MB. i removed the sound
> beforehand though:
>
> https://commons.wikimedia.org/wiki/File:180522-alpaca-spinnen-gotthard-passh%C3%B6he-silent.webm
>
> using:
> https://tools.wmflabs.org/video2commons/
>
> which used the following (i removed the paths ...)
> ffmpeg -y -i dl.unknown_video -threads 0 -skip_threshold 0 -bufsize
> 6000k -rc_init_occupancy 4000 -qmin 19 -qmax 19 -vcodec libvpx-vp9
> -tile-columns 4 -f webm -ss 0 -an -pass 1 -passlogfile
> dl.unknown_video.an.vp9.webm.log /dev/null
>
> ffmpeg -y -i dl.unknown_video -threads 0 -skip_threshold 0 -bufsize
> 6000k -rc_init_occupancy 4000 -qmin 19 -qmax 19 -vcodec libvpx-vp9
> -tile-columns 4 -f webm -ss 0 -an -pass 2 -passlogfile
> dl.unknown_video.an.vp9.webm.log dl.unknown_video.an.vp9.webm
>
> the size increased by 30 mb, the quality on a smaller screen is the
> same, i did not verify on a high resolution screen if a difference can
> be noticed.
>
> rupert
>
>
> On Fri, Jul 27, 2018 at 2:47 AM, Brion Vibber 
> wrote:
> > Oh and one more thing!
> >
> > For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
> > resolutions, which people can manually bump up to when watching videos
> with
> > a suitable 4K source on a high-res screen. They use higher data rates,
> but
> > only a small fraction of input files are 4K so should not significantly
> > increase disk space projections for now.
> >
> > These can take a long time to compress, so if we find it's problematic
> we'll
> > turn them back off until the jobs can be split into tiny chunks (future
> work
> > planned!), but it works in my testing and shouldn't clog the servers now
> > that we have more available.
> >
> > (Note that the ogv.js player shim for Safari will not handle
> greater-than-HD
> > resolutions fast enough for playback, even on a fast Mac or iPad; for
> best
> > results for 4K playback use Firefox, Chrome, or a Chromium-based
> browser.)
> >
> > -- brion
> >
> > On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
> wrote:
> >>
> >> Ok, after some delay for re-tweaking the encoding settings for higher
> >> quality when needed, and pulling in some other improvements to the
> config
> >> system, all related updates to TimedMediaHandler have been merged. :D
> >>
> >> If all goes well with the general deployments in the next few days,
> expect
> >> the beginning of VP9 rollout starting next week.
> >>
> >> Changes since the earlier announcement:
> >> * the new row-multithreading will be available, which allows higher
> >> threading usage at all resolutions; encoding times will be more like
> 1.5-2x
> >> slower instead of 3-4x slower.
> >> * switch to constrained quality with a larger max bitrate: many files
> will
> >> become significantly smaller in their VP9 versions, but some will
> actually
> >> increase in exchange for a huge increase in quality -- this is mostly
> 60fps
> >> high-rate files, and those with lots of motion and detail that didn't
> >> compress well at the default low data rates.
> >>
> >> -- brion
> >>
> >> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
> >> wrote:
> >>>
> >>> Awesome sauce. Thanks Moritz!
> >>>
> >>> -- brion
> >>>
> >>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff
> >>>  wrote:
> >>>>

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Oh and one more thing!

For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
resolutions, which people can manually bump up to when watching videos with
a suitable 4K source on a high-res screen. They use higher data rates, but
only a small fraction of input files are 4K so should not significantly
increase disk space projections for now.

These can take a long time to compress, so if we find it's problematic
we'll turn them back off until the jobs can be split into tiny chunks
(future work planned!), but it works in my testing and shouldn't clog the
servers now that we have more available.

(Note that the ogv.js player shim for Safari will not handle
greater-than-HD resolutions fast enough for playback, even on a fast Mac or
iPad; for best results for 4K playback use Firefox, Chrome, or a
Chromium-based browser.)

-- brion

On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber  wrote:

> Ok, after some delay for re-tweaking the encoding settings for higher
> quality when needed, and pulling in some other improvements to the config
> system, all related updates to TimedMediaHandler have been merged. :D
>
> If all goes well with the general deployments in the next few days, expect
> the beginning of VP9 rollout starting next week.
>
> Changes since the earlier announcement:
> * the new row-multithreading will be available, which allows higher
> threading usage at all resolutions; encoding times will be more like 1.5-2x
> slower instead of 3-4x slower.
> * switch to constrained quality with a larger max bitrate: many files will
> become significantly smaller in their VP9 versions, but some will actually
> increase in exchange for a huge increase in quality -- this is mostly 60fps
> high-rate files, and those with lots of motion and detail that didn't
> compress well at the default low data rates.
>
> -- brion
>
> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
> wrote:
>
>> Awesome sauce. Thanks Moritz!
>>
>> -- brion
>>
>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>> mmuhlenh...@wikimedia.org> wrote:
>>
>>> Hi all,
>>>
>>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>>> > Current state on this:
>>> >
>>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>>> with
>>> > best performance; Moritz made a start on libvpx but we still have to
>>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>>> way to
>>> > 3.4)
>>>
>>> I've completed this today. We now have a separate repository component
>>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>>> (thus allowing us to follow the ffmpeg security updates released in
>>> Debian
>>> with a local rebuild) with backported row-mt support and linked against
>>> libvpx 1.7.0.
>>>
>>> I tested re-encoding
>>>
>>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>>> a size reduction from 48M to 31M.
>>>
>>> When using eight CPU cores on one of our video scaler servers, enabling
>>> row-mt
>>> gives a significant performance boost; encoding time went down from 5:31
>>> mins
>>> to 3:36 mins.
>>>
>>> All the details can be found at
>>> https://phabricator.wikimedia.org/T190333#4324995
>>>
>>> Cheers,
>>> Moritz
>>>
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Ok, after some delay for re-tweaking the encoding settings for higher
quality when needed, and pulling in some other improvements to the config
system, all related updates to TimedMediaHandler have been merged. :D

If all goes well with the general deployments in the next few days, expect
the beginning of VP9 rollout starting next week.

Changes since the earlier announcement:
* the new row-multithreading will be available, which allows higher
threading usage at all resolutions; encoding times will be more like 1.5-2x
slower instead of 3-4x slower.
* switch to constrained quality with a larger max bitrate: many files will
become significantly smaller in their VP9 versions, but some will actually
increase in exchange for a huge increase in quality -- this is mostly 60fps
high-rate files, and those with lots of motion and detail that didn't
compress well at the default low data rates.

-- brion

On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber  wrote:

> Awesome sauce. Thanks Moritz!
>
> -- brion
>
> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
> mmuhlenh...@wikimedia.org> wrote:
>
>> Hi all,
>>
>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>> > Current state on this:
>> >
>> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
>> with
>> > best performance; Moritz made a start on libvpx but we still have to
>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
>> way to
>> > 3.4)
>>
>> I've completed this today. We now have a separate repository component
>> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
>> (thus allowing us to follow the ffmpeg security updates released in Debian
>> with a local rebuild) with backported row-mt support and linked against
>> libvpx 1.7.0.
>>
>> I tested re-encoding
>>
>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
>> (which is a nice fast-paced test file) from VP8 to VP9, which results in
>> a size reduction from 48M to 31M.
>>
>> When using eight CPU cores on one of our video scaler servers, enabling
>> row-mt
>> gives a significant performance boost; encoding time went down from 5:31
>> mins
>> to 3:36 mins.
>>
>> All the details can be found at
>> https://phabricator.wikimedia.org/T190333#4324995
>>
>> Cheers,
>> Moritz
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] AV1? (was Re: [Multimedia] Video output changing to WebM VP9/Opus soon)

[Brion Vibber]

On Mon, Jul 23, 2018, 5:14 AM Jaime Crespo  wrote:

> Thanks to all people involved,
>
> I just read about this new video format in the making/released [0].
>
> Of course, I am not asking to support this, as this seems like the future,
> and not the present, but being a complete noob on video formats and codecs,
> I would like to know if someone more knolegeble has some insight about this
> and if it is something to keep in mind/someone has tested it and has
> experiences to share/client and vendor support?
>

AV1 is very much on my radar! It's the successor to VP8 and VP9, with
encoding techniques borrowed from Xiph's Daala and Cisco's Thor.

Client support is not yet widely rolled out, but there's broad industry
support in the working group with the exception of Apple. Until AV1 reaches
general usage many of the same companies are pushing VP9, including
Microsoft which now fully supports VP9/Opus in Edge.

I haven't integrated AV1 playback to ogv.js yet for Safari but it's worth
trying; the bitstream format is frozen and the codec library is being
improved.

Somewhere a year or two down the road we will likely want to support AV1
for its even better compression, but have to wait for things to shake out
first... it'll be even more demanding on the CPU so chunked encoding to
better utilize our server farm would be good to have ready before that
switch.

-- brion

>
> --
> Jaime
>
> [0]  https://blog.mozilla.org/blog/2018/07/11/royalty-free-web-video-codecs/>
>
> On Fri, Jun 29, 2018 at 6:46 PM, Brion Vibber 
> wrote:
>
> > Awesome sauce. Thanks Moritz!
> >
> > -- brion
> >
> > On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
> > mmuhlenh...@wikimedia.org> wrote:
> >
> > > Hi all,
> > >
> > > On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> > > > Current state on this:
> > > >
> > > > * still hoping to deploy the libvpx+ffmpeg backport first so we start
> > > with
> > > > best performance; Moritz made a start on libvpx but we still have to
> > > > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
> > way
> > > to
> > > > 3.4)
> > >
> > > I've completed this today. We now have a separate repository component
> > > for stretch-wikimedia (named component/vp9) which includes ffmpeg
> 3.2.10
> > > (thus allowing us to follow the ffmpeg security updates released in
> > Debian
> > > with a local rebuild) with backported row-mt support and linked against
> > > libvpx 1.7.0.
> > >
> > > I tested re-encoding
> > >
> > > https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_
> > Pitts_Todeswand_2017_-_Jagath_Perera.webm
> > > (which is a nice fast-paced test file) from VP8 to VP9, which results
> in
> > > a size reduction from 48M to 31M.
> > >
> > > When using eight CPU cores on one of our video scaler servers, enabling
> > > row-mt
> > > gives a significant performance boost; encoding time went down from
> 5:31
> > > mins
> > > to 3:36 mins.
> > >
> > > All the details can be found at
> > > https://phabricator.wikimedia.org/T190333#4324995
> > >
> > > Cheers,
> > > Moritz
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Awesome sauce. Thanks Moritz!

-- brion

On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
mmuhlenh...@wikimedia.org> wrote:

> Hi all,
>
> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> > Current state on this:
> >
> > * still hoping to deploy the libvpx+ffmpeg backport first so we start
> with
> > best performance; Moritz made a start on libvpx but we still have to
> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the way
> to
> > 3.4)
>
> I've completed this today. We now have a separate repository component
> for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
> (thus allowing us to follow the ffmpeg security updates released in Debian
> with a local rebuild) with backported row-mt support and linked against
> libvpx 1.7.0.
>
> I tested re-encoding
>
> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
> (which is a nice fast-paced test file) from VP8 to VP9, which results in
> a size reduction from 48M to 31M.
>
> When using eight CPU cores on one of our video scaler servers, enabling
> row-mt
> gives a significant performance boost; encoding time went down from 5:31
> mins
> to 3:36 mins.
>
> All the details can be found at
> https://phabricator.wikimedia.org/T190333#4324995
>
> Cheers,
> Moritz
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Current state on this:

* delaying to mid-July so I don't start a batch process and leave it
unattended for a week :)
* still hoping to deploy the libvpx+ffmpeg backport first so we start with
best performance; Moritz made a start on libvpx but we still have to
resolve ffmpeg (possibly by patching 3.2 instead of updating all the way to
3.4)
* after further testing of 30fps and 60fps files, I'm also planning to
increase the data rate a bit.

I want to do a little more testing on the ideal data rate, but it'll end up
looking more like a 30% reduction than a 38% reduction in file size. Our
current VP8 and provisional-VP9 configurations were tuned mostly on 24fps
files, while 30fps files are very common and require a moderately higher
data rate to reduce compression artifacts. (50fps and 60fps files are also
around, and will also benefit from an increase in data rate.)

(Possibly the data rate will end up scaled according to frame rate, but
frame rate is surprisingly hard to calculate reliably for WebM input.)

-- brion


On Sun, Jun 17, 2018 at 1:32 PM Brion Vibber  wrote:

> On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff  wrote:
>
>> Woo!
>>
>> Good to see us moving forward to newer codecs.
>>
>
> :D
>
>
>> What is the expected impact on transcode time/memory usage? (As in, will
>> large videos still transcode fine or would they potentially hit limits
>> sooner?)
>>
>
> Using the same CPU thread count, typical files take about 3-4 times longer
> with the configuration we've got ready. This doesn't seem to be a major
> problem at low resolutions, but higher resolutions may need to have the
> limits raised for very long videos like conference talks. Memory usage may
> be higher but I've not specifically noticed a major difference between VP8
> and VP9 there.
>
> This could be improved significantly by updating to libvpx 1.7 and a
> matching version of ffmpeg that supports macroblock row multithreading:
> this means that we'll be able to use more than 4 threads for HD videos,
> which should allow using all cores on a 20-core/40-thread machine if it's
> not loaded up with other files.
>
> The necessary libraries are released; we just need to backport the newer
> packages to Debian 9 I believe. Don't yet know whether this will be an easy
> or hard task.
>
> Now that you mention it I am concerned about the time limit on very long
> videos, so I'll take another look at the packaging backport and see if we
> can knock that out quickly. :)
>
> -- brion
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

On Sun, Jun 17, 2018 at 12:30 PM Brian Wolff  wrote:

> Woo!
>
> Good to see us moving forward to newer codecs.
>

:D


> What is the expected impact on transcode time/memory usage? (As in, will
> large videos still transcode fine or would they potentially hit limits
> sooner?)
>

Using the same CPU thread count, typical files take about 3-4 times longer
with the configuration we've got ready. This doesn't seem to be a major
problem at low resolutions, but higher resolutions may need to have the
limits raised for very long videos like conference talks. Memory usage may
be higher but I've not specifically noticed a major difference between VP8
and VP9 there.

This could be improved significantly by updating to libvpx 1.7 and a
matching version of ffmpeg that supports macroblock row multithreading:
this means that we'll be able to use more than 4 threads for HD videos,
which should allow using all cores on a 20-core/40-thread machine if it's
not loaded up with other files.

The necessary libraries are released; we just need to backport the newer
packages to Debian 9 I believe. Don't yet know whether this will be an easy
or hard task.

Now that you mention it I am concerned about the time limit on very long
videos, so I'll take another look at the packaging backport and see if we
can knock that out quickly. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

In the next couple weeks I'm planning to start switching our video
transcode output from WebM VP8/Vorbis to the newer WebM VP9/Opus profile,
which saves us about 38% on file size and bandwidth while retaining the
same quality.

This will not affect what kinds of files you upload; only the scaled
transcoded output files used for playback will change. All modern browsers
that support VP8 support VP9 as well, and our player shim for Safari and IE
will continue to work.

All the details:
https://www.mediawiki.org/wiki/Extension:TimedMediaHandler/VP9_transition

Comments and questions welcome!

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit as a shared community space

[Brion Vibber]

I'd just like to apologize for dragging the other thread into this one and
being overly personal and failing to assume good faith.

That was a failing on my part, and not good practice.

Please if you respond further to this thread, treat only the narrow issue
of ownership / maintainership expectations and where/how we should be more
clear on it.

Further discussion on people's motives about the code of conduct will
likely not be productive for anyone on this thread.

My apologies to all; I should do better. You all deserve better.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit as a shared community space

[Brion Vibber]

On Sat, Jun 9, 2018 at 10:55 AM Isarra Yos  wrote:

> Perhaps I was too subtle the last time I hinted at this: this is toxic.
> What you and others are doing misrepresenting what others are saying,
> the general heavy-handedness, the implications that anyone against a
> specific aspect of implementation is against the very concept of good
> conduct...
>
> Please stop.
>

Fair enough, I'll leave the thread and ponder for a bit.

Be good to yourselves, all, and don't fall into anger. I'm not immune
either.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit as a shared community space

[Brion Vibber]

On Sat, Jun 9, 2018 at 10:21 AM Alex Monk  wrote:

> On 9 June 2018 at 18:14, Brion Vibber  wrote:
>
> > On Sat, Jun 9, 2018 at 10:00 AM Alex Monk  wrote:
> >
> > > This is outrageous. Not only are you blatantly misrepresenting what
> > various
> > > people are saying in the other thread and their intentions,
> >
> >
> > Perhaps. I've tried to go by the plain readings of position statements
> and
> > I could have made a mistake?
> >
>
>  For example where you said "IMHO specifically because some people are
> trying to avoid being bound by it or protesting its existence by looking
> for loopholes to avoid it", which is not at all what that thread is about
> as has been made very clear in that thread.


I disagree that it has been made clear. I found the opposite to be true in
my experience of reading that thread -- for instance the ability to exclude
the code of conduct from some interactions between developers and users was
cited as a desired feature, and thus a reason to want to avoid advertising
the code of conduct in the repo.

Perhaps I'm reading too much into the idea of wanting to avoid the code of
conduct as a proxy for not liking it?



>
> On 9 June 2018 at 18:14, Brion Vibber  wrote:
>
> > you are now
> > > suggesting that repository owners do not in fact get to decide what
> goes
> > in
> > > their repository and what does not, as if this has been the case all
> > along.
> >
> >
> > Yes I'm definitely explicitly saying that. Same applies to pages on
> > Wikipedia: you don't get to own them and veto others' clarifications. And
> > sometimes an admit makes an edit stick that you don't like.
>
> This is a bad analogy. Repository owners *are* essentially the admins, and
> in this case get content control. The people involving themselves are more
> akin to global users like stewards trying to override local admin actions,
> and in this case they're not really supposed to have such control of
> content. Like with on-wiki stuff, it's not really so bad when a global user
> comes and does uncontroversial cleanup, but global permissions are not for
> the purpose of involving oneself in local controversy.


I'll let that one stand. Sounds like a good analogy except that this is
exactly the sort of thing a steward might have to intervene for.


>
> On 9 June 2018 at 18:14, Brion Vibber  wrote:
>
> > > It is incredibly ironic how against the spirit of the CoC this all is.
> >
>
> >
> > Can you clarify?
> >
>
> Making Wikimedia technical spaces less welcoming to outsiders.


Not just outsiders generally, but outsiders who have not had a fair shake
in the past because the place has been unwelcoming or full of toxic
interactions.

Reducing toxic interactions is an important part of that, and sometimes
that means telling people who cause toxic interactions that they are not
welcome because we would rather have other people who are less toxic and
can bring different perspectives and representation.

-- brion


> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit as a shared community space

[Brion Vibber]

On Sat, Jun 9, 2018 at 10:00 AM Alex Monk  wrote:

> This is outrageous. Not only are you blatantly misrepresenting what various
> people are saying in the other thread and their intentions,


Perhaps. I've tried to go by the plain readings of position statements and
I could have made a mistake?

you are now
> suggesting that repository owners do not in fact get to decide what goes in
> their repository and what does not, as if this has been the case all along.


Yes I'm definitely explicitly saying that. Same applies to pages on
Wikipedia: you don't get to own them and veto others' clarifications. And
sometimes an admit makes an edit stick that you don't like.


> It is incredibly ironic how against the spirit of the CoC this all is.


Can you clarify?

-- brion



>
> On 9 June 2018 at 16:58, Brion Vibber  wrote:
>
> > Recent threads have demonstrated there seems to be some disconnect about
> > what is expected about maintainership and ownership of repositories.
> >
> > This has spilled over into talk about the code of conduct, IMHO
> > specifically because some people are trying to avoid being bound by it or
> > protesting its existence by looking for loopholes to avoid it. Which I
> > think is a shame, but I don't expect much constructive talk to come out
> of
> > that thread.
> >
> > I think we should though clarify that code repositories on gerrit and
> > diffusion are not owned by any one person, but are technical community
> > spaces held in common for the benefit of the Wikimedia movement. And yes,
> > that means sometimes your favorite project will get documentation commits
> > you personally didn't like.
> >
> > If this has been unclear, it should be made clear. If that means some
> > people host their self-maintained code outside of Wikimedia technical
> > spaces, then that is their decision and I respect it.
> >
> > If some kind of official kerfluffle is needed to decide this, let's talk
> > about how to do that.
> >
> > -- brion
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Gerrit as a shared community space

[Brion Vibber]

Recent threads have demonstrated there seems to be some disconnect about
what is expected about maintainership and ownership of repositories.

This has spilled over into talk about the code of conduct, IMHO
specifically because some people are trying to avoid being bound by it or
protesting its existence by looking for loopholes to avoid it. Which I
think is a shame, but I don't expect much constructive talk to come out of
that thread.

I think we should though clarify that code repositories on gerrit and
diffusion are not owned by any one person, but are technical community
spaces held in common for the benefit of the Wikimedia movement. And yes,
that means sometimes your favorite project will get documentation commits
you personally didn't like.

If this has been unclear, it should be made clear. If that means some
people host their self-maintained code outside of Wikimedia technical
spaces, then that is their decision and I respect it.

If some kind of official kerfluffle is needed to decide this, let's talk
about how to do that.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Planet Wikimedia using new software "rawdog"

[Brion Vibber]

On Wed, May 30, 2018, 7:22 PM Daniel Zahn  wrote:

> On Wed, May 30, 2018 at 6:57 PM, Brion Vibber 
> wrote:
>
> > ... Well, that's a bit of an unfortunate name. o_O
> >
>
> :p  "rawdog is an RSS Aggregator Without Delusions Of Grandeur"
>

Perhaps. I would probably recommend googling software names for common
alternate meanings before choosing to put them in a public-facing role,
though.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Planet Wikimedia using new software "rawdog"

[Brion Vibber]

On Wed, May 30, 2018 at 6:34 PM Daniel Zahn  wrote:

> this is an announcement to let you know that the service ($lang).
> planet.wikimedia.org, an RSS feed aggregator for all Wikimedia related
> blogs, has switched software.
>
>
Thanks for the maintenance!


>
> Today we moved away from planet-venus and to a newer package called
> "rawdog" that does the same thing as before, fetching a bunch of RSS feed
> and combining them into a single page and feed.
>

... Well, that's a bit of an unfortunate name. o_O

-- brion


>
> The reason is that planet-venus has been dropped in Debian stable (stretch)
> because it was unmaintained, so we had to find an alternative to be able to
> upgrade the underlying servers to a current OS version.
>
> If you never heard of planet, here you can find more info:
>
> https://wikitech.wikimedia.org/wiki/Planet.wikimedia.org
>
> If you already use it but just subscribe to the "feed of feeds" then
> nothing should change for you.
>
> (Though note that we support RSS 2.0 but not a separate Atom feed anymore.
> We are redirecting the old atom.xml URL to the new (and old) URL
> rss20.xml.)
>
> If you already use it and look at the web UI, enjoy the new theme that
> Paladox imported from KDE to make it look about 150% better than before.
> (thanks to him for that theming work!)
>
> We also applied patches to make it look more like our former planet for a
> smooth transition.  A "wmf1" package has been built and uploaded at
> https://apt.wikimedia.org/wikimedia/pool/main/r/rawdog/
>
> If you want to know more about "rawdog":
>
> https://offog.org/code/rawdog/
> https://packages.debian.org/stretch/rawdog
>
> If you want to add your blog feed, feel free to upload changes or just drop
> me a mail.
>
> Bugs can be reported here:
> https://phabricator.wikimedia.org/project/view/413/
>
> Tickets are:  https://phabricator.wikimedia.org/T180498 ,
> https://phabricator.wikimedia.org/T168490
>
> Cheers,
>
> Daniel
>
> --
> Daniel Zahn 
> Operations Engineer
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki 1.31 branch (and PHP versions)

[Brion Vibber]

Exciting times! Looking forward to our PHP 7 future, if not just yet. :D

-- brion

On Fri, Apr 20, 2018 at 4:54 PM, Chad  wrote:

> Hi,
>
> I meant to send this Tuesday but I forgot.
>
> MediaWiki 1.31 has been branched from master! You should now see a REL1_31
> branch where appropriate items should be backported to.
>
> Core was branched at 69257de17fc899c447c9f1229b6ed319bc05d316.
> All extensions & skins were branched from their respective masters at about
> the same time as core. I plan to cut rc.0 sometime next week.
>
> PHP versions
> The current plan of action is to leave master as compatible with 5.5 for
> now. This is because Wikimedia production isn't ready quite yet. This is
> being tracked at T172165[0]. We will be moving the REL1_31 branch to 7.0+
> as the required minimum version. Once production is ready, we'll
> forward-port this change to master. It should be a little inconvenient, but
> not too terribly bad (and notably, makes life less stressful for our SREs).
> In the meantime, please do NOT introduce changes to master that require
> 7.0+ for core, vendor, or WMF-deployed extensions & skins. Doing so will
> make me sad :(
>
> Otherwise, great job on 1.31.x everyone! I'm rather pleased with what I'm
> seeing so far. Check out the workboard[1] for ways you can contribute to
> getting it wrapped up (and as always, tag issues with that tag if they
> should absolutely block release).
>
> Have a fantastic weekend!
>
> -Chad
>
> [0] https://phabricator.wikimedia.org/T172165
> [1] https://phabricator.wikimedia.org/project/view/3011/
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] First step for MCR merged: Deprecate and gut the Revision class

[Brion Vibber]

On Dec 22, 2017 3:03 PM, "C. Scott Ananian"  wrote:

I think a simple revert would be simplest.  Adding a feature flag adds new
possibilities of overlooked bugs, especially since this is "just" a
refactoring and so *in theory* shouldn't be changing anything.


Agreed, it's likely to introduce more bugs with a second code path that
way...


Maybe we could just cherry-pick a revert onto the Jan 2 branch, rather than
revert on master and then un-revert after the branch.  It shouldn't be
interpreted as any comment on code quality or "readiness", just a
reflection of the fact that the first deploy after the holiday will
inevitable cause *some* breakage, and in our eggnog-induced stupor it would
be nice if we could just rule out this refactor as contributing to whatever
the breakage turns out to be.  (That's why a feature flag doesn't seem
helpful to me; it would still be lines of code to comb through.  Better
either a clean revert or just deploy the thing as is.)


+1

-- brion

 --scott

On Fri, Dec 22, 2017 at 2:01 PM, Addshore  wrote:

> So the plan going forward will be to create a feature flag for the MCR
> Revision gutting.
> I'll crack on with that this evening.
>
> If that turns out to be too messy then we can revert the MCR patches for
> the next wmf branch.
> I'm currently keeping track of this @
> https://wikitech.wikimedia.org/wiki/User:Addshore/MCR_Revert
>
> On 22 December 2017 at 18:39, Ramsey Isler  wrote:
>
> > Fantastic news! Great work handling this behemoth of a technical
> challenge.
> >
> > On Fri, Dec 22, 2017 at 2:26 AM, Daniel Kinzler <
> > daniel.kinz...@wikimedia.de> wrote:
> >
> >> Hello all!
> >>
> >> Addshore last night merged the patch[1] that is the first major step
> >> towards
> >> Multi-Content-Revisions[2]: it completely guts the Revision class and
> >> turns it
> >> into a thin proxy for the new RevisionStore service. The new code is
now
> >> live
> >> on beta.
> >>
> >> This is our second attempt: The first one, on December 18th, thoroughly
> >> corrupted the beta database. It took us some time and a lot of help
from
> >> Aaron
> >> and especially Roan to figure out what was happening. A detailed
> >> post-mortem by
> >> Roan can be found at [3].
> >>
> >> Anyway - this stage of MCR development introduces the new
multi-revision
> >> capable
> >> interface for revision storage (and blob storage) [4]. It does not yet
> >> introduce
> >> the new database schema, that will be the next step [5][6]. While doing
> >> the
> >> refactoring, I tried to keep the structure of the existing code mostly
> >> intact,
> >> just moving functionality out of Revision into the new classes, most
> >> importantly
> >> RevisionRecord, RevisionStore, and BlobStore.
> >>
> >> Beware that with the next deployment (due January 2nd) the live sites
> >> will start
> >> using the new code. Please keep an eye out for any strangeness
regarding
> >> revision handling. Adam greatly improved test coverage of the relevant
> >> code
> >> (thanks Adam!), but it's always possible that we missed some edge case,
> >> maybe
> >> something about archived revisions that were partially migrated from on
> >> old
> >> schema or something similarly fun.
> >>
> >> Exiting times!
> >>
> >> Cheers
> >> Daniel
> >>
> >>
> >> [1] https://gerrit.wikimedia.org/r/#/c/399174/
> >> [2] https://www.mediawiki.org/wiki/Requests_for_comment/Multi-
> >> Content_Revisions
> >> [3] https://phabricator.wikimedia.org/T183252#3853749
> >> [4] https://phabricator.wikimedia.org/T174025
> >> [5] https://phabricator.wikimedia.org/T174024
> >> [6] https://phabricator.wikimedia.org/T174030
> >>
> >>
> >> --
> >> Daniel Kinzler
> >> Principal Platform Engineer
> >>
> >> Wikimedia Deutschland
> >> Gesellschaft zur Förderung Freien Wissens e.V.
> >>
> >>
> >> ___
> >> Wikitech-l mailing list
> >> Wikitech-l@lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> >
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



--
(http://cscott.net)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] First step for MCR merged: Deprecate and gut the Revision class

[Brion Vibber]

Woohoo! The pieces fall in place... :)

-- brion

On Dec 22, 2017 2:27 AM, "Daniel Kinzler" 
wrote:

> Hello all!
>
> Addshore last night merged the patch[1] that is the first major step
> towards
> Multi-Content-Revisions[2]: it completely guts the Revision class and
> turns it
> into a thin proxy for the new RevisionStore service. The new code is now
> live
> on beta.
>
> This is our second attempt: The first one, on December 18th, thoroughly
> corrupted the beta database. It took us some time and a lot of help from
> Aaron
> and especially Roan to figure out what was happening. A detailed
> post-mortem by
> Roan can be found at [3].
>
> Anyway - this stage of MCR development introduces the new multi-revision
> capable
> interface for revision storage (and blob storage) [4]. It does not yet
> introduce
> the new database schema, that will be the next step [5][6]. While doing the
> refactoring, I tried to keep the structure of the existing code mostly
> intact,
> just moving functionality out of Revision into the new classes, most
> importantly
> RevisionRecord, RevisionStore, and BlobStore.
>
> Beware that with the next deployment (due January 2nd) the live sites will
> start
> using the new code. Please keep an eye out for any strangeness regarding
> revision handling. Adam greatly improved test coverage of the relevant code
> (thanks Adam!), but it's always possible that we missed some edge case,
> maybe
> something about archived revisions that were partially migrated from on old
> schema or something similarly fun.
>
> Exiting times!
>
> Cheers
> Daniel
>
>
> [1] https://gerrit.wikimedia.org/r/#/c/399174/
> [2] https://www.mediawiki.org/wiki/Requests_for_comment/
> Multi-Content_Revisions
> [3] https://phabricator.wikimedia.org/T183252#3853749
> [4] https://phabricator.wikimedia.org/T174025
> [5] https://phabricator.wikimedia.org/T174024
> [6] https://phabricator.wikimedia.org/T174030
>
>
> --
> Daniel Kinzler
> Principal Platform Engineer
>
> Wikimedia Deutschland
> Gesellschaft zur Förderung Freien Wissens e.V.
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Best practices for WebAssembly and compiled JS in front-end modules?

[Brion Vibber]

The latest version of all major browsers support C/C++/etc code compiled to
WebAssembly  ("wasm") format, in addition to the
older "asm.js" style of compilation targeting a subset of JavaScript
directly.

As far as I know the first library that MediaWiki uses to support
WebAssembly is ogv.js, my codec & playback library which we use for playing
media on Safari, IE, and Edge. ogv.js is packaged with the
TimedMediaHandler extension, and we've used its asm.js mode for about two
years now.

I'd like to enable the WebAssembly mode in production for the faster
compilation times, but want to double-check that we've established some
best practices for WebAssembly usage first.

https://phabricator.wikimedia.org/T181451 -- I've started up a provisional
RFC, currently more of a document with a few pain points called out:

* packaging (check in binaries from a library, or use a package manager
like npm?)
* loading (loading .wasm or .js files as assets, vs what can use RL)
* security (where to concentrate for security reviews, etc)

This may grow into a recommendation on how to package & load things and
some tweaks to RL. If anybody has any questions about the scariness that is
wasm, give a shout. I want to make it not-too-scary. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HHVM vs. Zend divergence

[Brion Vibber]

On Mon, Sep 18, 2017 at 1:58 PM, Max Semenik  wrote:

> 2) Declare our loyalty to HHVM. This will result in most of our current
> users being unable to upgrade, eventually producing what amounts to a
> WMF-only product and lots of installations with outdated MediaWiki having
> security holes. At least we will be able to convert to Hack eventually.
> This is a very clean-cut case of vendor lock-in though, and if Facebook
> decides to switch their code base to something shinier, we'll be deep in
> trouble.
>

Hack has a couple interesting points, especially its async system, but
isn't _hugely_ compelling at this point.

If they're going to be dropping destructors and references we'd have to
retool our RAII patterns and our hook 'out-param' patterns to work with
future-Hack, but that's possible if we *want* to make such a change.

However I don't see a lot of interest in such a change in this discussion
or among TechCom, and buying into a single-vendor system is a risk both for
us (if they change the language in ways we don't like or drop it) and
others (harder to set up if few HHVM providers).


>
> 3) Revert WMF to Zend and forget about HHVM. This will result in
> performance degradation, however it will not be that dramatic: when we
> upgraded, we switched to HHVM from PHP 5.3 which was really outdated, while
> 5.6 and 7 provided nice performance improvements.
>

Migrating WMF's implementation to PHP 7 is probably the way to go. I leave
it up to ops to figure out how to make the change. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] How does a build process look like for a mediawiki extension repository?

[Brion Vibber]

For JS dependencies, image optimizations etc the state of the art still
seems to be to have a local one-off script and commit the build artifacts
into the repo. (For instance TimedMediaHandler fetches some JS libs via npm
and copies/patches them into the resources/ dir.)

For PHP deps we've got composer dependency installation for extensions, so
it seems like there's an opportunity to do other build steps in this
stage...

Not sure offhand if that can be snuck into composer directly or if we'd
need to replace the "run composer" step with "run this script, which runs
composer and also does other build steps".

-- brion


On Wed, Jun 7, 2017 at 10:18 AM, Joaquin Oltra Hernandez <
jhernan...@wikimedia.org> wrote:

> *Context*
>
> We'd like to have a build script/process for an extension so that I can
> perform certain commands to install dependencies and perform optimizations
> on the extension sources. For example, on front-end sources.
>
> Some examples could be:
>
>- Installing libraries from bower or npm and bundling them into the
>resources folder
>- Applying post processing steps to CSS with something like post css
>- Optimizing images
>
> We are aware of other projects that have build processes for building
> deployables, but not extensions.
> Such projects have different ways of dealing with this. A common way is
> having a repository called /deploy and in there you pull from
>  and run the build scripts, and that is the repository that gets
> deployed.
>
> *Current system*
>
> The current way we usually do this (if we do) is run those build
> scripts/jobs on the developers machines and commit them into the git
> repository on master.
>
> With this system, if you don't enforce anything in CI, then build processes
> may be skipped (human error).
>
> If you enforce it (by running the process and comparing with what has been
> committed in CI) then patches merged to master that touch the same files
> will produce merge conflicts with existing open patches, forcing a
> rebase+rebuild on open patches every time one is merged on master.
>
> *Questions*
>
> Can we have a shared configuration/convention/system for having a build
> step on mediawiki extensions?
>
>- So that a build process is run
>   - on CI jobs that require production assets like the selenium jobs
>   - on the deployment job that deploys the extension to the beta
>   cluster and to production
>
> How would it look like? Are any extensions doing a pre-deployment build
> step?
>
> Thanks.
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Turkey ban

[Brion Vibber]

The apps aren't currently designed with network-level blocking
circumvention in mind. It's certainly possible to be much more resilient
especially to something simple like a DNS blacklist, but:

* anything we do starts an arms race where smarter blocking could block
that too
* the apps are open source, so it's very easy to check how they work and
then design a fancier block
* such efforts might affect legal cases (IANAL)

-- brion



On Sat, Apr 29, 2017 at 1:47 PM, Denny Vrandečić 
wrote:

> Surely it should be possible to have a more resilient access to Wikipedia
> content in the app than most vanilla browsers provide?
>
> On Sat, Apr 29, 2017, 13:40 Florian Schmidt <
> florian.schmidt.wel...@t-online.de> wrote:
>
> > Because the app uses the same base URL for the requests to Wikipedia as
> > other access methods (e.g. internet browsers), a block a Wikipedia would
> > apply to the App, too. Sorry, but there's no way to work around this, if
> a
> > block is implemented to block all requests to a specific domain. One
> > workaround on the client side would be to use a VPN, so that the request
> > will be made from another origin and the answer forwarded to your
> > phone/computer. However, I'm not sure, if this is allowed in any way.
> >
> > Best,
> > Florian
> >
> > -Ursprüngliche Nachricht-
> > Von: Wikitech-l [mailto:wikitech-l-boun...@lists.wikimedia.org] Im
> > Auftrag von Denny Vrandecic
> > Gesendet: Samstag, 29. April 2017 21:16
> > An: Wikimedia developers 
> > Betreff: [Wikitech-l] Turkey ban
> >
> > Does it also affect the app?
> >
> > If so, why, and can we circumvent that?
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Questions over usage of ObjectCache

[Brion Vibber]

I've filed a task in phabricator to follow up in case there's any surprises
with the proposed fix. :) https://phabricator.wikimedia.org/T162077

-- brion

On Mon, Apr 3, 2017 at 11:52 AM, Brion Vibber <bvib...@wikimedia.org> wrote:

> On Mon, Apr 3, 2017 at 12:56 AM, Nischay Nahata <nischay...@gmail.com>
> wrote:
>
>>
>> I have been trying to add certain rate limiting actions for my extension's
>> API and noticed that it doesn't work with $wgMainCacheType set to
>> CACHE_NONE
>>
>> This is because  User::pingLimiter()
>> uses ObjectCache::getLocalClusterInstance() to store the current rate
>> limit
>> counts.
>>
>> Is this the expected behavior? Looks pretty deceitful to me
>> as $wgMainCacheType is assumed to define the cache type but is actually
>> disabling a core functionality like rate limiting and maybe few others?
>>
>
> I'm pretty sure this one's my fault -- at the time it was added we didn't
> expect the rate limit feature to be used in a minimal configuration without
> an object cache configured, and it ended up being kind of kept that way
> through the years.
>
> I would recommend probably changing this:
> $cache = ObjectCache::getLocalClusterInstance();
> to:
> $cache = ObjectCache::getInstance( CACHE_ANYTHING );
>
> That should set the rate limiter to use the database-backed cache when
> $wgMainCacheType is set to CACHE_NONE, and will use the main cache when
> it's not set off.
>
> -- brion
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Questions over usage of ObjectCache

[Brion Vibber]

On Mon, Apr 3, 2017 at 12:56 AM, Nischay Nahata 
wrote:

>
> I have been trying to add certain rate limiting actions for my extension's
> API and noticed that it doesn't work with $wgMainCacheType set to
> CACHE_NONE
>
> This is because  User::pingLimiter()
> uses ObjectCache::getLocalClusterInstance() to store the current rate
> limit
> counts.
>
> Is this the expected behavior? Looks pretty deceitful to me
> as $wgMainCacheType is assumed to define the cache type but is actually
> disabling a core functionality like rate limiting and maybe few others?
>

I'm pretty sure this one's my fault -- at the time it was added we didn't
expect the rate limit feature to be used in a minimal configuration without
an object cache configured, and it ended up being kind of kept that way
through the years.

I would recommend probably changing this:
$cache = ObjectCache::getLocalClusterInstance();
to:
$cache = ObjectCache::getInstance( CACHE_ANYTHING );

That should set the rate limiter to use the database-backed cache when
$wgMainCacheType is set to CACHE_NONE, and will use the main cache when
it's not set off.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

Summary from March 8 irc meeting:
https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-03-08-22.03.html

-- brion

On Mon, Mar 6, 2017 at 9:43 AM, Brion Vibber <bvib...@wikimedia.org> wrote:

> On Wed, Feb 15, 2017 at 3:06 PM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
>> Great feedback everybody -- I'll make more updates and we'll circle back
>> for another discussion in a week or two!
>>
>> Meeting summary (full logs linked from there):
>> https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wiki
>> media-office.2017-02-15-22.01.html
>>
>
> We're going to have another checkin during ArchCom IRC meeting time this
> Wednesday, 22:00 UTC / 2pm PST in #wikimedia-office
>
> Documents will be updated shortly reflecting the previous discussion &
> ongoing tweaks.
>
> Open questions include:
> * should we go straight to the MCR-ready schema or do this in two steps,
> one to break up tables & prep, and another for the MCR content model?
> * final model for updating archive & text
>
> -- brion
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Code of Conduct for Wikimedia technical spaces now out of draft state

[Brion Vibber]

Per the last round of discussion & votes closing on the amendments section
some days back, the draft period for the Code of Conduct for technical
spaces is complete.

I've taken the liberty of removing the 'draft' template and moving the
pages out of their "/Draft" subpage location to their final locations:

* https://www.mediawiki.org/wiki/Code_of_Conduct
* https://www.mediawiki.org/wiki/Code_of_Conduct/Cases
* https://www.mediawiki.org/wiki/Code_of_Conduct/Committee
* https://www.mediawiki.org/wiki/Code_of_Conduct/Amendments

As I understand, this means the committee will be bootstrapping soon --
Quim will have more information on the state of things there.

Thanks to everybody for all your input over the last many months.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

On Wed, Feb 15, 2017 at 3:06 PM, Brion Vibber <bvib...@wikimedia.org> wrote:

> Great feedback everybody -- I'll make more updates and we'll circle back
> for another discussion in a week or two!
>
> Meeting summary (full logs linked from there):
> https://tools.wmflabs.org/meetbot/wikimedia-office/2017/
> wikimedia-office.2017-02-15-22.01.html
>

We're going to have another checkin during ArchCom IRC meeting time this
Wednesday, 22:00 UTC / 2pm PST in #wikimedia-office

Documents will be updated shortly reflecting the previous discussion &
ongoing tweaks.

Open questions include:
* should we go straight to the MCR-ready schema or do this in two steps,
one to break up tables & prep, and another for the MCR content model?
* final model for updating archive & text

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] SHA-1 hash officially broken

[Brion Vibber]

Added a tracking/epic task on phab:
https://phabricator.wikimedia.org/T158986

Attach any specific things to that as subtasks, y'all.

-- brion


On Fri, Feb 24, 2017 at 10:51 AM, Florian Schmidt <
florian.schmidt.wel...@t-online.de> wrote:

> > but this doesn't need to be a "drop everything" hurry task.
>
> Totally agree! :)
>
> Thanks for the information with the page on mediawiki.org. I'll take a
> look and probably edit it, if I see something :P
>
> Best,
> Florian
>
> -Ursprüngliche Nachricht-
> Von: Wikitech-l [mailto:wikitech-l-boun...@lists.wikimedia.org] Im
> Auftrag von Brion Vibber
> Gesendet: Freitag, 24. Februar 2017 19:48
> An: Wikimedia developers <wikitech-l@lists.wikimedia.org>
> Betreff: Re: [Wikitech-l] SHA-1 hash officially broken
>
> On Fri, Feb 24, 2017 at 10:34 AM, Florian Schmidt <
> florian.schmidt.wel...@t-online.de> wrote:
>
> > I searched in phabricator, if we've a task already, but couldn't find
> any.
> > However, as the phabricator search and me aren't really good friends,
> > it's possible, that the search wasn't as honest to me, as I would wish
> > and I missed something, so I ask on this list :) Do we've a task
> > already to track the work on this topic? A short github search[1]
> > showed some usages of sha1 (at least the string), so I suspect, that
> > there're some places where we use it, right?
> >
> > [1]
> > https://github.com/wikimedia/mediawiki/search?utf8=%E2%9C%93=SHA1
>
>
> Usages are listed at https://www.mediawiki.org/wiki/Manual:Hashing --
> I've added a "purpose" column on the DB fields list. There might be a
> couple we missed, so feel free to edit!
>
>
> It looks like img_sha1 and fa_storage_key are the biggest practical
> dangers of collision, in that pairs of images could be created such that,
> say, one is a cute kitten and the other is a shock image, such that
> deletion and undeletion might surface the wrong file from what the
> undeleting admin expected.
>
> However since the attack requires creating a common prefix and suffix for
> the matching pair of files, there's little or no danger of an attack that
> replaces legitimate files already uploaded by someone else.
>
> I would probably recommend migrating rev_sha1 and img_sha1's usages in
> collision detection over to a rev_sha256 and img_sha256 columns, and
> reworking filearchve to use a sha-256 hash for content addressing of new
> files, but this doesn't need to be a "drop everything" hurry task.
>
> -- brion
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] SHA-1 hash officially broken

[Brion Vibber]

On Fri, Feb 24, 2017 at 10:34 AM, Florian Schmidt <
florian.schmidt.wel...@t-online.de> wrote:

> I searched in phabricator, if we've a task already, but couldn't find any.
> However, as the phabricator search and me aren't really good friends, it's
> possible, that the search wasn't as honest to me, as I would wish and I
> missed something, so I ask on this list :) Do we've a task already to track
> the work on this topic? A short github search[1] showed some usages of sha1
> (at least the string), so I suspect, that there're some places where we use
> it, right?
>
> [1] https://github.com/wikimedia/mediawiki/search?utf8=%E2%9C%93=SHA1


Usages are listed at https://www.mediawiki.org/wiki/Manual:Hashing -- I've
added a "purpose" column on the DB fields list. There might be a couple we
missed, so feel free to edit!


It looks like img_sha1 and fa_storage_key are the biggest practical dangers
of collision, in that pairs of images could be created such that, say, one
is a cute kitten and the other is a shock image, such that deletion and
undeletion might surface the wrong file from what the undeleting admin
expected.

However since the attack requires creating a common prefix and suffix for
the matching pair of files, there's little or no danger of an attack that
replaces legitimate files already uploaded by someone else.

I would probably recommend migrating rev_sha1 and img_sha1's usages in
collision detection over to a rev_sha256 and img_sha256 columns, and
reworking filearchve to use a sha-256 hash for content addressing of new
files, but this doesn't need to be a "drop everything" hurry task.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] SHA-1 hash officially broken

[Brion Vibber]

Google security have announced that they have a working collision attack
against the SHA-1 hash:

https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html

It's highly recommended to move to sha-256 where doable.

Note that MediaWiki uses sha-1 in a number of places; in some such as
revision hashes it's advisory for tools only, but in other places like
deleted files (filearchive table) we use it for addressing, and should
consider steps to mitigate attacks swapping in alternate files during
deletion/undeletion.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

Great feedback everybody -- I'll make more updates and we'll circle back
for another discussion in a week or two!

Meeting summary (full logs linked from there):
https://tools.wmflabs.org/meetbot/wikimedia-office/2017/wikimedia-office.2017-02-15-22.01.html

-- brion

On Wed, Feb 15, 2017 at 9:06 PM, Brion Vibber <bvib...@wikimedia.org> wrote:

> Correction: 22:00 UTC / 2pm PST in #wikimedia-office. Sorry, I calculated
> with the wrong time by mistake!
>
> -- brion
>
> On Tue, Feb 14, 2017 at 10:38 AM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
>> Whoops I forgot to mention in the list post -- we're planning to talk
>> about this topic in the public ArchCom IRC meeting this Wednesday (21:00
>> UTC / 2pm PDT).
>>
>> Already getting good feedback on the page, am updating it, and looking
>> forward to more.... Thanks all. :)
>>
>> -- brion
>>
>> On Mon, Feb 13, 2017 at 9:28 AM, Brion Vibber <bvib...@wikimedia.org>
>> wrote:
>>
>>> I've got an early draft of some notes
>>> <https://www.mediawiki.org/wiki/User:Brion_VIBBER/Compacting_the_revision_table_round_2>
>>> for a restructuring of the revision table, to support the following:
>>>
>>> * making the revision table itself smaller by breaking large things out
>>> * reducing duplicate string storage for content model/format,
>>> username/IP address, and edit comments
>>> * multi-content revisions ("MCR") - multiple Content blobs of different
>>> types on a page, revisioned consistently
>>>
>>> There's also some ideas going around about using denormalized summary
>>> tables more aggressively, perhaps changing where the indexes used for
>>> specific uses live. For instance, a 'contribs' table with just the bits
>>> needed for the index lookups for user-contribs, then joined to the other
>>> tables.
>>>
>>> Initial notes at https://www.mediawiki.org/w
>>> iki/User:Brion_VIBBER/Compacting_the_revision_table_round_2 -- I'll be
>>> cleaning this up a bit more in response to feedback and concerns.
>>>
>>> If we go through with this sort of change, we'll need to carefully
>>> consider the upgrade transition. We'll also need to make sure that all
>>> relevant queries are updated, and that folks using the databases indirectly
>>> (via tool labs, etc) are all able to cleanly handle the new fun stuff.
>>> Feedback will be crucial here. :)
>>>
>>> Potentially we might split this into a couple transitions instead, or
>>> otherwise make major changes to the plan. Nothing's set in stone yet!
>>>
>>> -- brion
>>>
>>
>>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

Correction: 22:00 UTC / 2pm PST in #wikimedia-office. Sorry, I calculated
with the wrong time by mistake!

-- brion

On Tue, Feb 14, 2017 at 10:38 AM, Brion Vibber <bvib...@wikimedia.org>
wrote:

> Whoops I forgot to mention in the list post -- we're planning to talk
> about this topic in the public ArchCom IRC meeting this Wednesday (21:00
> UTC / 2pm PDT).
>
> Already getting good feedback on the page, am updating it, and looking
> forward to more Thanks all. :)
>
> -- brion
>
> On Mon, Feb 13, 2017 at 9:28 AM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
>> I've got an early draft of some notes
>> <https://www.mediawiki.org/wiki/User:Brion_VIBBER/Compacting_the_revision_table_round_2>
>> for a restructuring of the revision table, to support the following:
>>
>> * making the revision table itself smaller by breaking large things out
>> * reducing duplicate string storage for content model/format, username/IP
>> address, and edit comments
>> * multi-content revisions ("MCR") - multiple Content blobs of different
>> types on a page, revisioned consistently
>>
>> There's also some ideas going around about using denormalized summary
>> tables more aggressively, perhaps changing where the indexes used for
>> specific uses live. For instance, a 'contribs' table with just the bits
>> needed for the index lookups for user-contribs, then joined to the other
>> tables.
>>
>> Initial notes at https://www.mediawiki.org/w
>> iki/User:Brion_VIBBER/Compacting_the_revision_table_round_2 -- I'll be
>> cleaning this up a bit more in response to feedback and concerns.
>>
>> If we go through with this sort of change, we'll need to carefully
>> consider the upgrade transition. We'll also need to make sure that all
>> relevant queries are updated, and that folks using the databases indirectly
>> (via tool labs, etc) are all able to cleanly handle the new fun stuff.
>> Feedback will be crucial here. :)
>>
>> Potentially we might split this into a couple transitions instead, or
>> otherwise make major changes to the plan. Nothing's set in stone yet!
>>
>> -- brion
>>
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki-Vagrant needs testers for Debian Jessie changes

[Brion Vibber]

Yesss! :) I'll test over the next few days.

-- brion

On Wed, Feb 15, 2017 at 2:53 AM, Bryan Davis  wrote:

> TL;DR See  and try the new
> jessie-migration branch.
>
>
> The Wikimedia Foundation has been working on converting the majority
> of its production server farm from Ubuntu to Debian for quite some
> time. In August of 2016 that project advanced to the point of actually
> converting the 252 servers running MediaWiki [0]. Today that work has
> largely been done and the vast majority of MediaWiki is running on
> Jessie.
>
> Now it's time (past time really) for MediaWiki-Vagrant to catch up. We
> have done this sort of thing once before with the switch from using
> Ubuntu Precise to Ubuntu Trusty as our base image [1] in mid-2014. We
> had fewer users then and only one supported virtual machine type. The
> switch from Trusty to Jessie is also slightly more complicated because
> the init system used is changing as well. This means that we have more
> Puppet code changes to make than last time and more people who will be
> impacted by the change.
>
> For a little over a month several of us have been working on the
> jessie-migration branch to get ready for the change over. I think we
> are at the point that we could change over, but I want to have a
> period of testing by a wider audience to see if we can iron out a few
> more common issues before forcing everyone to either upgrade or
> explicitly pin their MediaWiki-Vagrant virtual machines to the git tag
> for the last Trusty compatible build.
>
>
> == Testing the Jessie base image and Puppet profiles ==
>
> Its recommended to test with a fresh MediaWiki-Vagrant checkout so if
> things go badly you can easily switch back to your original install
> and keep working.
>
> $ git clone --recursive
> https://gerrit.wikimedia.org/r/mediawiki/vagrant mwv-jessie
> $ cd mwv-jessie
> $ git checkout jessie-migration
> $ ./setup.sh
> $ vagrant up
>
> You can run vagrant roles list -e -1 to get a nice list of the roles
> you have enabled on your normal Trusty VM install to copy over to your
> Jessie testing VM. This one-liner liner might even do it for you:
>
> $ cd mwv-jessie
> $ vagrant roles enable $(cd ../vagrant; vagrant roles list -e -1)
> $ vagrant provision
>
>
> Give things a try and report issues as children of the tracking task
> for this migration [2]. Barring major issues effecting many people, I
> would like to merge the jessie-migration branch with the master branch
> the week of March 13th.
>
>
> [0]: https://phabricator.wikimedia.org/T143536
> [1]: https://phabricator.wikimedia.org/rMWVAdc73e2bee9dff1e0755d15cfe
> 1376ee2dc6e141d
> [2]: https://phabricator.wikimedia.org/T136429
>
> Bryan
> --
> Bryan Davis  Wikimedia Foundation
> [[m:User:BDavis_(WMF)]]  Sr Software EngineerBoise, ID USA
> irc: bd808v:415.839.6885 x6855
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

and that'll be in #wikimedia-office on irc.freenode.net. :)

-- brion

On Tue, Feb 14, 2017 at 10:38 AM, Brion Vibber <bvib...@wikimedia.org>
wrote:

> Whoops I forgot to mention in the list post -- we're planning to talk
> about this topic in the public ArchCom IRC meeting this Wednesday (21:00
> UTC / 2pm PDT).
>
> Already getting good feedback on the page, am updating it, and looking
> forward to more Thanks all. :)
>
> -- brion
>
> On Mon, Feb 13, 2017 at 9:28 AM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
>> I've got an early draft of some notes
>> <https://www.mediawiki.org/wiki/User:Brion_VIBBER/Compacting_the_revision_table_round_2>
>> for a restructuring of the revision table, to support the following:
>>
>> * making the revision table itself smaller by breaking large things out
>> * reducing duplicate string storage for content model/format, username/IP
>> address, and edit comments
>> * multi-content revisions ("MCR") - multiple Content blobs of different
>> types on a page, revisioned consistently
>>
>> There's also some ideas going around about using denormalized summary
>> tables more aggressively, perhaps changing where the indexes used for
>> specific uses live. For instance, a 'contribs' table with just the bits
>> needed for the index lookups for user-contribs, then joined to the other
>> tables.
>>
>> Initial notes at https://www.mediawiki.org/w
>> iki/User:Brion_VIBBER/Compacting_the_revision_table_round_2 -- I'll be
>> cleaning this up a bit more in response to feedback and concerns.
>>
>> If we go through with this sort of change, we'll need to carefully
>> consider the upgrade transition. We'll also need to make sure that all
>> relevant queries are updated, and that folks using the databases indirectly
>> (via tool labs, etc) are all able to cleanly handle the new fun stuff.
>> Feedback will be crucial here. :)
>>
>> Potentially we might split this into a couple transitions instead, or
>> otherwise make major changes to the plan. Nothing's set in stone yet!
>>
>> -- brion
>>
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

Whoops I forgot to mention in the list post -- we're planning to talk about
this topic in the public ArchCom IRC meeting this Wednesday (21:00 UTC /
2pm PDT).

Already getting good feedback on the page, am updating it, and looking
forward to more Thanks all. :)

-- brion

On Mon, Feb 13, 2017 at 9:28 AM, Brion Vibber <bvib...@wikimedia.org> wrote:

> I've got an early draft of some notes
> <https://www.mediawiki.org/wiki/User:Brion_VIBBER/Compacting_the_revision_table_round_2>
> for a restructuring of the revision table, to support the following:
>
> * making the revision table itself smaller by breaking large things out
> * reducing duplicate string storage for content model/format, username/IP
> address, and edit comments
> * multi-content revisions ("MCR") - multiple Content blobs of different
> types on a page, revisioned consistently
>
> There's also some ideas going around about using denormalized summary
> tables more aggressively, perhaps changing where the indexes used for
> specific uses live. For instance, a 'contribs' table with just the bits
> needed for the index lookups for user-contribs, then joined to the other
> tables.
>
> Initial notes at https://www.mediawiki.org/wiki/User:Brion_VIBBER/
> Compacting_the_revision_table_round_2 -- I'll be cleaning this up a bit
> more in response to feedback and concerns.
>
> If we go through with this sort of change, we'll need to carefully
> consider the upgrade transition. We'll also need to make sure that all
> relevant queries are updated, and that folks using the databases indirectly
> (via tool labs, etc) are all able to cleanly handle the new fun stuff.
> Feedback will be crucial here. :)
>
> Potentially we might split this into a couple transitions instead, or
> otherwise make major changes to the plan. Nothing's set in stone yet!
>
> -- brion
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Provisional notes for proposed revision table restructuring

[Brion Vibber]

I've got an early draft of some notes

for a restructuring of the revision table, to support the following:

* making the revision table itself smaller by breaking large things out
* reducing duplicate string storage for content model/format, username/IP
address, and edit comments
* multi-content revisions ("MCR") - multiple Content blobs of different
types on a page, revisioned consistently

There's also some ideas going around about using denormalized summary
tables more aggressively, perhaps changing where the indexes used for
specific uses live. For instance, a 'contribs' table with just the bits
needed for the index lookups for user-contribs, then joined to the other
tables.

Initial notes at
https://www.mediawiki.org/wiki/User:Brion_VIBBER/Compacting_the_revision_table_round_2
-- I'll be cleaning this up a bit more in response to feedback and concerns.

If we go through with this sort of change, we'll need to carefully consider
the upgrade transition. We'll also need to make sure that all relevant
queries are updated, and that folks using the databases indirectly (via
tool labs, etc) are all able to cleanly handle the new fun stuff. Feedback
will be crucial here. :)

Potentially we might split this into a couple transitions instead, or
otherwise make major changes to the plan. Nothing's set in stone yet!

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress

[Brion Vibber]

Looks like the high-prio queue is back to normal, all is well. :) Please
continue uploading.

-- brion

On Thu, Feb 9, 2017 at 5:02 PM, James Hare <jamesmh...@gmail.com> wrote:

> Should people refrain from uploading videos at this time?
>
>
> On February 9, 2017 at 5:01:51 PM, Brion Vibber (bvib...@wikimedia.org)
> wrote:
>
> Quick update -- I tested some batch-reencoding of missing low-res files and
> have flooded the high-priority queue. :) Thanks for your patience as we get
> the balance right.
>
> I'll add a throttle to TimedMediaHandler's requeueTranscodes.php so I can
> "fire-and-forget" on the background jobs without disturbing things...
>
> -- brion
>
> On Thu, Feb 9, 2017 at 12:00 PM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
> > Ok, the video scaler queue change is now live: SD and low-resolution
> video
> > output runs on a new, high-priority queue, with HD video output and
> > transcodes of files over 15 minutes in length run on the old queue.
> >
> > Note that existing queued conversions will continue to run on the
> > low-priority queue until it runs out, but most new uploads from here out
> > will have web-ready output much faster.
> >
> > (Load on the servers may be unbalanced for a bit, and may require further
> > tweaking to best balance responsiveness and throughput, so we'll keep an
> > eye on the graphs.)
> >
> > -- brion
> >
> >
> > On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber <bvib...@wikimedia.org>
> > wrote:
> >
> >> We're in progress of deploying a small change to the video scalers[1],
> >> which should improve availability of newly uploaded video and audio
> files.
> >>
> >> The queue will now be split in two, one which covers low-resolution
> >> conversions for relatively short files, and one which covers long files
> and
> >> high-resolution conversions. When there's a flood of large uploads,
> other
> >> new files should still go through the high-priority queue while the
> large
> >> uploads and HD conversions may back up on the low-priority queue.
> >>
> >> The queue-runner side is updated now (done during the 'puppet SWAT'
> >> deployment window), with the MediaWiki side ready to roll around 19:00
> UTC
> >> (11am Pacific time, regular SWAT deployment window).
> >>
> >> [1] https://gerrit.wikimedia.org/r/#/c/336846/
> >>
> >> -- brion
> >>
> >> ___
> >> Wikivideo-l mailing list
> >> wikivide...@lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l
> >>
> >>
> >
> > ___
> > Wikivideo-l mailing list
> > wikivide...@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikivideo-l
> >
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress

[Brion Vibber]

Quick update -- I tested some batch-reencoding of missing low-res files and
have flooded the high-priority queue. :) Thanks for your patience as we get
the balance right.

I'll add a throttle to TimedMediaHandler's requeueTranscodes.php so I can
"fire-and-forget" on the background jobs without disturbing things...

-- brion

On Thu, Feb 9, 2017 at 12:00 PM, Brion Vibber <bvib...@wikimedia.org> wrote:

> Ok, the video scaler queue change is now live: SD and low-resolution video
> output runs on a new, high-priority queue, with HD video output and
> transcodes of files over 15 minutes in length run on the old queue.
>
> Note that existing queued conversions will continue to run on the
> low-priority queue until it runs out, but most new uploads from here out
> will have web-ready output much faster.
>
> (Load on the servers may be unbalanced for a bit, and may require further
> tweaking to best balance responsiveness and throughput, so we'll keep an
> eye on the graphs.)
>
> -- brion
>
>
> On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
>> We're in progress of deploying a small change to the video scalers[1],
>> which should improve availability of newly uploaded video and audio files.
>>
>> The queue will now be split in two, one which covers low-resolution
>> conversions for relatively short files, and one which covers long files and
>> high-resolution conversions. When there's a flood of large uploads, other
>> new files should still go through the high-priority queue while the large
>> uploads and HD conversions may back up on the low-priority queue.
>>
>> The queue-runner side is updated now (done during the 'puppet SWAT'
>> deployment window), with the MediaWiki side ready to roll around 19:00 UTC
>> (11am Pacific time, regular SWAT deployment window).
>>
>> [1] https://gerrit.wikimedia.org/r/#/c/336846/
>>
>> -- brion
>>
>> ___
>> Wikivideo-l mailing list
>> wikivide...@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l
>>
>>
>
> ___
> Wikivideo-l mailing list
> wikivide...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikivideo-l] Video transcode prioritization changes in progress

[Brion Vibber]

Ok, the video scaler queue change is now live: SD and low-resolution video
output runs on a new, high-priority queue, with HD video output and
transcodes of files over 15 minutes in length run on the old queue.

Note that existing queued conversions will continue to run on the
low-priority queue until it runs out, but most new uploads from here out
will have web-ready output much faster.

(Load on the servers may be unbalanced for a bit, and may require further
tweaking to best balance responsiveness and throughput, so we'll keep an
eye on the graphs.)

-- brion


On Thu, Feb 9, 2017 at 9:39 AM, Brion Vibber <bvib...@wikimedia.org> wrote:

> We're in progress of deploying a small change to the video scalers[1],
> which should improve availability of newly uploaded video and audio files.
>
> The queue will now be split in two, one which covers low-resolution
> conversions for relatively short files, and one which covers long files and
> high-resolution conversions. When there's a flood of large uploads, other
> new files should still go through the high-priority queue while the large
> uploads and HD conversions may back up on the low-priority queue.
>
> The queue-runner side is updated now (done during the 'puppet SWAT'
> deployment window), with the MediaWiki side ready to roll around 19:00 UTC
> (11am Pacific time, regular SWAT deployment window).
>
> [1] https://gerrit.wikimedia.org/r/#/c/336846/
>
> -- brion
>
> ___
> Wikivideo-l mailing list
> wikivide...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikivideo-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Video transcode prioritization changes in progress

[Brion Vibber]

We're in progress of deploying a small change to the video scalers[1],
which should improve availability of newly uploaded video and audio files.

The queue will now be split in two, one which covers low-resolution
conversions for relatively short files, and one which covers long files and
high-resolution conversions. When there's a flood of large uploads, other
new files should still go through the high-priority queue while the large
uploads and HD conversions may back up on the low-priority queue.

The queue-runner side is updated now (done during the 'puppet SWAT'
deployment window), with the MediaWiki side ready to roll around 19:00 UTC
(11am Pacific time, regular SWAT deployment window).

[1] https://gerrit.wikimedia.org/r/#/c/336846/

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] More useful Puppet docs for MediaWiki-Vagrant

[Brion Vibber]

Nice! I like that by expanding the 'source' option I can see dependencies
as well as what it's actually doing in terms of config etc. The textual
descriptions of the roles are mostly pretty vague so far; should we expand
them for instance to describe what will be installed and how to go about
tweaking if tweaking is needed, or just leave that to the source view?

-- brion

On Tue, Jan 10, 2017 at 1:57 PM, Bryan Davis  wrote:

> Hashar did some magic and replaced our use of `puppet doc` with yard
> which is now generating prettier and more useful documentation of the
> roles and other Puppet components in the MediaWiki-Vagrant project. Go
> check them out if you are interested:
> . Try searching for
> "role::" to get the list of all roles.
>
> Bryan
> --
> Bryan Davis  Wikimedia Foundation
> [[m:User:BDavis_(WMF)]]  Sr Software EngineerBoise, ID USA
> irc: bd808v:415.839.6885 x6855
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] No ArchCom IRC meeting this week

[Brion Vibber]

ArchCom IRC meeting was canceled for this week; sorry for the lack of
announcement!

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ParserAfterParse not called by VisualEditor?

[Brion Vibber]

On Sat, Oct 8, 2016 at 5:37 PM, James HK 
wrote:

> Hi,
>
> > VE's edits should be (indirectly) going through the action=edit API.
>
> I'm not sure this answer is very helpful.


To translate: "it should work exactly the same as edits made with the API",
or to elaborate further: "yes, a ParserAfterParse hook should be called at
parse time after an edit made with VisualEditor, though it won't be
reflected while you're looking at Visual Editor".

There might, however, be something about how the hook is being used that
causes it not to do what is expected when editing is done via the API
versus when it's done using the classic editor form.

For instance, if the hook assumes it's happening during action=edit and
tries to pull data from $_REQUEST or $wgRequest, it will very likely fail
in this and other circumstances when parsing happens separately from an
edit, or happens during an edit but with different parameters and
circumstances than the classic HTML form editor.

Without knowing what the hook is doing, it's hard to say whether something
else might be going wrong.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[Brion Vibber]

On Thu, Sep 1, 2016 at 11:11 AM, bawolff  wrote:

> Yes, we certainly do have issues with follow-through on summit decisions.
>

*nod*


> For me personally, I've found the dev summits mostly useful as a
> community building type thing (For the MediaWiki developer community).
> As a remotee (Or at other various points in time, as a volunteer), its
> rare I actually see everyone in real life. The dev summit provides a
> venue to actually interact with everyone. While it may not actually be
> the best at resolving architectural issues, I feel like it helps me
> understand where everyone is coming from.
>
> In particular, I find that the dev summit is more effective for this
> purpose than hackathons, as the unstructured nature of hackathons tend
> to get people clumping in groups that already know each other. The dev
> summit on the other hand better provides for cross-pollination of
> ideas in my experience. (Don't get me wrong, I love hackathons too,
> just for different reasons).
>

That's a very good point! It may be good to have distinct spaces for these
environments, and 'hackathon' type events tend to have a different focus on
bringing people in with shorter-term projects.

I think we may want to look at ways to "boost signal" on input to and
output from MWDS. Even if we don't have as much physical cross-pollination
between devs and users as we could co-hosting with a bigger, less
dev-focused event like Wikimania, it's important to retain that focus on
user needs -- both as input to make technical decisions based on, and as
output when we're reporting back what we expect to work on and if/how we
can either assign resources within WMF, WMDE etc or if we need help from
outside and how to organize that.


> However, use-cases and users is why we're here, so I'm certainly not
> opposed to that focus. I just hope we continue to retain this as an
> event that's more talky and less hacky, as I feel that's where a lot
> of the uniqueness of the event came from.
>

Yeah, I get that. Thanks for bringing up the positive side of less-hacky. :)

One aspect of the first MediaWiki architecture summit that I really
> liked but has been mostly lost, was inviting non-Wikimedia mediawiki
> users. They're a group that has use-cases that we don't often hear
> about, and provide a unique perspectives. Although I suppose its not
> surprising that their involvement has kind of been lost. I would love
> to see them come back, although I'm not exactly holding my breath for
> that.
>

*nod* Some of those use-cases are great for potential Wikimedia-world uses
too; we shouldn't forget those "other" users. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Opening up MediaWiki dev summit in January?

[Brion Vibber]

On Thu, Sep 1, 2016 at 10:32 AM, Brian Wolff  wrote:

> Last year there was an attempt to sort of do this (mostly by extending the
> word "developer" to mean new things). Largely those types of people didnt
> attend (although there were a few exceptions), however I remember being
> left wondering if they did attend, what would they do? It seems to me most
> sessions were about architecture design decisions that actually didnt
> affect anyone not working on the code (ie we were going to make the user
> visible feature either way, the question was do we use method x or method y
> in the backend). With that in mind. Otoh, its entirely possible that some
> of the sessions i didnt attend were more applicable to these groups.
>
> With that in mind are you proposing the focus of event also change? Or do
> you think that these groups would be interested in it as is?
>

Yes, I think we *should* provide a focus for the event, and that the focus
should be on users, use cases, and what we as developers need to do to
achieve those things.

In my opinion we haven't had a strong focus to the event in the past, and
it's limited what we accomplish there to largely making a set of technical
presentations and having a few discussions that either don't produce a
decision or don't have much affect on what actually happens after the
summit.

(I'd be very interested also in some feedback on things that *have* worked
well at MWDS in the past, as I'd love to encourage anything that has been
productive! But I think we've not been successful in an architectural focus
so far.)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Opening up MediaWiki dev summit in January?

[Brion Vibber]

The last couple years we've done a big MediaWiki Dev Summit in January,
around the time of the Wikimedia Foundation all-hands meeting. Invitations
have been fairly broad to known developers, but there's a very strong
feeling that newbies, non-technical people, and in general *the people
MediaWiki is created and maintained for* are not welcome.

I think we should change this.

I would really like a broader MediaWiki Dev Summit that asks our users to
participate, and asks "developers" to interact with them to prioritize and
work on things that really matter to them.

I want template authors, Lua module authors, template users, power editors,
folks working on the lines of defense for vandalism patrol and copyvio
checking. I want people with opinions on discussion systems. I want people
who have been editing for years and have experience with what works and
what doesn't. I want people who wish they could edit but have a bad
experience when they try, and want to share that with us so we can help
make it better.

Thoughts?

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [WikimediaMobile] Mobile site is now lazy loading images

[Brion Vibber]

Awesome! This will also make it easier to experiment with alternate
delivery formats: WebP, direct SVG usage, etc that depend on knowing
whether there's client-side support. I'll fire up some more research... :D

-- brion

On Tue, Aug 23, 2016 at 2:20 PM, Jon Robson  wrote:

> FYI after much experimentation, research and testing the mobile site has
> been lazy loading images [1] since Thursday 18th August. This means if you
> do not see an image you will not download it. We have taken care to ensure
> users without JavaScript can still view images and that most users will
> barely notice the difference.
>
> We are currently crunching the data this change has made and we plan to
> write a blog post to reporting the results.
>
> In our experiments on Japanese Wikipedia we saw a drop in image bytes per
> page view by 54% On the Japanese Japan article bytes shipped to users
> dropped from 1.443 MB to 142 kB.
>
> This is pretty huge since bytes equate to money [3] and we expect this to
> be significant on wikis where mobile data is more expensive. In a nutshell
> Wikipedia mobile is cheaper.
>
> As I said blog post to follow once we have more information, but please
> report any bugs you are seeing with the implementation (we have already
> found a few thanks to our community of editors).
>
> ~Jon
>
> [1] https://www.mediawiki.org/wiki/Reading/Web/Projects/
> Performance/Lazy_loading_images
> [2] https://www.mediawiki.org/wiki/Reading/Web/Lazy_loading_
> of_images_on_Japanese_Wikipedia
> [3] https://whatdoesmysitecost.com/
>
>
>
> ___
> Mobile-l mailing list
> mobil...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Better search results on wiki via TextCat

[Brion Vibber]

That is super awesome! :D

-- brion

On Tue, Jul 19, 2016 at 6:41 PM, Deborah Tankersley <
dtankers...@wikimedia.org> wrote:

> We're happy to announce that after numerous tests and analyses[1] and a
> fully operational demo[2], the Discovery Team is ready to release
> TextCat[3] into production on wiki.
>
> What is TextCat? It detects the language that the search query was written
> in which allows us to look for results on a different wiki. TextCat is a
> language detection library based on n-grams[4]. During a search, TextCat
> will only kick in when the following three things occur:
> 1. fewer than 3 results are returned from the query on the current wiki
> 2. language detection is successful (meaning that TextCat is reasonably
> certain what language the query is in, and that it is different from the
> language of the current wiki)
> 3. the other wiki (in the detected language) has results
>
> Our analysis of the A/B test[5] (for English, French, Spanish, Italian and
> German Wikipedia's) showed that:
>
> "...The test groups not only had a substantially lower zero results rate
> (57% in control group vs 46% in the two test groups), but they had a higher
> clickthrough rate (44% in the control group vs 49-50% in the two test
> groups), indicating that we may be providing users with relevant results
> that they would not have gotten otherwise."
>
>
> This update will be scheduled for production release during the week of
> July 25, 2016 on the following Wikipedia's:
>
>- English [6]
>- German [7]
>- Spanish [8]
>- Italian [9]
>- French [10]
>
> TextCat will then be added to this next group of Wikipedia's at a later
> date:
>
>- Portugese[11]
>- Russian[12]
>- Japanese[13]
>
> This is a huge step forward in creating a search mechanism that is able to
> detect - with a high level of accuracy - the language that was used and
> produce results in that language. Another forward-looking aspect of TextCat
> is investigating a confidence measuring algorithm[14], to ensure that the
> language detection results are the best they can be.
>
> We will also be doing more[15] A/B tests using TextCat on non Wikipedia
> sites, such as Wikibooks and Wikivoyage. These new tests will give us
> insight into whether applying the same language detection configuration
> across projects would be helpful.
>
> Please let us know if you have any questions or concerns, on the TextCat
> discussion page[16]. Also, for screenshots of what this update will look
> like, please see this one[17] showing an existing search typed in on enwiki
> in Russian "первым экспериментом" and this one[18] for showing what it will
> look like once TextCat is in production on enwiki.
>
>
> Thanks!
>
>
> [1] https://phabricator.wikimedia.org/T118278
> [2] https://tools.wmflabs.org/textcatdemo/
> [3] https://www.mediawiki.org/wiki/TextCat
> [4] https://en.wikipedia.org/wiki/N-gram
> [5]
>
> https://commons.wikimedia.org/wiki/File:Report_on_Cirrus_Search_TextCat_AB_Test_-_Language_Detection_on_English,_French,_Spanish,_Italian,_and_German_Wikipedias.pdf
> [6] https://en.wikipedia.org/
> [7] https://de.wikipedia.org/
> [8] https://es.wikipedia.org/
> [9] https://it.wikipedia.org/
> [10] https://fr.wikipedia.org/
> [11] https://pt.wikipedia.org/
> [12] https://ru.wikipedia.org/
> [13] https://ja.wikipedia.org/
> [14] https://phabricator.wikimedia.org/T140289
> [15] https://phabricator.wikimedia.org/T140292
> [16] https://www.mediawiki.org/wiki/Talk:TextCat
> [17]
> https://commons.wikimedia.org/wiki/File:Existing-search_no-textcat.png
> [18] https://commons.wikimedia.org/wiki/File:New-search_with-textcat.png
>
> --
> Deb Tankersley
> Product Manager, Discovery
> IRC: debt
> Wikimedia Foundation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What's the "correct" content model when rev_content_model is NULL?

[Brion Vibber]

On Tuesday, July 12, 2016, Daniel Kinzler 
wrote:

> Am 12.07.2016 um 18:00 schrieb Rob Lanphier:
> > On Tue, Jul 12, 2016 at 1:40 AM, Daniel Kinzler <
> daniel.kinz...@wikimedia.de 
> >> The original design of ContentHandler used integer IDs for content
> models
> >> and formats in the DB. A mapping to human readable names is only needed
> >> for logging and error messages anyway.
> >
> > This oversimplifies things greatly.  Integer IDs need to be mapped to
> some
> > well-specified, non-local (global?) identifier for many many purposes
> > (reading exports, writing exports, reading site content, displaying site
> > content for many contexts, etc)
>
> Yea, sorry. That we only need this for logging is what I assumed back
> then. Not
> exposing the numeric ID at all, and using the canonical name in dumps, the
> API,
> etc, avoids a lot of trouble (but doesn't come free).


Yes, numeric ids are internal and never to be exposed ideally. We should've
done same wth namespaces but got dragged into compat hell. :)


>
> > We need to put a lot of thought into content model management generally.
> > This statement implies managing content models outside of the database is
> > easy.
>
> Well, it's the same as namespaces: they are easy to set up, but also too
> easy to
> change, so it's easy to create a mess...
>
> As explained in my earlier response, I now realized that content models
> differ
> from namespaces in that they are not really configured by people, but
> rather
> registered by extensions. That makes it a lot less awkward to have them in
> the
> database. We still have to agree on a good trigger for the registration,
> but it
> doesn't seem to be a tricky issue.


Yeah an auto insert if needed is good in theory, though I worry about write
contention on the central mapping table. If no write locks kept in the
common case of no insertion needed then I think the ideas proposed should
work.


>
> What we still need to figure out is how to solve the chicken-and-egg
> situation
> with Multi-Content-Rev. At the moment, I'm thinking this might work:
>
> * introduce content model (and format) registry in the DB, and populate it.
> * leave page and revision table as they are for now.
> * introduce slots table, use the new content_model (and content_format)
> table.
> * stop using the content model (and format) from the page and revision
> tables
> * drop the content model (and format) from the page and revision tables
>
> Does that sound liek a good plan? Let's for a moment assume we can get
> slots
> fully rolled out by the end of the year.


This sounds good to me - lets us introduce a more space efficient model
mapping and drop the extra fields from page and rev later.

-- brion


>
> --
> Daniel Kinzler
> Senior Software Developer
>
> Wikimedia Deutschland
> Gesellschaft zur Förderung Freien Wissens e.V.
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Crediting the work of our community members

[Brion Vibber]

On Mon, Jul 4, 2016 at 9:06 AM, Jon Robson  wrote:

> Sorry I dropped the ball on this.
> I've created the following 2 actionable tasks:
> https://phabricator.wikimedia.org/T139300
> https://phabricator.wikimedia.org/T139301
>
> I hope we can reach a decision somewhat promptly with this.
>

Jon, I like both of these.

First one: deciding what to do with CREDITS as a format process.
On some of my smaller projects I've started adding more patch contributors
into a main CREDITS file, which I think is a lot better than trying to
distinguish who's More Creditable. I lean towards putting in everyone who
submits a patch, in alpha order.

(I will say when I was asked to add my name to the AUTHORS file on the
'emscripten' cross-compiler for what I felt was a small patch, it made me
feel very appreciated -- and made me want to contribute more!)

I think we should either adopt that, or at least adopt _some_ consistent
process, and update the file to match based on what we can see.


Second one: removing the @author attributions on individual files. While it
can be nice to go back and say "hey it says I wrote this code!" it's also
very misleading to people who see attributions from 10 years ago and then
ask those people for help, and we have to say "sorry it's been completely
refactored since, but I'll try!" :) Major refactors may, or may not, update
the @authors bits.

I suspect either we should remove them, or have a rule that we're much more
amenable about including patchers and refactorers in them.


These should probably both go through our ArchCom RfC process, which we're
trying to get more involvement in both from WMFers and others.

-- brion



>
>
> On Tue, Jun 7, 2016 at 8:14 PM, Scott MacLeod  wrote:
> > Rob and Wikimedians,
> >
> > To further credit the work of Wikimedia community members, could we
> explore
> > using the Douglas Adams' SQID example (which Markus shared with the
> > Wikidata list recently ... [Wikidata] SQID evolved again: references
> > http://tools.wmflabs.org/sqid/#/view?id=Q42 ) ... and build upon this
> > incorporating the Wikipedia user pages (and your above examples) a
> > Wikipedia crediting process anticipating/planning for all 11 billion
> people
> > (an estimate from Swedish statistician Hans Rosling and many others) in
> all
> > 8,000 languages. (CC World University and school is planning in parallel
> to
> > be in all 8,000 languages and plan for all people on earth by 2100, as
> well
> > as seek to build in Bitcoin and Blockchain in conjunction with developing
> > best STEM CC OpenCourseWare centric law schools in all countries' main
> > languages, even as WUaS develops CC university degrees accrediting  on CC
> > MIT OCW in 7 languages and CC Yale OYC).
> >
> > This would potentially lead to further planning for integrating
> > Wikidata/Wikibase via SQID with Wikitech/Wikimedia/Wikipedia community
> > members' contributions and in many languages.
> >
> > Cheers, Scott
> >
> >
> >
> > On Tue, Jun 7, 2016 at 11:14 AM, Rob Lanphier 
> wrote:
> >
> >> On Tue, Jun 7, 2016 at 9:19 AM, Andre Klapper 
> >> wrote:
> >> > Is there a Phabricator task [associated with MediaWiki CREDITS file
> >> membership] so this topic does not get forgotten?
> >>
> >> Not that I'm aware of.  It's easy to get lost looking through the
> >> various attempts to objectively characterize contributions (he says,
> >> just emerging from the fog of doing so himself).  Here's a few places
> >> a person could go:
> >> * 
> >> * 
> >> * 
> >> * 
> >>
> >> ...and that's hardly comprehensive.  The "productivity of mediawiki
> >> developers" thread from April[1] probably has some other sources I've
> >> missed.  If I were to spend more time on this, I would start looking
> >> for the Phab tickets associated with the stats on Korma.
> >>
> >> I concur with Jon that we should endeavor to move to a more objective
> >> (and ideally, more automated) mechanism for acknowledgement, so that
> >> we don't have to rely on contributors confidently declaring that they
> >> deserve acknowledgement.
> >>
> >> Rob
> >> [1]
> >>
> http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/86127
> >>
> >> ___
> >> Wikitech-l mailing list
> >> Wikitech-l@lists.wikimedia.org
> >> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >>
> >
> >
> >
> > --
> >
> > - Scott MacLeod - Founder & President
> > - 415 480 4577
> > - http://scottmacleod.com
> > - Please donate to tax-exempt 501 (c) (3)
> > - World University and School
> > - via PayPal, or credit card, here -
> > - http://worlduniversityandschool.org
> > - or send checks to
> > - PO Box 442, (86 Ridgecrest Road), Canyon, CA 94516
> > 

[Wikitech-l] Exploring VP9 as a progressive still image codec

[Brion Vibber]

I spent some time between projects today exploring the idea of progressive
image decoding using the VP9 video codec -- sort of a mashup of progressive
JPEG and WebP.

Like a progressive JPEG, each resolution step (a separate frame of the
"video") encodes only the differences from the previous resolution step.
Like WebP, it's more space-efficient than the ancient JPEG codec.

This sort of technique might be useful for lazy-loading images in our
modern internet, where screen densities keep going up and network speeds
can vary by a factor of thousands. On a slow network the user sees
immediate feedback during load, and on a fast network they can reach full
resolution quickly, still in less bandwidth than a JPEG. And since JS would
have control of loading, we can halt cleanly if the image scrolls
offscreen, or pick a maximum resolution based on actual network speed.


Detail notes on my blog:
https://brionv.com/log/2016/06/14/exploring-vp9-as-a-progressive-still-image-codec/

Sample page if you just want to look at some decoded images at various
resolutions (loading not optimized for slow networks yet!):
https://media-streaming.wmflabs.org/pic9/


It looks plausible, and should be able to use native VP9 decoding in
Firefox, Chrome, and eventually MS Edge in some configurations with a
JavaScript fallback for Safari/etc. Currently my demo just plops all the
frames into a single .webm, but to avoid loading unneeded high-resolution
frames they should eventually be in separate files.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] File handler options & InstantCommons (was Re: Update from the Perf Team)

[Brion Vibber]

On Sun, May 29, 2016 at 11:22 AM, Gergo Tisza <gti...@wikimedia.org> wrote:

> On Sun, May 29, 2016 at 7:08 PM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
> > We could change all that if there was a stable enough URL/linking
> > API... Can probably fake it for common cases and supported handlers by
> > generating a URL like we would for a local 404-handler repo; my main
> > concern is for future expansion where we add more fancy types on Commons
> > where the local site may not have an extension installed locally.
>
>
> Not sure if trying to use an unhandled file type via InstantCommons makes
> sense; the image parameters in the wikitext and the image parameters in the
> URL (or imageinfo API or hypothetical linking API) don't necessarily use
> the same syntax; what would transform that? (Granted, you get similar
> problems when parameter transformation is handled by the local extension
> and there is a version mismatch between the local and remote installation.)
>

Yep, it all comes back to T66214 <https://phabricator.wikimedia.org/T66214>
doesn't it. :D



> > (Iframe embedding as a fallback for unknown handlers sounds possible,
> > could expose
> > a frame URL in the imageinfo data.)
> >
>
> In general that would be a nice idea (see T27854
> <https://phabricator.wikimedia.org/T27854>). Specifically for embedding
> unknown file types in MediaWiki though, we would probably have to introduce
> some standard key-value syntax for image parameters first, instead of the
> current mess where it's parsed by an arbitrary set of regexes defined by
> the media handlers and generated in a similarly flexible way. And then you
> still have the problem of handling unknown media types in VisualEditor
> which would need some kind of parameter metadata at least.
>

Indeed. iframe embedding of video might be a good place to start for
unknown types; since we already have an iframe embedding system for video
in TimedMediaHandler, it should be easy to adapt it as a provider so that
sites can use videos from Commons without installing TimedMediaHandler
locally.


Key issues for options:

* separate the layout parameters from the file rendering parameters (eg
things like 'thumb' or 'frame' affect the layout, but the file
handler/renderer only cares about the resulting size)
* define standardish keys for file options already in use
* if extra file options are needed, have a standard option for extended
options!
* define that extra options will not be localizable or have magical
regexes; they should just be key-value pairs with fixed, predictable key
names
* define a standard way to pass file options to the iframe


Existing file options:
* 'width' and 'height' are implied through thumb/frame/upright/default
size, or explicitly via /\d+px/ or /\d+x\d+px' etc regexes defined in core.
* SvgHandler uses 'lang'; regex defined in core.
* PdfHandler and PagedTiffHandler use 'page'; regex defined in core.
* PagedTiffHandler uses 'lossy' switch to allow overriding the default
thumbnail type (JPG vs PNG); regex defined in extension
* TimedMediaHandler adds 'start', 'end', and 'thumbtime'; regex defined in
extension

There might be a couple more floating around out there, but that should
cover those already in use in Wikimedia content. For compatibility we'd
need to keep the core regexes, and might or might not want to move the
'lossy' and 'start/end/thumbtime' ones to core.


I can imagine some extended attributes I'd like for video:
* 'loop'
* 'mute'
* 'autoplay' (only allowed with 'loop' and 'mute', for GIF-like display)

for zoomable panoramic images (flat or spherical):
* 'zoom' - initial zoom level
* 'center' - coordinates of initial centering (what coord system? separate
x/y params or just one? should spherical use lat/lon? etc)

for viewing a 3d model:
* 'zoom' - just like panoramas
* 'distance' - viewer distance from origin point; this is different from
zoom!
* 'center' - origin point/center of view, extended to 3 dimensions (for
instance to concentrate on a portion of a larger model)

Let's imagine a file type that uses everything! An animated 3d model that's
localizable, and has multiple selectable "pages" of which we show one:

  [[File:Cool file type.mytype|

thumb
|alt=A bouncing ball changes colors while a calendar in background
counts off seasons in French.

|640x480px

|lang=fr
|page=3

|start=25
|thumbtime=27.5
|end=35
|lossy=1

|zoom=1.5
|center-x=0.5
|center-y=0.667
|center-z=0.25
|distance=0.75
|loop=1
|muted=1
|autoplay=1

|Check out this an awesome [[3d model]] that's [[animation|animated]]!
  ]]


There is some ambiguity with accepting any unknown key followed by '='...
is [[File:Equation.mytype|E=mc^2]] using "E=mc^2" as a caption or passing a
parameter &quo

Re: [Wikitech-l] [Engineering] Update from the Perf Team

[Brion Vibber]

On Saturday, May 28, 2016, Brian Wolff  wrote:

> Pages with large number of thumbnails tends to be a slow path in the
> parser. This is even more true for people using instant commons. I dont
> know for sure, but I imagine it could be improved by batching the checks
> for finding image width/height (that might be a difficult change to make
> though).


Batching requests sounds possible but would need to retool some things. The
good news is the file links should all be visible in the expanded
preprocessor DOM, so it should be possible to walk the tree, make a
list, batch the file lookup requests and then pass the batch cache into the
Linker functions.

Uh, that's assuming I'm not misremembering the order of operations with
respect to template expansion. :)


IIRC, InstantCommons is slightly trickier as you also need to make multiple
API calls to fetch thumbnail URLs, made even worse by needing to fetch
multiple resolutions for srcset. This gets real nasty if there are a lot of
files and the thumbnail ref data isn't cached...

We could change all that if there was a stable enough URL/linking
API... Can probably fake it for common cases and supported handlers by
generating a URL like we would for a local 404-handler repo; my main
concern is for future expansion where we add more fancy types on Commons
where the local site may not have an extension installed locally. (Iframe
embedding as a fallback for unknown handlers sounds possible, could expose
a frame URL in the imageinfo data.)

-- brion


>
> --
> Bawolff
>
> On Saturday, May 28, 2016, Pine W >
> wrote:
> > Thank you for the update. FWIW, a page that I load frequently that takes
> a
> > long time is Commons' featured picture candidates. My guess is that
> > starting with the implementation of HHVM and with smaller improvements
> > thereafter, the page load time had been reduced by more than 50%, which
> is
> > impressive and much appreciated.
> >
> > Pine
> > On May 27, 2016 15:28, "Ori Livneh" >
> wrote:
> >
> >> Hello,
> >>
> >> Here's what the performance team has been up to.
> >>
> >> == Dashboards & instrumentation ==
> >> We spent time instrumenting software and curating displays of
> performance
> >> data. We have several new dashboards to share with you:
> >>
> >> * Global edit rate and save failures (new)
> >>   https://grafana.wikimedia.org/dashboard/db/edit-count
> >>
> >> * Performance metrics (revamped)
> >>   https://grafana-admin.wikimedia.org/dashboard/db/performance-metrics
> >>
> >> * Page load performance
> >>   https://grafana.wikimedia.org/dashboard/db/navigation-timing
> >>
> >>   ...by continent:
> >>
> https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-continent
> >>   ...by country  :
> >>
> https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-geolocation
> >>   ...by browser  :
> >> https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-browser
> >>
> >> * We found that certain browsers were reporting wildly inaccurate timing
> >> data and skewing our summary performance metrics, and reacted by
> validating
> >> browser metric data more strictly against Navigation Timing API specs.
> >>
> >>
> >> == ResourceLoader ==
> >> ResourceLoader is the MediaWiki subsystem responsible for loading CSS,
> >> JavaScript, and i18n interface messages for dynamic site features. It is
> >> critical to site performance. Changes to ResourceLoader are focused on
> >> reducing backend response time, ensuring we make efficient use of the
> >> browser cache, and reducing time to first paint (the time it takes any
> >> content to appear). This work is led by Timo Tijhof.
> >>
> >> * The "/static/$mwBranch" entry point has been deprecated and removed in
> >> favor of wmfstatic - a new multiversion-powered entrypoint accessed via
> >> "/w" (via RewriteRule)
> >>   https://phabricator.wikimedia.org/T99096
> >>
> >> * Restricting addModuleStyles() to style-only modules (ongoing)
> >>   https://phabricator.wikimedia.org/T92459
> >>
> >> * Startup module check is now based on a feature test instead of browser
> >> blacklist
> >>   https://phabricator.wikimedia.org/T102318
> >>
> >>
> >> == WebPageTest ==
> >> Page load performance varies by browser, platform, and network. To
> >> anticipate how code changes will impact page performance for readers and
> >> editors, we use WebPageTest (
> >> https://wikitech.wikimedia.org/wiki/WebPageTest),
> >> a web performance browser automation tool. WebPageTest loads pages on
> >> Wikimedia wikis using real browsers and collects timing metrics. This
> work
> >> is led by Peter Hedenskog.
> >>
> >> * We now generate waterfall charts for page loads on Firefox. Previously
> we
> >> were only able to produce them with Chrome.
> >>
> >> * We tracked downs two bugs in WebPageTest that caused it to report an
> >> incorrect value for time-to-first-byte and reported them upstream.
> >>   

Re: [Wikitech-l] Reviving SVG client-side rendering task

[Brion Vibber]

On Fri, May 13, 2016 at 1:28 PM, Jon Robson  wrote:

> Thanks for starting this conversation Brion!
>
> On mobile we are constantly tackling the trade offs between data
> shipped (cost for end user) and quality. Hopefully we'll have a better
> solution for this by the end of the quarter which will allow us to
> reconsider srcset usage.


Note that srcset usage for raster images is a different issue from native
SVG, in that the higher-resolution renderings *do* require more bandwidth
than the lower-resolution images -- especially for photos.


> That said relying on srcset to render SVGs
> well seems like a hack to me and thus I'd rather we fixed the problem
> at the source.
>

The problem is the intersection of:
* some older browsers do not support SVG in 
*  provides no system for automatic fallback based on the browser's
format support
*  does provide that, but the spec is not finalized, browser
support is not finalized, and it may have implications for scripting and
styling by introducing a new wrapper element

Using  with PNG in the src and a "1x" SVG in the srcset resolves the
first case by feeding a compatible rasterization to old browsers, solves
the second problem by using srcset support as a proxy check for SVG support
(native srcset support implies SVG support in all known cases), and avoids
the third problem by not using a new, different element for images.

We could jump straight to , but it'll have bigger QA implications
in that it may affect styling and JS gadgets that look for  elements
specifically.

 would be nice for other things though -- future support of WebP
for alternate, more bandwidth-efficient thumbnails, or if we start doing
fancier types of responsive images based on "art direction" usages where a
narrow column gets a different image than a wide window.



> A few thoughts
> * The ResourceLoaderImage module is being used widely to generate SVG
> icons with png fallbacks. I'd be interested in seeing if we can use
> this in some way for optimising SVGs and removing meta data.
>

Definitely yes! Just as we minify CSS and JS source, we should make sure
the SVG icons are efficiently packed as well. (There's usually good
opportunity to do that at commit time, but it's nice to be able to include
comments and such in the source.)



> * I wonder if there is an opportunity here to use BetaFeatures and
> mobile beta to get a sense of performance impact of these changes?
> This would allow people to opt in and for us to crowdsource feedback
> in a safe enviroment.
>

Yes, I am suggesting exactly that. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Reviving SVG client-side rendering task

[Brion Vibber]

On Wed, May 11, 2016 at 11:10 AM, Chris Steipp <cste...@wikimedia.org>
wrote:

> On Thu, May 5, 2016 at 6:49 AM, Brion Vibber <bvib...@wikimedia.org>
> wrote:
>
> >
> > And then there are long term goals of taking more advantage of SVGs
> dynamic
> > nature -- making things animated or interactive. That's a much bigger
> > question and has implementation and security issues!
>
>
> Sorry for the late response (and if this is covered in one of the linked
> bugs), but getting back to this-- are you envisioning SVG's inlined into
> the html?


Maybe someday that might be interesting for interactive bits, but not in
the short term.


> Or would they be served from a domain like upload.wm.o, like we
> currently use for uploaded images?
>

Yes for now.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Mediawiki-i18n] Providing the effective language of messages

[Brion Vibber]

Tagging language and directionality on fallback entries sounds reasonable,
and not a huge addition.

I'm not sure why the fallbacks are cached in the main body instead of
letting them through to the backing language's cache, but if that's a
necessary or useful choice I've no reason to hop in and change it. I also
am a bit mystified why we moved from a filesystem-backed database to a
giant in-memory cache that requires deserializing the entire language cache
on every request (if I'm reading correctly) but apparently that performs
better than I think it should, so take me with a grain of salt... :)

I would recommend getting someone from Performance to weigh in on this to
confirm the proposed solution has no surprises. Probably best to have at
least a preliminary patch which can be run through some paces...

More generally, we should capture the other note in the mediawiki-i18n
thread about having the templating system pick up language and
directionality and fill out the necessary buts for tagging and isolation on
the HTML end.

We may want to think about a class to represent a bit of text or HTML
that's tagged with a language and an isolation factor (not sure if
necessary), which can be mixed and matched with Message objects and passed
into Html class generators or template things. And of course, equivalent on
the JS side.

-- brion
Hi everyone,

Niklas brought this message[1] to my attention as something that
probably deserves more attention than it has gotten, and I trust he's
correct.  What he said back in April: "I added
wikitech-l to CC in hopes that people who have worked on localisation
cache more recently would comment on whether [Adrian's proposed option
to not merge
messages in LocalisationCache] would make more sense nowadays."

Adrian: assuming Niklas and I are correct, my suggestion for moving
this forward would be to turn your design thoughts into an
ArchCom-RFC[2] for more explicit consideration by ArchCom.  My attempt
at abstract for those who haven't followed this:

Adrian was trying to figure out how to output pages that need to have
multiple languages in a single page, which becomes difficult when
fallbacks are missing.  It results in some oddball behavior where
placeholder text is output with incorrect i18n attributes in the
surrounding div.  He provided several alternatives for how to solve
the problem in his mail to mediawiki-i18n.  Niklas replied, providing
the "if we stick with the status quo" answer (tag message strings in
LocalisationCache with the correct language), and then is trying to
figure out if the status quo makes the right space vs speed tradeoff
given the quantity of languages and messages we have in 2016.

Adrian & Niklas: did I get the gist of it?

Rob

[1]: The April 2016 mediawiki-i18n thread: "Providing the effective
language of messages"
 <
https://lists.wikimedia.org/pipermail/mediawiki-i18n/2016-April/thread.html#1059
>
[2]: My unofficial guide on how to turn something into an ArchCom-RFC:


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ResourceLoader addModuleStyles() issues

[Brion Vibber]

On Monday, May 9, 2016, Krinkle <krinklem...@gmail.com> wrote:

> On Wed, May 4, 2016 at 1:43 PM, Brion Vibber <bvib...@wikimedia.org
> <javascript:;>> wrote:
>
> > Quick notes on the migration path:
> >
> >
> > *Cached page HTML*
> >
> > HTML cached under the old regime would still be served for a while, but
> > ResourceLoader would load the newer style. I *think* that should work as
> > long as the new versions of the modules that were included before still
> > list the style-only modules as dependencies, assuming load.php doesn't
> > throw an exception when the code-plus-style modules are requested.
> >
> > If load.php would throw an exception in this case, then that's going to
> be
> > a tricky problem to figure out for end-users, who will be presented with
> > unstyled pages and no visible error message.
> >
>
>
> The restriction added for T92459 only applies to addModuleStyles() calls.
> No changes to load.php HTTP response behaviour.
>
> There is one change required before we enable the new restriction: Convert
> internal details of how user/site/gadget modules are registered and loaded.
> These will be forward and backward compatible.
>
> The new rule doesn't enable conceptually new features. We are currently
> avoiding a certain kind of relationship for performance reasons (dynamic
> modules depending on style modules) - which we won't need to avoid any
> longer. We'll need to wait for cache to roll over in between two steps to
> avoid styles from going missing.
>
> Currently:
> * Output A:  (loads site styles), mw.loader.load('site')
> - (loads site scripts and styles)
>
> Step 1:
> * Create 'site.styles'
> * Output A behaviour unchanged
> * New Output B: , site.styles=ready,
> mw.loader.load('site') - (loads site scripts and styles)
>
> Step 2:
> * Remove styles from 'site', make 'site' depend on 'site.styles' (only
> affects startup module)
> * No change in HTML output
> * Output B new behaviour: , site.styles=ready,
> mw.loader.load('site') - (loads site scripts)


So, step 2 happens at least 30 days after deploying step 1? Or step 2
retains same behavior for Output A?

-- brion


>
>
> On Wed, May 4, 2016 at 1:43 PM, Brion Vibber <bvib...@wikimedia.org
> <javascript:;>> wrote:
>
>
> > *Extension authors and users of extensions*
> >
> > If addModuleStyles() throws a PHP exception at page-output time for
> modules
> > that include scripts, that's a breaking change for extension authors and
> > the people who use their extensions; but at least the error message will
> be
> > very direct and immediate.
> >
> >
> Yeah, we'll provide good release notes upfront. And probably 1 or 2 week
> iteration in git-master with warning only before enforcing with a run-time
> exception.
>
>
> -- Timo
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org <javascript:;>
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] historical trivia: who first picked UTC as Wikimedia time, when and why?

[Brion Vibber]

In 2001 when Magnus was writing the initial attempt at a custom wiki engine
in PHP backed by MySQL, he chose to use the TIMESTAMP column type.

TIMESTAMPs in MySQL 3 were automatically filled out by the server at INSERT
time, normalized to UTC, and exposed in the 14-digit MMDDHHMMSS format
we still know and love today.

The first TIMESTAMP column in a row also got automatically updated when you
changed something in a row, so we ended up switching them from TIMESTAMP
type to text strings and just filled out the initial values on the PHP
side. We could have used DATETIME but that would have been a slightly
harder transition at the time, and would have introduced the fun of the
server settings trying to give you local time half the time...

-- brion

On Mon, May 9, 2016 at 1:39 AM, David Gerard  wrote:

> Question about obscure historical detail: Who picked UTC as Wikimedia
> time? When was this, and what was the thought process?
>
> (the answer is almost certainly "Brion or Jimbo, early 2001, it's the
> obvious choice", but I'm just curious as to details.)
>
>
> - d.
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Reviving SVG client-side rendering task

[Brion Vibber]

On Thursday, May 5, 2016, Brad Jorsch (Anomie) <bjor...@wikimedia.org>
wrote:

> On Thu, May 5, 2016 at 9:49 AM, Brion Vibber <bvib...@wikimedia.org
> <javascript:;>> wrote:
> There's also the font issue: with the current rasterizing, only certain
> free fonts are available and we know what it falls back to. Serve the SVG
> to the client and those free fonts may well be missing (falling back to
> who-knows-what), while other fonts with different metrics that our
> rasterizer ignores might be present on the client.


Yep... It's worth experimenting with injecting fonts into the
"thumbnail" SVG output. That'll increase total download size again, but
common fonts should be shared between files, making them neatly cacheable,
and if there are no relevant characters we have no need to inject them.

I think all the CSS @font stuff works in SVG as it does in HTML, so if we
have known client-side-safe fonts we can reference them as we do in
HTML land... For common cases where text is in the language of the
surrounding site, that means we may have loaded the same font already if
one was needed.


Making SVG safe for fallback fonts can indeed be tricky if the metrics
don't match, though for simple labels it's usually a matter of setting
reasonable bounding boxes and making use of the alignment specifications,
rather than using the tightest possible bounding box and aligning things
manually by x/y.

For extreme cases where exact glyph shapes and metrics matter because the
text is mixed with graphical elements directly, files should probably
use paths instead of text (or at least embed a specific SVG font made of
specific paths). That can be something that good authoring tools can help
with.

-- brion

>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Reviving SVG client-side rendering task

[Brion Vibber]

For the last decade we've supported uploading SVG vector images to
MediaWiki, but we serve them as rasterized PNGs to browsers. Recently,
display resolutions are going up and up, but so is concern about
low-bandwidth mobile users.

This means we'd like sharper icons and diagrams on high-density phone
displays, but are leery of adding extra srcset entries with 3x or 4x
size PNGs which could become very large. (In fact currently MobileFrontend
strips even the 1.5x and 2x renderings we have now, making diagrams very
blurry on many mobile devices. See https://phabricator.wikimedia.org/T133496 -
fix in works.)


Here's the base bug for SVG client side rendering:
https://phabricator.wikimedia.org/T5593
I've turned it into an "epic" story tracking task and hung some blocking
tasks off it; see those for more details.

TL;DR stop reading here. ;)


One of the basic problems in the past was reliably showing them natively in
an , with the same behavior as before, without using JavaScript hacks
or breaking the hamlet caching layer. This is neatly resolved for current
browsers by using the "srcset" attribute -- the same one we use to specify
higher-resolution rasterizations. If instead of PNGs at 1.5x and 2x
density, we specify an SVG at 1x, the SVG will be loaded instead of the
default PNG.

Since all srcset-supporting browsers allow SVG in  this should "just
work", and will be more compatible than using the experimental 
element or the classic  which deals with events differently. Older
browsers will still see the PNG, and we can tweak the jquery.hidpi srcset
polyfill to test for SVG support to avoid breaking on some older browsers.

This should let us start testing client-side SVG via a beta feature (with
parser cache split on the user pref) at which point we can gather more
real-world feedback on performance and compatibility issues.


Rendering consistency across browser engines is a concern. Supposedly
modern browsers are more consistent than librsvg but we haven't done a
compatibility survey to confirm this or identify problematic constructs.
This is probably worth doing.


Performance is a big question. While clean simple SVGs are often nice and
small and efficient, it's also easy to make a HUGEly detailed SVG that is
much larger than the rasterized PNGs. Or a fairly simple small file may
still render slowly due to use of filters.

So we probably want to provide good tools for our editors and image authors
to help optimize their files. Show the renderings and the bandwidth balance
versus rasterization; maybe provide in-wiki implementation of svgo or other
lossy optimizer tools. Warn about things that are large or render slowly.
Maybe provide a switch to run particular files through rasterization always.

And we'll almost certainly want to strip comments and white space to save
bandwidth on page views, while retaining them all in the source file for
download and reediting.


Feature parity also needs more work. Localized text in SVGs is supported
with our server side rendering but this won't be reliable in the client;
which means we'll want to perform a server side transformation that creates
per-language "thumbnail" SVGs. Fonts for internationalized text are a big
deal, and may require similar transformations if we want to serve them...
Which may mean additional complications and bandwidth usage.


And then there are long term goals of taking more advantage of SVGs dynamic
nature -- making things animated or interactive. That's a much bigger
question and has implementation and security issues!

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Phabricator upgrade and UI changes

[Brion Vibber]

I find it very aggravating the that the author of a task is now very far
away on screen from their comment. Makes it difficult to tell what's going
on and harder to target a reply directly at them.

-- brion

On Wed, May 4, 2016 at 8:57 AM, Federico Leva (Nemo) 
wrote:

> AFAICT, in the last few days there were two unannounced upgrades of
> Phabricator, to an unspecified version of the software, the second of which
> applied rather radical UI changes, especially to Maniphest.
>
> Initially I wasn't sure most/which UI changes were bad/good or why, but
> now I've observed myself for a few days and I took the time to take
> screenshots and describe my findings. Of course, most are about
> * the consistent enlargement of some items to the expense of others,
> * the introduction of a sidebar to host an ever-growing amount of buttons,
> * the move of most task information under the description or under the
> sidebar.
>
> Please comment on the reports, add your findings and file new reports for
> separate issues (I've not tested areas outside Maniphest, for instance):
> * https://phabricator.wikimedia.org/T134393
> * https://phabricator.wikimedia.org/T134394
> * https://phabricator.wikimedia.org/T134398
>
> It would also be nice to have some precise and authoritative information
> on when Phabricator was upgrade to what version, what the release notes and
> gotchas are, etc. Did I miss such an announcement and if yes where is it?
>
> Nemo
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ResourceLoader addModuleStyles() issues

[Brion Vibber]

On Wed, May 4, 2016 at 8:23 AM, Brad Jorsch (Anomie) 
wrote:

> On Wed, May 4, 2016 at 6:34 AM, Krinkle  wrote:
> > [1] If one would allow page style modules to have dependencies and
> resolve
> > them server-side in the HTML output, this would cause corruption when the
> > relationship between two modules changes as existing pages would have the
> > old relationship cached but do get the latest content from the server.
> > Adding versions wouldn't help since the server can't feasibly have access
> > to previous versions (too many page/skin/language combinations).
> >
>
> But don't we have the corruption anyway? Say page Foo has a page style
> module 'foo', so it calls addModuleStyles( [ 'foo' ] ). Then module 'foo'
> is changed so it also needs 'bar', so page Foo now has to call
> addModuleStyles( [ 'foo', 'bar' ] ). What is avoided there that isn't
> avoided when addModuleStyles( [ 'foo' ] ) is smart enough to internally see
> that 'foo' depends on 'bar' and act as if it were passed [ 'foo', 'bar' ]?
> Or what case am I missing?
>
> On the other hand, dependencies avoid the case where the developer
> modifying 'foo' doesn't realize that he has to search for everything
> everywhere that passes 'foo' to addModuleStyles() and manually add 'bar' to
> each one.
>

H... wait, does load.php not resolve dependencies server-side when
producing concatenated CSS output? That would seem to break the
transitional model I proposed if so.

Ah, fun. :)

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ResourceLoader addModuleStyles() issues

[Brion Vibber]

Quick notes on the migration path:


*Cached page HTML*

HTML cached under the old regime would still be served for a while, but
ResourceLoader would load the newer style. I *think* that should work as
long as the new versions of the modules that were included before still
list the style-only modules as dependencies, assuming load.php doesn't
throw an exception when the code-plus-style modules are requested.

If load.php would throw an exception in this case, then that's going to be
a tricky problem to figure out for end-users, who will be presented with
unstyled pages and no visible error message.


*Extension authors and users of extensions*

If addModuleStyles() throws a PHP exception at page-output time for modules
that include scripts, that's a breaking change for extension authors and
the people who use their extensions; but at least the error message will be
very direct and immediate.

It'll need to be called out in the release notes with a section about
breaking upgrade changes.

Ideally, the resulting error message that is displayed should also be very
clear and include a link to a documentation page explaining how extension
authors can update their code appropriately.

-- brion



On Wed, May 4, 2016 at 3:34 AM, Krinkle  wrote:

> TL;DR: The current addModuleStyles() method no longer meets our
> requirements. This mismatch causes bugs (e.g. user styles load twice). The
> current logic also doesn't support dynamic modules depending on style
> modules. I'm looking for feedback on how to best address these issues.
>
>
> ResourceLoader is designed with two module types: Page style modules and
> dynamic modules.
>
> Page style modules are part of the critical rendering path and should work
> without JavaScript being enabled or supported. Their URL must be referenced
> directly in the HTML to allow browsers to discover them statically for
> optimal performance.  As such, this URL can't use dynamic information from
> the startup module (no dependencies[1], no version hashes).
>
> Dynamic modules have their names listed in the page HTML. Their
> dependencies and version hashes are resolved at run-time by JavaScript via
> the startup manifest. We then generate the urls and request them from the
> server. There is also a layer of object caching in-between (localStorage)
> which often optimises the module request to not involve an HTTP request at
> all.
>
> Below I explain the two issues, followed by a proposed solution.
>
> ## Dependency
>
> Historically there was no overlap between these two kinds of modules. Page
> style modules were mostly dominated by the skin and apply to skin-specific
> server-generated html. Dynamic modules (skin agnostic) would make their own
> DOM and apply their own styles.
>
> Now that we're reusing styles more, we're starting seeing to see overlap.
> In the past we used jQuery UI - its styles never applied to PHP-generated
> output. Now, with OOUI, its styles do apply to both server-side and
> client-side generated elements. Its styles are preloaded as a page style
> module on pages that contain OOUI widgets.
>
> The OOjs UI bundle (and its additional styles) shouldn't need to know
> whether the current page already got some of the relevant styles. This is
> what dependencies are for. For OOUI specifically, we currently have a
> hardcoded work around that adds a hidden marker to pages where OOUI is
> preloaded. The OOUI style module has a skipFunction that forgoes loading if
> the marker is present.
>
> ## Implicit type
>
> Aside from the need for dependencies between dynamic and page style
> modules. There is another bug related to this. We don't currently require
> modules to say whether they are a dynamic module or a page style module. In
> most cases this doesn't matter since a developer would only load it one way
> and the module type is implied. For example, if you only ever pass a module
> to addModules() or mw.loader() it is effectively a dynamic module. If you
> only ever pass a module to addModuleStyles() loaded via a  rel=stylesheet> then it is a page style module.
>
> A problem happens if one tries to load the same module both ways. This
> might seem odd (and it is), but happens unintentionally right now with wiki
> modules (specifically, user/site modules and gadgets).
>
> For user/site modules, we don't know whether common.css relates to the page
> content, or whether it relates to dynamic content produced by common.js.
> The same for gadgets. A gadget may produce an AJAX interface and register
> styles for it, or the gadget may be styles-only and intended as a skin
> customisation. Right now the Gadgets extension works around this problem
> with a compromise: Load it both ways. First it loads all gadgets as page
> style modules (ignoring the script portion), and then it loads the same
> modules again as dynamic modules. Thus loading the styles twice.
>
> ## Proposed solution
>
> In order to allow dependency relationships between a dynamic 

[Wikitech-l] ImageMagick reported security vulnerability

[Brion Vibber]

There's a reported ImageMagick security vulnerability making the rounds:
http://arstechnica.com/security/2016/05/easily-exploited-bug-exposes-huge-number-of-sites-to-code-execution-attacks/

Many MediaWiki sites are configured to use ImageMagick's 'convert' command
to perform image rescaling/thumbnailing, so it's worth double-checking that
everything is secure...

MediaWiki already performs file type validation checks on uploads, which
*should* prevent exploitation of the vulnerability for the standard image
types (JPEG, PNG, GIF, SVG etc) -- this is one of the recommended
mitigation strategies. But I am not confident enough to pronounce us immune
without actually checking!

Some folks are also recommending tweaking the ImageMagick policy.xml to
disable the most dangerous code modules; see the article linked above for
further links.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Getting rid of $wgWellFormedXml = false;

[Brion Vibber]

I'd say an HTML5 output mode *ought* to work like this:

*Don't try to be clever.*
* Consistency and predictability are key to both security review and data
consumability.

*Quote attributes consistently and predictably.*
* Always use double-quotes on attributes in output.

*Output specced empty tags in HTML style.*
* , ,  are fine and not ambiguous at all to an HTML parser.
There's no need to go adding a "/" in at the end!
* These are already whitelisted in the Html class so it's easy to not mess
this up.

*Don't do other silly things for old-school XHTML 1.*
* CDATA wrapping of 

Re: [Wikitech-l] Rob Lanphier appointed to the Architecture Committee

[Brion Vibber]

Woot!

-- brion

On Monday, April 25, 2016, Tim Starling  wrote:

> At the previous meeting of the MediaWiki Architecture Committee (April
> 20), the members present approved the appointment of Rob Lanphier to
> the committee.
>
> Rob was the main instigator in the formation of the committee in 2014.
> Lately he has been taking an active role, chairing the weekly meetings
> and writing the meeting agenda. In recognition of the excellent work
> he has been doing, and in the interests of transparency, we decided to
> formalise his membership.
>
> -- Tim Starling
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Docs, use of, and admin privileges for wikimedia github project?

[Brion Vibber]

There seems to be extremely little documentation on Wikimedia's GitHub
project https://github.com/wikimedia ... I can only find
https://www.mediawiki.org/wiki/Gerrit/GitHub which mostly says we mirror a
bunch of stuff from gerrit. And I know we have continuous integration of
some kind set up for some projects, but it doesn't seem to be well
documented in a place I could find.

There are also some repos that are mirrors of gerrit, and other repos that
are primary repos, and it's a bit unclear what's what. More importantly,
when folks have repos that they've been running on GitHub already and want
to move into the wikimedia project (rather than switch to gerrit), what's
the procedure? I'm an admin/owner so I can manually import people's repos
but I'm not sure whether I'm supposed to... :)

We also have a lot of admins, which I wonder is necessary:
https://github.com/orgs/wikimedia/people?utf8=✓=role%3Aowner+

Do we do any security review / removal of old accounts, or have a procedure
for adding new admins?

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ResourceLoader, latency, & progressive enhancement

[Brion Vibber]

I also didn't really distinguish three separate perf points in my blurb:
* time to first paint
* time to bare page interactivity
* time to enhanced page display / interactivity

Mainly I was concentrating on the third point, but the first two -- which
y'all are already doing a great job on -- are even more important and
should not regress or we will be sad pandas. :)

/me gets on a plane, will comment more later

-- brion

On Monday, April 25, 2016, Brion Vibber <bvib...@wikimedia.org> wrote:

> On Monday, April 25, 2016, Ori Livneh <o...@wikimedia.org
> <javascript:_e(%7B%7D,'cvml','o...@wikimedia.org');>> wrote:
>
>> Not so straight-forward. Khan Academy tried unbundling JavaScript on
>> HTTP/2
>> page views last November and found that performance got worse. They
>> attribute the regression primarily to the fact that bundling improves
>> compression. They concluded that "it is premature to give up on bundling
>> JavaScript files at this time, even for HTTP/2.0 clients."
>>
>> (http://engineering.khanacademy.org/posts/js-packaging-http2.htm)
>
>
> Nice, I'll go read that. :)
>
>
>
>> On most browsers, we take advantage of localStorage pretty heavily in
>> order
>> to have a durable cache of individual modules. Without it, slight
>> variations in the module requirements would occasion re-downloading a lot
>> of JavaScript, as the browser had no way of reusing JavaScript and CSS
>> delivered under a different URL. (Service Workers now provide more
>> sophisticated means of doing that, but global browser support is still
>> only
>> at 53%.
>>
>> We had to disable localStorage caching in Firefox because of the way it
>> manages quotas. Is your primary mobile browser Firefox for Android / iOS?
>
>
> Service workers are sounding more and more attractive here -- we could
> rewrite the requests as necessary to bundle when it makes sense etc, and
> avoid clogging up the synchronous, space-limited localStorage. Needs more
> research...
>
>
>> Lastly, we have good evidence that above-the-fold external CSS is a bigger
>> contributor to page latency than JavaScript. Gabriel documented that
>> pretty
>> well in T124966 <https://phabricator.wikimedia.org/T124966>. That CSS is
>> a
>> bigger issue than JavaScript is not surprising (top-loaded CSS is
>> render-blocking, whereas all of our JavaScript is loaded asynchronously),
>> but the magnitude of its impact is impressive.
>>
>> Krinkle is working on an arc of tasks that would get us there; T127328
>> <https://phabricator.wikimedia.org/T127328> is the master task.
>
>
> Awesome, I'll read up and comment!
>
> -- brion
>
>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ResourceLoader, latency, & progressive enhancement

[Brion Vibber]

On Monday, April 25, 2016, Ori Livneh  wrote:

> Not so straight-forward. Khan Academy tried unbundling JavaScript on HTTP/2
> page views last November and found that performance got worse. They
> attribute the regression primarily to the fact that bundling improves
> compression. They concluded that "it is premature to give up on bundling
> JavaScript files at this time, even for HTTP/2.0 clients."
>
> (http://engineering.khanacademy.org/posts/js-packaging-http2.htm)


Nice, I'll go read that. :)



> On most browsers, we take advantage of localStorage pretty heavily in order
> to have a durable cache of individual modules. Without it, slight
> variations in the module requirements would occasion re-downloading a lot
> of JavaScript, as the browser had no way of reusing JavaScript and CSS
> delivered under a different URL. (Service Workers now provide more
> sophisticated means of doing that, but global browser support is still only
> at 53%.
>
> We had to disable localStorage caching in Firefox because of the way it
> manages quotas. Is your primary mobile browser Firefox for Android / iOS?


Service workers are sounding more and more attractive here -- we could
rewrite the requests as necessary to bundle when it makes sense etc, and
avoid clogging up the synchronous, space-limited localStorage. Needs more
research...


> Lastly, we have good evidence that above-the-fold external CSS is a bigger
> contributor to page latency than JavaScript. Gabriel documented that pretty
> well in T124966 . That CSS is a
> bigger issue than JavaScript is not surprising (top-loaded CSS is
> render-blocking, whereas all of our JavaScript is loaded asynchronously),
> but the magnitude of its impact is impressive.
>
> Krinkle is working on an arc of tasks that would get us there; T127328
>  is the master task.


Awesome, I'll read up and comment!

-- brion


> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] ResourceLoader, latency, & progressive enhancement

[Brion Vibber]

On Sunday, April 24, 2016, Daniel Friesen <dan...@nadir-seen-fire.com>
wrote:

> Tangentially related, Chrome plans to drop support for SPDY and go
> HTTP/2 only this year, Edge already dropped support for SPDY, and other
> browsers may too.
> So before this is actually implemented, Wikimedia might want to upgrade
> web server software to support HTTP/2 (currently MediaWiki.org looks to
> only be using SPDY).
>
> Though a sad realization is that IE11 only supports HTTP/2 on Windows 10
> (other Windows versions will only support SPDY) and same goes for Safari
> only doing HTTP/2 on OSX 10.11+.
> Which is relevant because some webservers like Nginx intentionally drop
> the SPDY implementation in the release they implement HTTP/2.


Yeah, transition will be "fun". We need to make sure we either have
something that works well enough on both http 1.1 and 2 if we can't keep
SPDY for the slightly older browsers, or play fun games with variant
caching so we have a concatenated loader setup and a non-concatenated
loader setup. :)

For those not familiar, SPDY is roughly the experimental predecessor of the
new HTTP/2, providing most of the same benefits but not quite compatible
with the final standard. As a transitional technology it's getting dropped
from some of the things that are updating, but we're going to see some
folks stuck on browser versions in the middle with SPDY but no HTTP/2...
And others with older browsers that support neither:

http://caniuse.com/#feat=spdy
http://caniuse.com/#feat=http2

-- brion


> ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
>
> On 2016-04-23 3:08 PM, Brion Vibber wrote:
> > Started as quick thoughts, turned into more of an essay, so I've posted
> the
> > bulk on mediawiki.org:
> >
> >
> https://www.mediawiki.org/wiki/User:Brion_VIBBER/ResourceLoader_and_latency
> >
> >
> > tl;dr summary:
> >
> > On slow networks, latency in loading large JS and HTML resources means
> > things don't always work right when we first see them.
> >
> > If we take advantage of HTTP 2 we could skip the concatenation of
> separate
> > ResourceLoader modules to reduce latency until each module _runs_,
> without
> > adding _network_ latency.
> >
> > And if we're more clever about handling 'progressive enhancement' via JS
> > _while_ an HTML page loads, we could reduce the time before large pages
> > become fully interactive.
> >
> > -- brion
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org <javascript:;>
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org <javascript:;>
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] ResourceLoader, latency, & progressive enhancement

[Brion Vibber]

Started as quick thoughts, turned into more of an essay, so I've posted the
bulk on mediawiki.org:

https://www.mediawiki.org/wiki/User:Brion_VIBBER/ResourceLoader_and_latency


tl;dr summary:

On slow networks, latency in loading large JS and HTML resources means
things don't always work right when we first see them.

If we take advantage of HTTP 2 we could skip the concatenation of separate
ResourceLoader modules to reduce latency until each module _runs_, without
adding _network_ latency.

And if we're more clever about handling 'progressive enhancement' via JS
_while_ an HTML page loads, we could reduce the time before large pages
become fully interactive.

-- brion
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Best practices for read/write vs read-only requests, and our multi-DC future

[Brion Vibber]

I've opened a phab task https://phabricator.wikimedia.org/T133448 about
writing up good intro docs and updating other docs to match it.

Feel free y'all to add to that or hang additional tasks onto it like better
utility classes to help folks transition code to background jobs... And
maybe infrastructure to make sure we're handling those jobs reliably on
small sites without dedicated job runners.

-- brion
On Apr 21, 2016 5:26 PM, "Brion Vibber" <bvib...@wikimedia.org> wrote:

> On Thu, Apr 21, 2016 at 4:59 PM, Erik Bernhardson <
> ebernhard...@wikimedia.org> wrote:
>
>> On Apr 20, 2016 10:45 PM, "Brion Vibber" <bvib...@wikimedia.org> wrote:
>> > Note that we could fire off a job queue background task to do the actual
>> > removal... But is it also safe to do that on a read-only request?
>> >
>>
>> https://www.mediawiki.org/wiki/Requests_for_comment/Master_%26_slave_datacenter_strategy_for_MediaWiki
>> > seems to indicate job queueing will be safe, but would like to confirm
>> > that. :)
>> >
>>
>> I think this is the preferred method. My understanding is that the jobs
>> will get shipped to the primary DC job queue.
>>
>
> *nod* looks like per spec that should work with few surprises.
>
>
>>
>> > Similarly in https://gerrit.wikimedia.org/r/#/c/284269/ we may wish to
>> > trigger missing transcodes to run on demand, similarly. The actual re
>> > encoding happens in a background job, but we have to fire it off, and we
>> > have to record that we fired it off so we don't duplicate it...
>> [snip]
>> >
>> The job queue can do deduplication, although you would have to check if
>> that is active while the job is running and not only while queued. Might
>> help?
>>
>
> Part of the trick is we want to let the user know that the job has been
> queued; and if the job errors out, we want the user to know that the job
> errored out.
>
> Currently this means we have to update a row in the 'transcode' table
> (TimedMediaHandler-specific info about the transcoded derivative files)
> when we fire off the job, then update its state again when the job actually
> runs.
>
> If that's split into two queues, one lightweight and one heavyweight, then
> this might make sense:
>
> * N web requests hit something using File:Foobar.webm, which has a missing
> transcode
> * they each try to queue up a job to the lightweight queue that says
> "start queueing this to actually transcode!"
> * when the job queue runner on the lightweight queue sees the first such
> job, it records the status update to the database and queues up a
> heavyweight job to run the actual transcoding. The N-1 remaining jobs duped
> on the same title/params either get removed, or never got stored in the
> first place; I forget how it works. :)
> * ... time passes, during which further web requests don't yet see the
> updated database table state, and keep queueing in the lightweight queue.
> * lightweight queue runners see some of those jobs, but they have the
> updated master database state and know they don't need to act.
> * database replication of the updated state hits the remote DC
> * ..time passes, during which further web requests see the updated
> database table state and don't bother queueing the lightweight job
> * eventually, the heavyweight job runs, completes, updates the states at
> start and end.
> * eventually, the database replicates the transcode state completion to
> the remote DC.
> * web requests start seeing the completed state, and their output includes
> the updated transcode information.
>
> It all feels a bit complex, and I wonder if we could build some common
> classes to help with this transaction model. I'm pretty sure we can be
> making more use of background jobs outside of TimedMediaHandler's slow
> video format conversions. :D
>
> -- brion
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l