Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Pine W]

I meant end of August. Sorry for the extra email noise.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

On Tue, Jul 31, 2018 at 2:15 AM, Pine W  wrote:

> Thanks, Brion. At my current rate of progress, I'll be making use of this
> feature by the end of the month.
>
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
> On Tue, Jul 31, 2018 at 12:51 AM, Brion Vibber 
> wrote:
>
>> Configuration has been updated and VP9 mode swapped in. Newly uploaded
>> videos should start encoding as VP9 starting now.
>>
>> I'll start the backfill batch process tonight or tomorrow, and will
>> likely stop and restart it a few times over the coming weeks if anything
>> needs adjustment. Please file tasks in phabricator and/or give me a ping on
>> IRC if any issues come up with the new conversions, or with old files!
>> (Existing VP8 files will be left as-is for now until we're sure
>> everything's up to snuff.)
>>
>> Big thanks to everybody who's helped prepping this, with libvpx and
>> ffmpeg deployments, with the patches and review, and with final deployment
>> which always takes longer than you expect. :) This'll be a nice improvement
>> to our video output for now, and lays a lot of groundwork for next steps.
>>
>> -- brion
>>
>>
>> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber 
>> wrote:
>>
>>> Oh and one more thing!
>>>
>>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>>> resolutions, which people can manually bump up to when watching videos with
>>> a suitable 4K source on a high-res screen. They use higher data rates, but
>>> only a small fraction of input files are 4K so should not significantly
>>> increase disk space projections for now.
>>>
>>> These can take a long time to compress, so if we find it's problematic
>>> we'll turn them back off until the jobs can be split into tiny chunks
>>> (future work planned!), but it works in my testing and shouldn't clog the
>>> servers now that we have more available.
>>>
>>> (Note that the ogv.js player shim for Safari will not handle
>>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>>> Chromium-based browser.)
>>>
>>> -- brion
>>>
>>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
>>> wrote:
>>>
 Ok, after some delay for re-tweaking the encoding settings for higher
 quality when needed, and pulling in some other improvements to the config
 system, all related updates to TimedMediaHandler have been merged. :D

 If all goes well with the general deployments in the next few days,
 expect the beginning of VP9 rollout starting next week.

 Changes since the earlier announcement:
 * the new row-multithreading will be available, which allows higher
 threading usage at all resolutions; encoding times will be more like 1.5-2x
 slower instead of 3-4x slower.
 * switch to constrained quality with a larger max bitrate: many files
 will become significantly smaller in their VP9 versions, but some will
 actually increase in exchange for a huge increase in quality -- this is
 mostly 60fps high-rate files, and those with lots of motion and detail that
 didn't compress well at the default low data rates.

 -- brion

 On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
 wrote:

> Awesome sauce. Thanks Moritz!
>
> -- brion
>
> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
> mmuhlenh...@wikimedia.org> wrote:
>
>> Hi all,
>>
>> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
>> > Current state on this:
>> >
>> > * still hoping to deploy the libvpx+ffmpeg backport first so we
>> start with
>> > best performance; Moritz made a start on libvpx but we still have to
>> > resolve ffmpeg (possibly by patching 3.2 instead of updating all
>> the way to
>> > 3.4)
>>
>> I've completed this today. We now have a separate repository component
>> for stretch-wikimedia (named component/vp9) which includes ffmpeg
>> 3.2.10
>> (thus allowing us to follow the ffmpeg security updates released in
>> Debian
>> with a local rebuild) with backported row-mt support and linked
>> against
>> libvpx 1.7.0.
>>
>> I tested re-encoding
>> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitt
>> s_Todeswand_2017_-_Jagath_Perera.webm
>> (which is a nice fast-paced test file) from VP8 to VP9, which results
>> in
>> a size reduction from 48M to 31M.
>>
>> When using eight CPU cores on one of our video scaler servers,
>> enabling row-mt
>> gives a significant performance boost; encoding time went down from
>> 5:31 mins
>> to 3:36 mins.
>>
>> All the details can be found at
>> https://phabricator.wikimedia.org/T190333#4324995
>>
>> Cheers,
>> Moritz
>>
>> ___

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Pine W]

Thanks, Brion. At my current rate of progress, I'll be making use of this
feature by the end of the month.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

On Tue, Jul 31, 2018 at 12:51 AM, Brion Vibber 
wrote:

> Configuration has been updated and VP9 mode swapped in. Newly uploaded
> videos should start encoding as VP9 starting now.
>
> I'll start the backfill batch process tonight or tomorrow, and will likely
> stop and restart it a few times over the coming weeks if anything needs
> adjustment. Please file tasks in phabricator and/or give me a ping on IRC
> if any issues come up with the new conversions, or with old files!
> (Existing VP8 files will be left as-is for now until we're sure
> everything's up to snuff.)
>
> Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
> deployments, with the patches and review, and with final deployment which
> always takes longer than you expect. :) This'll be a nice improvement to
> our video output for now, and lays a lot of groundwork for next steps.
>
> -- brion
>
>
> On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber 
> wrote:
>
>> Oh and one more thing!
>>
>> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
>> resolutions, which people can manually bump up to when watching videos with
>> a suitable 4K source on a high-res screen. They use higher data rates, but
>> only a small fraction of input files are 4K so should not significantly
>> increase disk space projections for now.
>>
>> These can take a long time to compress, so if we find it's problematic
>> we'll turn them back off until the jobs can be split into tiny chunks
>> (future work planned!), but it works in my testing and shouldn't clog the
>> servers now that we have more available.
>>
>> (Note that the ogv.js player shim for Safari will not handle
>> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
>> iPad; for best results for 4K playback use Firefox, Chrome, or a
>> Chromium-based browser.)
>>
>> -- brion
>>
>> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
>> wrote:
>>
>>> Ok, after some delay for re-tweaking the encoding settings for higher
>>> quality when needed, and pulling in some other improvements to the config
>>> system, all related updates to TimedMediaHandler have been merged. :D
>>>
>>> If all goes well with the general deployments in the next few days,
>>> expect the beginning of VP9 rollout starting next week.
>>>
>>> Changes since the earlier announcement:
>>> * the new row-multithreading will be available, which allows higher
>>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>>> slower instead of 3-4x slower.
>>> * switch to constrained quality with a larger max bitrate: many files
>>> will become significantly smaller in their VP9 versions, but some will
>>> actually increase in exchange for a huge increase in quality -- this is
>>> mostly 60fps high-rate files, and those with lots of motion and detail that
>>> didn't compress well at the default low data rates.
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
>>> wrote:
>>>
 Awesome sauce. Thanks Moritz!

 -- brion

 On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
 mmuhlenh...@wikimedia.org> wrote:

> Hi all,
>
> On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
> > Current state on this:
> >
> > * still hoping to deploy the libvpx+ffmpeg backport first so we
> start with
> > best performance; Moritz made a start on libvpx but we still have to
> > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
> way to
> > 3.4)
>
> I've completed this today. We now have a separate repository component
> for stretch-wikimedia (named component/vp9) which includes ffmpeg
> 3.2.10
> (thus allowing us to follow the ffmpeg security updates released in
> Debian
> with a local rebuild) with backported row-mt support and linked against
> libvpx 1.7.0.
>
> I tested re-encoding
> https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_
> Pitts_Todeswand_2017_-_Jagath_Perera.webm
> (which is a nice fast-paced test file) from VP8 to VP9, which results
> in
> a size reduction from 48M to 31M.
>
> When using eight CPU cores on one of our video scaler servers,
> enabling row-mt
> gives a significant performance boost; encoding time went down from
> 5:31 mins
> to 3:36 mins.
>
> All the details can be found at
> https://phabricator.wikimedia.org/T190333#4324995
>
> Cheers,
> Moritz
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l


> ___
> Multimedia mailing list
> multime...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/li

Re: [Wikitech-l] [Multimedia] Video output changing to WebM VP9/Opus soon

[Brion Vibber]

Configuration has been updated and VP9 mode swapped in. Newly uploaded
videos should start encoding as VP9 starting now.

I'll start the backfill batch process tonight or tomorrow, and will likely
stop and restart it a few times over the coming weeks if anything needs
adjustment. Please file tasks in phabricator and/or give me a ping on IRC
if any issues come up with the new conversions, or with old files!
(Existing VP8 files will be left as-is for now until we're sure
everything's up to snuff.)

Big thanks to everybody who's helped prepping this, with libvpx and ffmpeg
deployments, with the patches and review, and with final deployment which
always takes longer than you expect. :) This'll be a nice improvement to
our video output for now, and lays a lot of groundwork for next steps.

-- brion


On Thu, Jul 26, 2018 at 5:47 PM Brion Vibber  wrote:

> Oh and one more thing!
>
> For the VP9 configuration I'll be enabling 1440p and 2160p ("4K")
> resolutions, which people can manually bump up to when watching videos with
> a suitable 4K source on a high-res screen. They use higher data rates, but
> only a small fraction of input files are 4K so should not significantly
> increase disk space projections for now.
>
> These can take a long time to compress, so if we find it's problematic
> we'll turn them back off until the jobs can be split into tiny chunks
> (future work planned!), but it works in my testing and shouldn't clog the
> servers now that we have more available.
>
> (Note that the ogv.js player shim for Safari will not handle
> greater-than-HD resolutions fast enough for playback, even on a fast Mac or
> iPad; for best results for 4K playback use Firefox, Chrome, or a
> Chromium-based browser.)
>
> -- brion
>
> On Thu, Jul 26, 2018 at 5:39 PM Brion Vibber 
> wrote:
>
>> Ok, after some delay for re-tweaking the encoding settings for higher
>> quality when needed, and pulling in some other improvements to the config
>> system, all related updates to TimedMediaHandler have been merged. :D
>>
>> If all goes well with the general deployments in the next few days,
>> expect the beginning of VP9 rollout starting next week.
>>
>> Changes since the earlier announcement:
>> * the new row-multithreading will be available, which allows higher
>> threading usage at all resolutions; encoding times will be more like 1.5-2x
>> slower instead of 3-4x slower.
>> * switch to constrained quality with a larger max bitrate: many files
>> will become significantly smaller in their VP9 versions, but some will
>> actually increase in exchange for a huge increase in quality -- this is
>> mostly 60fps high-rate files, and those with lots of motion and detail that
>> didn't compress well at the default low data rates.
>>
>> -- brion
>>
>> On Fri, Jun 29, 2018 at 9:46 AM Brion Vibber 
>> wrote:
>>
>>> Awesome sauce. Thanks Moritz!
>>>
>>> -- brion
>>>
>>> On Fri, Jun 29, 2018 at 7:39 AM Moritz Muehlenhoff <
>>> mmuhlenh...@wikimedia.org> wrote:
>>>
 Hi all,

 On Thu, Jun 28, 2018 at 01:54:18PM -0700, Brion Vibber wrote:
 > Current state on this:
 >
 > * still hoping to deploy the libvpx+ffmpeg backport first so we start
 with
 > best performance; Moritz made a start on libvpx but we still have to
 > resolve ffmpeg (possibly by patching 3.2 instead of updating all the
 way to
 > 3.4)

 I've completed this today. We now have a separate repository component
 for stretch-wikimedia (named component/vp9) which includes ffmpeg 3.2.10
 (thus allowing us to follow the ffmpeg security updates released in
 Debian
 with a local rebuild) with backported row-mt support and linked against
 libvpx 1.7.0.

 I tested re-encoding

 https://commons.wikimedia.org/wiki/File:Wall_of_Death_-_Pitts_Todeswand_2017_-_Jagath_Perera.webm
 (which is a nice fast-paced test file) from VP8 to VP9, which results in
 a size reduction from 48M to 31M.

 When using eight CPU cores on one of our video scaler servers, enabling
 row-mt
 gives a significant performance boost; encoding time went down from
 5:31 mins
 to 3:36 mins.

 All the details can be found at
 https://phabricator.wikimedia.org/T190333#4324995

 Cheers,
 Moritz

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>
>>>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikidata-tech] Normalization of change tag schema

[Derk-Jan Hartman]

That is an impressive difference !

On Mon, Jul 30, 2018 at 6:22 PM Amir Sarabadani <
amir.sarabad...@wikimedia.de> wrote:

> And this is the load on vslow database nodes on s7:
>
> https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1532794373712&to=1532967173714&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13317
>
> You can see similar drops on other sections from exactly the moment it got
> deployed:
> s1:
>
> https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1531757700702&to=1532967300702&var-dc=eqiad%20prometheus%2Fops&var-server=db1106&var-port=9104
> s2
> 
> :
>
> https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1532794561870&to=1532967361872&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13312
>
> Best
>
> On Mon, 30 Jul 2018 at 13:13, Amir Sarabadani <
> amir.sarabad...@wikimedia.de>
> wrote:
>
> > Hey,
> > Using the new table as backend of Special:Tags (and similar APIs) is now
> > enabled everywhere. Contact me if there's any issues with that.
> >
> > Best
> >
> > On Wed, 25 Jul 2018 at 19:17, Amir Sarabadani <
> > amir.sarabad...@wikimedia.de> wrote:
> >
> >> Hello,
> >> One update regarding this.
> >> We enabled using the new table for Special:Tags in several large wikis
> >> which caused a massive improvement in the performance of the page. For
> >> example loading Special:Tags on Wikidata used to take around a minute
> and
> >> now it takes less than a second. English Wikipedia is down from ten
> seconds
> >> to less than one and so on.
> >>
> >> There is a lot of work needs to be done and maintenance scripts is being
> >> ran to backpopulate the ct_tag_id column in change_tag table (If you
> want
> >> to follow the progress, see https://phabricator.wikimedia.org/T193873)
> >> and then we need start reading from the new table in mediawiki and
> finally
> >> we can drop ct_tag column entirely. If you want to help in review,
> writing
> >> code or anything, just let me know.
> >>
> >> Best
> >>
> >> On Wed, 27 Jun 2018 at 15:15, Léa Lacroix 
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> Our team is refactoring some code around the change tags on Recent
> >>> Changes. This can impact people using the database on ToolForge.
> >>>
> >>> Currently, the tags are stored in the table change_tag in the column
> >>> ct_tag.
> >>>
> >>> In the next days, we will add a column ct_tag_id with a unique
> >>> identifier for these tags. A new table change_tag_def that will store
> >>> the tag id, the message, and more information like how many times this
> tag
> >>> is used on the local wiki.
> >>>
> >>> On the long term, we plan to drop the column ct_tag since the tag will
> >>> be identified with ct_tag_id.
> >>>
> >>> This change will happen on:
> >>> - French Wikipedia: Monday July 2nd
> >>> - All other wikis: from July 9th
> >>>
> >>> If there is any problem (trouble with saving edits, slow down of recent
> >>> changes…) please  create a subtask of T185355
> >>>  or contact Ladsgroup
> >>> .
> >>>
> >>> Cheers,
> >>> --
> >>> Léa Lacroix
> >>> Project Manager Community Communication for Wikidata
> >>>
> >>> Wikimedia Deutschland e.V.
> >>> Tempelhofer Ufer 23-24
> >>> 10963 Berlin
> >>> www.wikimedia.de
> >>>
> >>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
> >>>
> >>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> >>> unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
> Finanzamt
> >>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
> >>> ___
> >>> Wikidata-tech mailing list
> >>> wikidata-t...@lists.wikimedia.org
> >>> https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
> >>>
> >>
> >>
> >> --
> >> Amir Sarabadani
> >> Software Engineer
> >>
> >> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> >> Tel. (030) 219 158 26-0
> >> http://wikimedia.de
> >>
> >> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> >> Wissens frei teilhaben kann. Helfen Sie uns dabei!
> >> http://spenden.wikimedia.de/
> >>
> >> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
> >> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> unter
> >> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> >> Körperschaften I Berlin, Steuernummer 27/029/42207.
> >>
> >
> >
> > --
> > Amir Sarabadani
> > Software Engineer
> >
> > Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> > Tel. (030) 219 158 26-0
> > http://wikimedia.de
> >
> > Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> 

Re: [Wikitech-l] [Wikidata-tech] Normalization of change tag schema

[Jaime Crespo]

Awesome news, Amir,

You made some function calls one million times faster, making contributors
happier and making available resources that can now be used for other
mission-related tasks. That also made, at least, WMF DBAs very happy!

Cheers,

On Mon, Jul 30, 2018 at 6:22 PM Amir Sarabadani <
amir.sarabad...@wikimedia.de> wrote:

> And this is the load on vslow database nodes on s7:
>
> https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1532794373712&to=1532967173714&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13317
>
> You can see similar drops on other sections from exactly the moment it got
> deployed:
> s1:
>
> https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1531757700702&to=1532967300702&var-dc=eqiad%20prometheus%2Fops&var-server=db1106&var-port=9104
> s2
> 
> :
>
> https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1532794561870&to=1532967361872&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13312
>
> Best
>
> On Mon, 30 Jul 2018 at 13:13, Amir Sarabadani <
> amir.sarabad...@wikimedia.de>
> wrote:
>
> > Hey,
> > Using the new table as backend of Special:Tags (and similar APIs) is now
> > enabled everywhere. Contact me if there's any issues with that.
> >
> > Best
> >
> > On Wed, 25 Jul 2018 at 19:17, Amir Sarabadani <
> > amir.sarabad...@wikimedia.de> wrote:
> >
> >> Hello,
> >> One update regarding this.
> >> We enabled using the new table for Special:Tags in several large wikis
> >> which caused a massive improvement in the performance of the page. For
> >> example loading Special:Tags on Wikidata used to take around a minute
> and
> >> now it takes less than a second. English Wikipedia is down from ten
> seconds
> >> to less than one and so on.
> >>
> >> There is a lot of work needs to be done and maintenance scripts is being
> >> ran to backpopulate the ct_tag_id column in change_tag table (If you
> want
> >> to follow the progress, see https://phabricator.wikimedia.org/T193873)
> >> and then we need start reading from the new table in mediawiki and
> finally
> >> we can drop ct_tag column entirely. If you want to help in review,
> writing
> >> code or anything, just let me know.
> >>
> >> Best
> >>
> >> On Wed, 27 Jun 2018 at 15:15, Léa Lacroix 
> >> wrote:
> >>
> >>> Hello all,
> >>>
> >>> Our team is refactoring some code around the change tags on Recent
> >>> Changes. This can impact people using the database on ToolForge.
> >>>
> >>> Currently, the tags are stored in the table change_tag in the column
> >>> ct_tag.
> >>>
> >>> In the next days, we will add a column ct_tag_id with a unique
> >>> identifier for these tags. A new table change_tag_def that will store
> >>> the tag id, the message, and more information like how many times this
> tag
> >>> is used on the local wiki.
> >>>
> >>> On the long term, we plan to drop the column ct_tag since the tag will
> >>> be identified with ct_tag_id.
> >>>
> >>> This change will happen on:
> >>> - French Wikipedia: Monday July 2nd
> >>> - All other wikis: from July 9th
> >>>
> >>> If there is any problem (trouble with saving edits, slow down of recent
> >>> changes…) please  create a subtask of T185355
> >>>  or contact Ladsgroup
> >>> .
> >>>
> >>> Cheers,
> >>> --
> >>> Léa Lacroix
> >>> Project Manager Community Communication for Wikidata
> >>>
> >>> Wikimedia Deutschland e.V.
> >>> Tempelhofer Ufer 23-24
> >>> 10963 Berlin
> >>> www.wikimedia.de
> >>>
> >>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
> >>>
> >>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> >>> unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
> Finanzamt
> >>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
> >>> ___
> >>> Wikidata-tech mailing list
> >>> wikidata-t...@lists.wikimedia.org
> >>> https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
> >>>
> >>
> >>
> >> --
> >> Amir Sarabadani
> >> Software Engineer
> >>
> >> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> >> Tel. (030) 219 158 26-0
> >> http://wikimedia.de
> >>
> >> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> >> Wissens frei teilhaben kann. Helfen Sie uns dabei!
> >> http://spenden.wikimedia.de/
> >>
> >> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
> >> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> unter
> >> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> >> Körperschaften I Berlin, Steuernummer 27/029/42207.
> >>
> >
> >
> > --
> > Amir Sarabadani
> > Software Engineer

Re: [Wikitech-l] [Wikidata-tech] Normalization of change tag schema

[Amir Sarabadani]

And this is the load on vslow database nodes on s7:
https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1532794373712&to=1532967173714&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13317

You can see similar drops on other sections from exactly the moment it got
deployed:
s1:
https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1531757700702&to=1532967300702&var-dc=eqiad%20prometheus%2Fops&var-server=db1106&var-port=9104
s2:
https://grafana.wikimedia.org/dashboard/db/mysql?panelId=3&fullscreen&orgId=1&from=1532794561870&to=1532967361872&var-dc=eqiad%20prometheus%2Fops&var-server=db1090&var-port=13312

Best

On Mon, 30 Jul 2018 at 13:13, Amir Sarabadani 
wrote:

> Hey,
> Using the new table as backend of Special:Tags (and similar APIs) is now
> enabled everywhere. Contact me if there's any issues with that.
>
> Best
>
> On Wed, 25 Jul 2018 at 19:17, Amir Sarabadani <
> amir.sarabad...@wikimedia.de> wrote:
>
>> Hello,
>> One update regarding this.
>> We enabled using the new table for Special:Tags in several large wikis
>> which caused a massive improvement in the performance of the page. For
>> example loading Special:Tags on Wikidata used to take around a minute and
>> now it takes less than a second. English Wikipedia is down from ten seconds
>> to less than one and so on.
>>
>> There is a lot of work needs to be done and maintenance scripts is being
>> ran to backpopulate the ct_tag_id column in change_tag table (If you want
>> to follow the progress, see https://phabricator.wikimedia.org/T193873)
>> and then we need start reading from the new table in mediawiki and finally
>> we can drop ct_tag column entirely. If you want to help in review, writing
>> code or anything, just let me know.
>>
>> Best
>>
>> On Wed, 27 Jun 2018 at 15:15, Léa Lacroix 
>> wrote:
>>
>>> Hello all,
>>>
>>> Our team is refactoring some code around the change tags on Recent
>>> Changes. This can impact people using the database on ToolForge.
>>>
>>> Currently, the tags are stored in the table change_tag in the column
>>> ct_tag.
>>>
>>> In the next days, we will add a column ct_tag_id with a unique
>>> identifier for these tags. A new table change_tag_def that will store
>>> the tag id, the message, and more information like how many times this tag
>>> is used on the local wiki.
>>>
>>> On the long term, we plan to drop the column ct_tag since the tag will
>>> be identified with ct_tag_id.
>>>
>>> This change will happen on:
>>> - French Wikipedia: Monday July 2nd
>>> - All other wikis: from July 9th
>>>
>>> If there is any problem (trouble with saving edits, slow down of recent
>>> changes…) please  create a subtask of T185355
>>>  or contact Ladsgroup
>>> .
>>>
>>> Cheers,
>>> --
>>> Léa Lacroix
>>> Project Manager Community Communication for Wikidata
>>>
>>> Wikimedia Deutschland e.V.
>>> Tempelhofer Ufer 23-24
>>> 10963 Berlin
>>> www.wikimedia.de
>>>
>>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
>>>
>>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
>>> unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt
>>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
>>> ___
>>> Wikidata-tech mailing list
>>> wikidata-t...@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
>>>
>>
>>
>> --
>> Amir Sarabadani
>> Software Engineer
>>
>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>> Tel. (030) 219 158 26-0
>> http://wikimedia.de
>>
>> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
>> Wissens frei teilhaben kann. Helfen Sie uns dabei!
>> http://spenden.wikimedia.de/
>>
>> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
>> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
>> Körperschaften I Berlin, Steuernummer 27/029/42207.
>>
>
>
> --
> Amir Sarabadani
> Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Tel. (030) 219 158 26-0
> http://wikimedia.de
>
> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> Wissens frei teilhaben kann. Helfen Sie uns dabei!
> http://spenden.wikimedia.de/
>
> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> Körperschaften I Berlin, Steuernummer 27/029/42207.
>


-- 
Amir Sarabadani
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens fr

Re: [Wikitech-l] Proposal for removal without deprecation of MagicWord::clearCache

[Aryeh Gregor]

On Mon, Jul 30, 2018 at 4:28 PM, Stephan Gambke via Wikitech-l
 wrote:
> There have been several proposals for removal without deprecation in the past 
> few days, each with more impact on existing code - from "virtually certain 
> nobody ever used" to "the fix is trivial".
>
> I know it is annoying as heck to keep outdated code, but if you ask me, 
> unless there is an immediate compelling reason for the removal the 
> deprecation policy should not be bypassed. And in my opinion "I did not find 
> any extension that uses this method" is not sufficient. Not all 
> extensions/skins are hosted on WMF servers. And being trivial to fix does not 
> help, either. It would be great if everybody could monitor MW master, run CI 
> tests regularly for all their extensions and fix any breakages. Unfortunately 
> many people do not have the time to do that, so they need the deprecation 
> period (and at times the deprecation popups to even notice that something is 
> wrong).
>
> So, I would request to keep the clearCache() method for the foreseen 
> deprecation period. (And generally speaking, though not explicitly required 
> by the deprecation policy and probably not applicable in this instance, it 
> would be great if deprecated methods would actually be kept in working 
> condition and not reduced to a (literally) empty husk just to stay in line 
> with the letter of the policy.)

I personally have maintained local hacks to various software packages
in the past, and definitely symphathize with the headache of having to
keep up-to-date.  However, the deprecation policy doesn't try to help
people who aren't keeping up-to-date.  It will just postpone their
extension's breakage by two releases, not decrease the overall amount
of work they have to do.  The deprecation policy is just to give
extension maintainers a bit of breathing room so they don't have to
scramble to update their extension before it breaks.

As far as these specific proposals, the wording of the deprecation
policy suggests that removal without deprecation should only be cases
where it's absolutely necessary.  I think we should also allow such
removal when it seems probable that basically nobody actually used the
feature.  For instance, the feature being removed in this case is only
useful for tests of MagicWords.  It doesn't seem worth the hassle to
go through a whole procedure to remove it if it will probably affect
nobody, and almost certainly affect at most one or two people, and the
affected parties will just have to make a one-line change.  I'm not
saying the burden on extension developers will be zero, but a lot more
time will be needed for us to follow the procedure than we'd save
anyone.

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Train] 1.32.0-wmf.14 status update

[Željko Filipin]

Hi,

all problems have been resolved and I have deployed 1.32.0-wmf.14 to all
wikis about 30 minutes ago. So far logs look good to me.

I have started working on incident report:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20180724-Train

If you can help me finish it, please do.

Thanks,

Željko

On Thu, Jul 26, 2018 at 8:27 PM Željko Filipin 
wrote:

> Hi,
>
> the train is still blocked.
>
> Problems from the previous e-mail message are resolved, the new version is
> deployed to groups 0 and 1 but can proceed no further until these issues
> are resolved:
>
> - MapCacheLRU::has called with invalid key. Must be string or integer -
> https://phabricator.wikimedia.org/T200456
>
> Once these issues are resolved train can resume. If the task is resolved
> on Friday or over the weekend, the train will resume Monday.
>
> Thank you for your help resolving these issues!
>
> Željko
>
>
> On Thu, Jul 26, 2018 at 5:33 PM Željko Filipin 
> wrote:
>
>> The train is still blocked.
>>
>> Problems from the previous e-mail message are resolved, the new version
>> is deployed to groups 0 and 1 (except wikidata) but can proceed no further
>> until these issues are resolved:
>>
>> * Wikidata dispatching stuck - https://phabricator.wikimedia.org/T200420
>>
>> Once these issues are resolved train can resume. Since it's already
>> Thursday afternoon for me, if the task is resolved on Friday or over the
>> weekend, the train will resume Monday.
>>
>> Thank you for your help resolving these issues!
>>
>> Željko
>>
>> On Wed, Jul 25, 2018 at 6:14 PM Željko Filipin 
>> wrote:
>>
>>> Hi,
>>>
>>> 1.32.0-wmf.14 version of MediaWiki is blocked [0].
>>>
>>> The new version is deployed to group 0 [1], but can proceed no further
>>> until these issues are resolved:
>>>
>>> - Wikibase\DataModel\Entity\EntityIdParsingException $serialization must
>>> not be an empty string - https://phabricator.wikimedia.org/T200340
>>>
>>> - wmf.14 failing to execute ThumbnailRender jobs "error:
>>> ThumbnailRenderJob::run: HTTP request failure" -
>>> https://phabricator.wikimedia.org/T200346
>>>
>>> Once these issues are resolved train can resume. If these issues are
>>> resolved on a Friday the train will resume Monday.
>>>
>>> Thank you for your help resolving these issues!
>>>
>>> Željko
>>> --
>>> [0] https://phabricator.wikimedia.org/T191060
>>> [1] https://tools.wmflabs.org/versions
>>>
>>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] MW Train status: Incomplete rollout, blocked on two issues

[Željko Filipin]

This will be my final notification about 1.32.0-wmf.13. I have started
working on incident report:

https://wikitech.wikimedia.org/wiki/Incident_documentation/20180717-Train

Any help in completing the report is greatly appreciated.

Thanks,

Željko

On Tue, Jul 24, 2018 at 7:50 PM Željko Filipin 
wrote:

> Good news!
>
> A few minutes ago we have deployed 1.32.0-wmf.13 to all wikis. Logs look
> OK to me so far.
>
> A big thank you to everybody that reported problems and helped resolve
> them. I want to tank every single one of you, but I'm to tired right now,
> so I'll leave that for tomorrow.
>
> For now, I would like to thank Tyler Cipriani and Greg Grossmeier for all
> the help with finally getting 1.32.0-wmf.13 everywhere.
>
> Željko
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Proposal for removal without deprecation of MagicWord::clearCache

[Stephan Gambke via Wikitech-l]

There have been several proposals for removal without deprecation in the past 
few days, each with more impact on existing code - from "virtually certain 
nobody ever used" to "the fix is trivial".

I know it is annoying as heck to keep outdated code, but if you ask me, unless 
there is an immediate compelling reason for the removal the deprecation policy 
should not be bypassed. And in my opinion "I did not find any extension that 
uses this method" is not sufficient. Not all extensions/skins are hosted on WMF 
servers. And being trivial to fix does not help, either. It would be great if 
everybody could monitor MW master, run CI tests regularly for all their 
extensions and fix any breakages. Unfortunately many people do not have the 
time to do that, so they need the deprecation period (and at times the 
deprecation popups to even notice that something is wrong).

So, I would request to keep the clearCache() method for the foreseen 
deprecation period. (And generally speaking, though not explicitly required by 
the deprecation policy and probably not applicable in this instance, it would 
be great if deprecated methods would actually be kept in working condition and 
not reduced to a (literally) empty husk just to stay in line with the letter of 
the policy.)

Stephan

‐‐‐ Original Message ‐‐‐
On July 30, 2018 2:51 PM, Aryeh Gregor  wrote:

> Patch to remove: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/447647
> Bug: https://phabricator.wikimedia.org/T200247
> Existing uses: 
> https://codesearch.wmflabs.org/search/?q=MagicWord%3A%3AclearCache&i=nope&files=&repos=
>
> I'm in the middle of creating a MagicWordFactory service to replace
> the MagicWord static methods. This obsoletes the clearCache() method.
> Instead, just reset the service with resetServiceForTesting(). Since
> clearCache() is used in only three places in public code, and is only
> useful to begin with for tests, and the fix is trivial, I propose
> removing it without deprecation rather than bothering to maintain code
> for compatibility.
>
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Proposal for removal without deprecation of MagicWord::clearCache

[Aryeh Gregor]

Patch to remove: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/447647
Bug: https://phabricator.wikimedia.org/T200247
Existing uses: 
https://codesearch.wmflabs.org/search/?q=MagicWord%3A%3AclearCache&i=nope&files=&repos=

I'm in the middle of creating a MagicWordFactory service to replace
the MagicWord static methods.  This obsoletes the clearCache() method.
Instead, just reset the service with resetServiceForTesting().  Since
clearCache() is used in only three places in public code, and is only
useful to begin with for tests, and the fix is trivial, I propose
removing it without deprecation rather than bothering to maintain code
for compatibility.

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikidata-tech] Normalization of change tag schema

[Amir Sarabadani]

Hey,
Using the new table as backend of Special:Tags (and similar APIs) is now
enabled everywhere. Contact me if there's any issues with that.

Best

On Wed, 25 Jul 2018 at 19:17, Amir Sarabadani 
wrote:

> Hello,
> One update regarding this.
> We enabled using the new table for Special:Tags in several large wikis
> which caused a massive improvement in the performance of the page. For
> example loading Special:Tags on Wikidata used to take around a minute and
> now it takes less than a second. English Wikipedia is down from ten seconds
> to less than one and so on.
>
> There is a lot of work needs to be done and maintenance scripts is being
> ran to backpopulate the ct_tag_id column in change_tag table (If you want
> to follow the progress, see https://phabricator.wikimedia.org/T193873)
> and then we need start reading from the new table in mediawiki and finally
> we can drop ct_tag column entirely. If you want to help in review, writing
> code or anything, just let me know.
>
> Best
>
> On Wed, 27 Jun 2018 at 15:15, Léa Lacroix 
> wrote:
>
>> Hello all,
>>
>> Our team is refactoring some code around the change tags on Recent
>> Changes. This can impact people using the database on ToolForge.
>>
>> Currently, the tags are stored in the table change_tag in the column
>> ct_tag.
>>
>> In the next days, we will add a column ct_tag_id with a unique
>> identifier for these tags. A new table change_tag_def that will store
>> the tag id, the message, and more information like how many times this tag
>> is used on the local wiki.
>>
>> On the long term, we plan to drop the column ct_tag since the tag will
>> be identified with ct_tag_id.
>>
>> This change will happen on:
>> - French Wikipedia: Monday July 2nd
>> - All other wikis: from July 9th
>>
>> If there is any problem (trouble with saving edits, slow down of recent
>> changes…) please  create a subtask of T185355
>>  or contact Ladsgroup
>> .
>>
>> Cheers,
>> --
>> Léa Lacroix
>> Project Manager Community Communication for Wikidata
>>
>> Wikimedia Deutschland e.V.
>> Tempelhofer Ufer 23-24
>> 10963 Berlin
>> www.wikimedia.de
>>
>> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
>>
>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
>> unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt
>> für Körperschaften I Berlin, Steuernummer 27/029/42207.
>> ___
>> Wikidata-tech mailing list
>> wikidata-t...@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikidata-tech
>>
>
>
> --
> Amir Sarabadani
> Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Tel. (030) 219 158 26-0
> http://wikimedia.de
>
> Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> Wissens frei teilhaben kann. Helfen Sie uns dabei!
> http://spenden.wikimedia.de/
>
> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> Körperschaften I Berlin, Steuernummer 27/029/42207.
>


-- 
Amir Sarabadani
Software Engineer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de

Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/

Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l