[Wikitech-l] Re: Changes in schema of pagelinks tables

[Amir Sarabadani]

Hi,
It's me again. The data have been migrated in all wikis except Turkish
Wikipedia and Chinese Wikipedia (last two wikis of s2) and these two wikis
will be also migrated in a week or so. So if you rely on the old columns of
pagelinks, you need to migrate them to use the normalized schema ASAP.

We will start dropping the old columns gradually in two weeks.

Best

Am Di., 16. Jan. 2024 um 11:56 Uhr schrieb Amir Sarabadani <
asarabad...@wikimedia.org>:

> Hello,
> The data migration for several sections have been completed. We will start
> dropping the old columns on s4
> <https://noc.wikimedia.org/conf/dblists/s4.dblist> and s5
> <https://noc.wikimedia.org/conf/dblists/s5.dblist> this and next week and
> right after will start dropping the old columns on s3
> <https://noc.wikimedia.org/conf/dblists/s3.dblist>.
>
> The data has not been fully migrated in s1 (enwiki) and s8 (wikidata) but
> migrated in some wikis of the rest of the sections.
>
> If your tools rely on pagelinks, you might need to update your tools now.
>
> Best
>
> Am Mi., 18. Okt. 2023 um 13:46 Uhr schrieb Amir Sarabadani <
> asarabad...@wikimedia.org>:
>
>> (If you don’t work with pagelinks table, feel free to ignore this message)
>>
>> Hello,
>>
>> Here is an update and reminder on the previous announcement
>> <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/U2U6TXIBABU3KDCVUOITIGI5OJ4COBSW/>
>> regarding normalization of links tables that was sent around a year ago.
>>
>> As part of that work, soon the pl_namespace and pl_title columns of
>> pagelinks table will be dropped and you will need to use pl_target_id
>> joining with the linktarget table instead. This is basically identical to
>> the templatelinks normalization that happened a year ago.
>>
>> Currently, MediaWiki writes to both data schemes of pagelinks for new
>> rows in all wikis except English Wikipedia and Wikimedia Commons (we will
>> start writing to these two wikis next week). We have started to backfill
>> the data with the new schema but it will take weeks to finish in large
>> wikis.
>>
>> So if you query this table directly or your tools do, You will need to
>> update them accordingly. I will write a reminder before dropping the old
>> columns once the data has been fully backfilled.
>>
>> You can keep track of the general long-term work in T300222
>> <https://phabricator.wikimedia.org/T300222> and the specific work for
>> pagelinks in T299947 <https://phabricator.wikimedia.org/T299947>. You
>> can also read more on the reasoning in T24
>> <https://phabricator.wikimedia.org/T24> or the previous announcement
>> <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/U2U6TXIBABU3KDCVUOITIGI5OJ4COBSW/>
>> .
>>
>> Thank you,
>> --
>> *Amir Sarabadani (he/him)*
>> Staff Database Architect
>> Wikimedia Foundation <https://wikimediafoundation.org/>
>>
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [Breaking Change] Deprecation of indirect calls to IDBAccessObject constants

[Amir Sarabadani]

Hello,
For many years, some classes in MediaWiki implemented the IDBAccessObject
interface (which only provides several public constants) and then would
call them indirectly (e.g. self::READ_NORMAL). Since these constants were
public, other parts of MediaWiki started to call them through the
implementing class as well (e.g. calling User::READ_LATEST).

This is inconsistent with the access pattern of other constants in
MediaWiki. it's also confusing (e.g. it's unclear to a newcomer why
UserFactory is implementing IDBAccessObject) and it's prone to clashes
(e.g. BagOStuff class has a clashing constant).

Since it's not possible to trigger a deprecation warning in such cases, It
can't follow the usual stable interface policy path [1] and here is the
email to wikitech-l as required by the policy.

In three weeks we will remove indirect access to these constants by
removing IDBAccessObject from implementation. This will take effect from
1.42 release.

Classes that won't have those constants anymore are including but not
limited to:
* User
* WikiPage (and its subclasses)
* File (and its subclasses)
* Title
* ActorStore
* RevisionStore
* UserOptionsManager
* And more.

To find such cases in extensions you maintain, you can search for
"(?https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/993112>) in
T354194 <https://phabricator.wikimedia.org/T354194>.

[1] From the policy
<https://www.mediawiki.org/wiki/Stable_interface_policy#Hard_deprecation>:
"If it is not reasonably possible for the deprecated code to emit
deprecation warnings, hard deprecation can be applied by announcing the
removal on wikitech-l in a timely manner."

Thank you and sorry for the inconvenience,
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Changes in schema of pagelinks tables

[Amir Sarabadani]

Hello,
The data migration for several sections have been completed. We will start
dropping the old columns on s4
<https://noc.wikimedia.org/conf/dblists/s4.dblist> and s5
<https://noc.wikimedia.org/conf/dblists/s5.dblist> this and next week and
right after will start dropping the old columns on s3
<https://noc.wikimedia.org/conf/dblists/s3.dblist>.

The data has not been fully migrated in s1 (enwiki) and s8 (wikidata) but
migrated in some wikis of the rest of the sections.

If your tools rely on pagelinks, you might need to update your tools now.

Best

Am Mi., 18. Okt. 2023 um 13:46 Uhr schrieb Amir Sarabadani <
asarabad...@wikimedia.org>:

> (If you don’t work with pagelinks table, feel free to ignore this message)
>
> Hello,
>
> Here is an update and reminder on the previous announcement
> <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/U2U6TXIBABU3KDCVUOITIGI5OJ4COBSW/>
> regarding normalization of links tables that was sent around a year ago.
>
> As part of that work, soon the pl_namespace and pl_title columns of
> pagelinks table will be dropped and you will need to use pl_target_id
> joining with the linktarget table instead. This is basically identical to
> the templatelinks normalization that happened a year ago.
>
> Currently, MediaWiki writes to both data schemes of pagelinks for new rows
> in all wikis except English Wikipedia and Wikimedia Commons (we will start
> writing to these two wikis next week). We have started to backfill the data
> with the new schema but it will take weeks to finish in large wikis.
>
> So if you query this table directly or your tools do, You will need to
> update them accordingly. I will write a reminder before dropping the old
> columns once the data has been fully backfilled.
>
> You can keep track of the general long-term work in T300222
> <https://phabricator.wikimedia.org/T300222> and the specific work for
> pagelinks in T299947 <https://phabricator.wikimedia.org/T299947>. You can
> also read more on the reasoning in T24
> <https://phabricator.wikimedia.org/T24> or the previous announcement
> <https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/U2U6TXIBABU3KDCVUOITIGI5OJ4COBSW/>
> .
>
> Thank you,
> --
> *Amir Sarabadani (he/him)*
> Staff Database Architect
> Wikimedia Foundation <https://wikimediafoundation.org/>
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Comments like Google Docs

[Amir Sarabadani]

I don't think comments on live articles would be useful (as mentioned
above) but there are many cases which it can be a game changer. I can think
of a couple:

   - Collaborative drafting: When you want to draft policy, a proposal, a
   new article, etc. with one or more fellow Wikimedians
   - Using as a replacement for google docs in many private wikis (as both
   for Wikimedia and third party corporate installations). We already have
   collab-pad
   
   which allows users to turn VE into an etherpad. I would love to see that
   get off the ground.
   - Reviewing a nomination for good or featured article: For example, take
   a look at a recent FAC
   
.
   The reviewer highlights a sentence in the discussion page and makes a
   comment about it and that's quite...labor intensive. Having a way to allow
   commenting which would be only visible to a small group of users would be
   quite nice.

It shouldn't be too hard to implement but not super trivial either.
MediaWiki is open source and extendable (via extensions) and I would
appreciate any work on it!

Am Fr., 8. Dez. 2023 um 21:09 Uhr schrieb Risker :

> I don't have a particular opinion on this, except that inline comments
> that are publicly visible must be able to be moderated by the community,
> and that smaller communities in particular should be able to opt out of
> this extension.  They sound like a great idea, but we're much more likely
> to get comments like "this isn't true" or "this [highly unreliable website]
> disagrees", and that just creates problems for readers. We can assume good
> faith until the cows come home, but we should also be realistic and realize
> that those comments are going to make our articles look more like Twitter
> and Facebook; that is, they'll be opportunities for disinformation.
>
> Risker/Anne
>
> On Fri, 8 Dec 2023 at 05:43, Kosta Harlan  wrote:
>
>> There is an extension
>> https://www.mediawiki.org/wiki/Extension:InlineComments
>>
>> There's some past discussion of this type of thing here:
>>
>>
>>- https://phabricator.wikimedia.org/T149667
>>- https://phabricator.wikimedia.org/T312760
>>
>>
>> Kosta
>>
>> On 8. Dec 2023 at 10:12:08, Felipe Schenone  wrote:
>>
>>> Hi! I'm thinking on writing a gadget to add inline comments to articles,
>>> similar to how Google Docs comments work.
>>>
>>> However, I'm sure I recently read somewhere about someone developing an
>>> extension or something with the same goal, but now I can't find it
>>> anywhere. Anyone knows?
>>>
>>> Thanks!
>>> ___
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Announcing Codex 1.0

[Amir Sarabadani]

I want to echo what the DJ has said. I managed to write a pretty decent
gadget in just a couple of hours thanks to Codex, something that used to
take days at least. This is really exciting to see. Thank you to all who
have worked on this tirelessly for years. Kudos and congratulations.

Am Mi., 25. Okt. 2023 um 20:28 Uhr schrieb Derk-Jan Hartman <
d.j.hartman+wmf...@gmail.com>:

> Congratulations team (and the predecessor teams) !
>
> We started talking about needing a new way forward all the way back in
> 2017, with an RFC started in 2019, we choose Vue in 2020 and the teams have
> been building new foundations since, so far culminating in Codex !!
>
> I'm sure there is much work left to do as the web and MediaWiki is ever
> evolving. Just recently the web finally added CSS nesting
> 
> for instance, and new features like :has()
>  and :user-valid()
>  selectors.
> Unfortunately Wikipedia/MediaWiki and specifically user scripts can have
> problems trying to keep up with so many changes. It is great to see that
> es6 , less
> , import,
> require  and
> recently even source maps  are
> slowly finding their way into our ecosystem, all while supporting as many
> older browsers as possible and building one of the fastest websites in the
> world.
>
> These improvements are only possible because dedicated people work hard to
> analyze the problems and these changes, devise solutions to slowly
> introduce them without breaking everything, align peers, fight to get time
> to work on it. All to shepherd them into existence, laborious and
> challenging as it may be at times.
>
> Keep up the great work, but lets take a moment and celebrate the official
> birth of an entire new design system !
>
> DJ
>
>
>
> On Wed, Oct 25, 2023 at 6:07 PM Roan Kattouw 
> wrote:
>
>> Today the Design Systems Team
>>  is announcing the
>> release of Codex 1.0!
>> What is Codex?
>>
>> Codex  is the new design system
>> for Wikimedia. Over the past 2 years, the Design Systems Team and
>> contributors from the Wikimedia Foundation, Wikimedia Deutschland, and the
>> volunteer communities have collaborated to create a centralized design
>> system to serve Wikimedia projects. Codex provides more equitable
>> experiences for all Wikimedia movement participants, and makes it easier
>> and faster to design and build consistent user interfaces. With Codex, we
>> aim to enable more people to contribute to the mission.
>>
>> Codex provides a library of design tokens
>> ,
>> user interface components
>> , and
>> catalog of icons
>>  to use with
>> these components. Through the Codex Figma libraries, designers can reuse
>> these shared components
>> 
>> , tokens
>> ,
>> and assets
>> 
>>  in
>> their designs. For developers, Codex provides components built with Vue.js,
>> as well as some CSS-only components that do not require JavaScript to use.
>>
>> Codex is already being used for Wikifunctions
>> , Vector 2022
>> ,
>> the Growth Mentor Dashboard
>>  and Impact
>> Module ,
>> the New Pages Feed
>> 
>> , MediaSearch ,
>> NearbyPages ,
>> QuickSurveys ,
>> and ReadingLists .
>> Projects currently under development using Codex include Accessibility
>> for reading
>>  and
>> the Incident Reporting System
>> .
>>
>> Codex provides a set of core components
>> 

[Wikitech-l] Changes in schema of pagelinks tables

[Amir Sarabadani]

(If you don’t work with pagelinks table, feel free to ignore this message)

Hello,

Here is an update and reminder on the previous announcement
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/U2U6TXIBABU3KDCVUOITIGI5OJ4COBSW/>
regarding normalization of links tables that was sent around a year ago.

As part of that work, soon the pl_namespace and pl_title columns of
pagelinks table will be dropped and you will need to use pl_target_id
joining with the linktarget table instead. This is basically identical to
the templatelinks normalization that happened a year ago.

Currently, MediaWiki writes to both data schemes of pagelinks for new rows
in all wikis except English Wikipedia and Wikimedia Commons (we will start
writing to these two wikis next week). We have started to backfill the data
with the new schema but it will take weeks to finish in large wikis.

So if you query this table directly or your tools do, You will need to
update them accordingly. I will write a reminder before dropping the old
columns once the data has been fully backfilled.

You can keep track of the general long-term work in T300222
<https://phabricator.wikimedia.org/T300222> and the specific work for
pagelinks in T299947 <https://phabricator.wikimedia.org/T299947>. You can
also read more on the reasoning in T24
<https://phabricator.wikimedia.org/T24> or the previous announcement
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/U2U6TXIBABU3KDCVUOITIGI5OJ4COBSW/>
.

Thank you,
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Three new rdbms changes

[Amir Sarabadani]

Hello,

We have several more changes to the rdbms library’s interface to announce.
We previously made such announcements in May 2023
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/GATRFC63FPQJ6Z43WZDIKSHYZQJCI62H/>
and February 2023
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/YNLVJVTYTK3IGQF4HY7ITGMIJ2W5Q7CG/>
.
Query builders for all common query types

We now have query builders for all common query types:

   -

   SELECT
   -

   INSERT
   -

   UPDATE
   -

   DELETE
   -

   UPSERT
   -

   REPLACE


We encourage you to migrate calls using the old methods (such as
Database::insert) to use the query builders instead. Starting from 1.41,
all the old methods are considered internal and might change without prior
notice.

See T335377 <https://phabricator.wikimedia.org/T335377> and previous
announcements such as the one in May 2023
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/GATRFC63FPQJ6Z43WZDIKSHYZQJCI62H/>
for more information.
Script for migrating to SelectQueryBuilder

Given that there are many calls to Database::select in extensions we wrote
a simple script to help with the migration to SelectQueryBuilder. It is not
perfect and doesn’t cover many aspects such as joins but it takes away a
lot of tedious work.

Using that script we made hundreds of migrations in the MediaWiki core. As
of now, we have more than 800 calls to newSelectQueryBuilder() and only
roughly 100 direct calls to Database::select() left in core.

For more information, check out
https://gitlab.wikimedia.org/ladsgroup/migrateselect
Access to external clusters simplified

If you ever had to make database queries against external cluster databases
(such as extension1, otherwise known as x1), you will like this change.
Many of our extensions store their data in x1, for example: Echo,
UrlShortener, Cognate, Translate, ContentTranslation, GrowthExperiments,
Campaigns and many more. And it requires a lot of complexity with many
caveats (see T330590 <https://phabricator.wikimedia.org/T330590> for more
information).

Now, an extension can introduce one or more virtual domains in their
extension.json and then if you set the configuration such as:

$wgVirtualDomainsMapping['urlshortener'] = [ 'cluster' => 'extension1',
'db' => 'wikishared' ];

then $lbf->getPrimaryDatabase( 'urlshortener' ) will make the right
connection. If it’s not set in the mapping of virtual domains
configuration, it simply makes the connection to the local database. You
don’t need to do any special coding anymore which simplifies the logic a
lot and removes the need for extra configuration variables. This also helps
in facilitating proper support of external clusters in the database updater
and improvements in testing in the future.

You can take a look at the example of url shortener (gerrit:963293
<https://gerrit.wikimedia.org/r/c/mediawiki/extensions/UrlShortener/+/963293>)
for how adoption is done.

Thank you,

--
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Wikitech-ambassadors] [invitation to participate] Listening Sessions for the Technical Decision-Making Retrospective

[Amir Sarabadani]

Hi,
I think the email somehow didn't reach wikitech-l and some other mailing
lists (except ambassadors)

Am Mo., 4. Sept. 2023 um 15:46 Uhr schrieb Benoît Evellin (Trizek) <
bevel...@wikimedia.org>:

> Hello all,
>
>
> Sorry for cross-posting.
>
> The Technical Decision-Making Forum Retrospective
>  team invites
> you to join one of our “listening sessions” about the Wikimedia's technical
> decision-making processes.
>
> We are running the listening sessions to provide a venue for people to
> tell us about their experience, thoughts, and needs regarding the process
> of making technical decisions across the Wikimedia technical spaces. This
> complements the survey
> , which closed
> on August 7.
>
> Who should participate in the listening sessions?
>
> People who do technical work that relies on software maintained by the
> Wikimedia Foundation (WMF) or affiliates. If you contribute code to
> MediaWiki or extensions used by Wikimedia, or you maintain gadgets or tools
> that rely on WMF infrastructure, and you want to tell us more than could be
> expressed through the survey, the listening sessions are for you.
>
> How can I take part in a listening session?
>
> There will be four sessions on two days, to accommodate all time zones.
> The two first sessions are scheduled:
>
>- Wednesday, September 13, 14:00 – 14:50 UTC
>
>- Wednesday, September 13, 20:00 – 20:50 UTC
>
>
>
> The sessions will be held on the Zoom platform.
>
> If you want to participate, please sign up for the one you want to attend:
> <
> https://www.mediawiki.org/wiki/Technical_decision_making/Listening_Sessions>.
> If none of the times work for you, please leave a message on the talk page.
> It will help us schedule the two last sessions.
>
> The sessions will be held in English. If you want to participate but you
> are not comfortable speaking English, please say so when signing up so that
> we can provide interpretation services.
>
> The sessions will be recorded and transcribed so we can later go back and
> extract all relevant information. The recordings and transcripts will not
> be made public, except for anonymized summaries of the outcomes.
>
> What will the Retrospective Team do with the information?
>
> The retrospective team will collect the input provided through the survey,
> the listening sessions and other means, and will publish an anonymized
> summary that will help leadership make decisions about the future of the
> process.
>
> In the listening sessions, we particularly hope to gather information on
> the general needs and perceptions about decision-making in our technical
> spaces. This will help us understand what kind of decisions happen in the
> spaces, who is involved, who is impacted, and how to adjust our processes
> accordingly.
>
> Are the listening sessions the best way to participate?
>
> The primary way for us to gather information about people’s needs and
> wants with respect to technical decision making was the survey
> . The
> listening sessions are an important addition that provides a venue for free
> form conversations, so we can learn about aspects that do not fit well with
> the structure of the survey.
>
> In addition to the listening sessions and the survey, there are two more
> ways to share your thoughts about technical decision making: You can post
> on the talk page
> ,
> or you can send an email to .
>
> Where can I find more information?
>
> There are several places where you can find more information about the
> Technical Decision-Making Process Retrospective:
>
>-
>
>The original announcement about the retrospective from Tajh Taylor:
>
> https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/IF3OJARL4WYKJYQAS7NTP4FT72R626EG/
>-
>
>The Technical Decision-Making Process general information page:
>https://www.mediawiki.org/wiki/Technical_decision_making
>-
>
>The Technical Decision-Making Process Retrospective page:
>
> https://www.mediawiki.org/wiki/Technical_decision_making/Technical_Decision-Making_Process_Retrospective_and_Consultation_plan
>-
>
>The Phabricator ticket: https://phabricator.wikimedia.org/T333235
>
>
> Who is running the technical decision making retrospective?
>
> The retrospective was initiated by Tajh Taylor. The core group running the
> process consists of Moriel Schottlender (chair), Daniel Kinzler, Chris
> Danis, Kosta Harlan, and Temilola Adeleye. You can contact us at <
> tdf-retro-2...@lists.wikimedia.org>.
>
> Thank you for participating!
>
> --
>
> Benoît Evellin - Trizek (he/him)
>
> 

[Wikitech-l] Re: ORES To Lift Wing Migration

[Amir Sarabadani]

Am Fr., 4. Aug. 2023 um 12:53 Uhr schrieb Brian Wolff :

> > Our understanding is that renaming extensions in MediaWiki is a long
> and complicated process, so we'll likely not be able to rename it in the
> foreseeable future.
>
> Why?
>
> Renaming is usually a bad thing because it often confuses the hell out of
> users, but from a technical perspective it is pretty trivial.
>

Renaming an extension that's deployed to production is basically
impossible. e.g. attempt of renaming Extension:Flow to
StructuredDiscussions.

Basically the only viable option is to undeploy the extension, rename the
extension, and deploy it again.


> --
> Bawolff
>
> On Friday, August 4, 2023, Luca Toscano  wrote:
>
>> Hi Amir!
>>
>> Answering inline:
>>
>> On Thu, Aug 3, 2023 at 10:11 PM Amir E. Aharoni <
>> amir.ahar...@mail.huji.ac.il> wrote:
>>
>>>
>>> The email says that "All ML models currently accessible on ORES are also
>>> currently accessible on Lift Wing", and if I understand correctly, this
>>> means that this feature in Recent Changes will keep working. Do I
>>> understand correctly? :)
>>>
>>
>> Definitely yes, we are working on migrating the ORES extension to Lift
>> Wing, without any change required for users. The tracking task is
>> https://phabricator.wikimedia.org/T319170. At the moment all wikis with
>> the ORES extension enabled, except fi/en/wikidata, are already using models
>> from Lift Wing.
>>
>>
>>> In addition, I have some followup questions:
>>>
>>> 1. The MediaWiki extension that implements the frontend in Recent
>>> Changes is itself named "ORES". It's an internal name that isn't seen much
>>> by wiki editors except if they go to Special:Version or to translatewiki.
>>> Nevertheless, as the time goes by, seeing the old name may start getting
>>> weird. So what's the plan about it? Will this extension remain as is? Will
>>> it be renamed? Will it be replaced with a new frontend extension in the
>>> foreseeable future?
>>>
>>
>> This is a good question and we don't have a definitive answer at the
>> moment. Our understanding is that renaming extensions in MediaWiki is a
>> long and complicated process, so we'll likely not be able to rename it in
>> the foreseeable future. We would definitely like to add more models to RC
>> Filters, for example Revert Risk (for the curious, see
>> https://meta.wikimedia.org/wiki/Machine_learning_models/Proposed/Language-agnostic_revert_risk),
>> but we are not sure yet if it is worth to create a new extension or just to
>> expand the ORES one. We'll get back to this list as soon as we have a
>> better plan :)
>>
>>
>>> 2. Back when ORES was originally developed and deployed around 2017,
>>> several wiki editors' communities participated in the development by
>>> adapting the product to the needs of their wikis and languages by
>>> translating the ORES extension's user interface and, more importantly, by
>>> labelling a sample of several thousands of diffs from their wiki using the
>>> Wikilabels tool. The communities that did that whole process were, more or
>>> less, the communities to which this Recent Changes enhancement was
>>> deployed. Will anything like that have to be done again along with the move
>>> away from ORES?
>>>
>>
>> The first goal of Lift Wing is to provide a more modern and easy-to-use
>> infrastructure to host models at the WMF, for internal teams and for the
>> community. The focus of the Machine Learning team is to provide
>> infrastructure to run models on, so other teams and the community will be
>> able to propose what to host and we'll vet what is possible and what not
>> (following strict criteria like security of data and PII, model
>> architecture feasibility, etc..). Once a model is deployed on Lift Wing,
>> there will be a team or a community group owning it, namely responsible for
>> its development in terms of features etc.. (more info in
>> https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing#Hosting_a_model
>> ).
>> To summarize:
>> * All the work done so far with ORES models will be preserved, it is
>> already available on Lift Wing and anybody can use it. We hope that it is
>> now easier to play with model servers and improve them (for WMF and the
>> community), but we are open to any suggestion and feedback about how to
>> improve it. For the curious, more details in the Lift Wing Wikitech page (
>> https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing).
>> * The future work will be split into two main areas (as I see it):
>> ** The ML team will keep working on improving the infrastructure,
>> documentation, performance, etc.. of Lift Wing, to provide better tools and
>> data access for any new idea related to models and their usage. We'll
>> maintain the infrastructure with monitoring/alarms/etc.., so the day-to-day
>> ops will not fall on the model owners (WMF and community), so that they
>> will be able to concentrate themselves only on the models and their future
>> steps.
>> ** Other WMF teams like 

[Wikitech-l] [Breaking Change] Two changes regarding searching for proto-relative URLs in external links

[Amir Sarabadani]

Hello,

As part of work to make storage of external links in MediaWiki continue to
scale without risking site stability (T312666
<https://phabricator.wikimedia.org/T312666>), we are deprecating most of
the special functionalities around proto-relative URLs (URLs that start
with // instead of https:// or http://).

Proto-relative URLs were beneficial a decade ago, when Wikimedia projects
were being served in both encrypted and unencrypted traffic (http and
https). However, since 2015, all of our traffic has been served encrypted
only, and this functionality doesn’t provide much user benefit any more for
Wikimedia wikis. With HTTP/2, a similar circumstance applies to all
MediaWiki users.

As well as being low-value, our external links storage (the externallinks
table) has grown to be one of the biggest tables for each production wiki.
This is due to many duplications of URL information, added to serve
different use cases. With the changes, we are removing these duplications,
and some of the functionality. You can read more about the work in T312666
<https://phabricator.wikimedia.org/T312666>.
Storage of proto-relative URLs has changed to only store HTTPS URLs

Previously, if a proto-relative was added in an edit, MediaWiki internally
treated it as two links one with http:// and one with https://. From this
week forward, for all Wikimedia wikis, the storage will change to store
only https:// URLs. Once those wikis are switched to read the new database
schema, the links will be presented as https only in Special:LinkSearch and
their API counter-parts. This means effectively a proto-relative external
link will be treated like a HTTPS one. This change will also apply to
non-Wikimedia wikis using MediaWiki 1.41+.
expandurl option is deprecated and ignored in the exturlusage and extlinks
MediaWiki action API modules

This means “expandurl” argument in exturlusage and extlinks
<https://www.mediawiki.org/wiki/API:Extlinks> API modules will be ignored
and proto-relative URLs will be always expanded to HTTPS. This will happen
any time a wiki is switched to read from the new externallinks fields. (You
can track the progress in T335343
<https://phabricator.wikimedia.org/T335343>) This change will also apply to
non-Wikimedia wikis using MediaWiki 1.41+.

If your wiki heavily uses proto-relative URLs in articles' wikitext, we
recommend changing them to https instead which also improves storage as
every proto-relative URLs takes up two rows.

Thank you,

--

Amir Sarabadani, Staff Database Architect

James Forrester, Staff Software Engineer

Timo Tijhof, Principal Performance Engineer
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Amir Sarabadani]

I made the patch: https://gerrit.wikimedia.org/r/c/labs/codesearch/+/919924
I can merge and deploy it soon.

Am Di., 16. Mai 2023 um 00:23 Uhr schrieb Daniel Zahn :

> I have used codesearch to search for the config of codesearch with things
> like
>
>
> https://codesearch.wmcloud.org/search/?q=codesearch===
>
> I did find the puppet module codesearch and a hound config file in there.
>
> But somehow I have not found yet where the "gerrit-replica" URL s
> configured.
>
> Do you see it? Could that be in Horizon Hiera instead of the repos maybe?
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>


-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Gerrit replica downtime (30 minutes) tomorrow Tue, 16 May 13:00-15:00 UTC

[Amir Sarabadani]

This means codesearch will be affected (and won't get updated) and possibly
even will be down during that time.

Best

Am Mo., 15. Mai 2023 um 22:03 Uhr schrieb Tyler Cipriani <
tcipri...@wikimedia.org>:

> Hello
>
> The read-only Gerrit replica[0] will be down for 30 minutes tomorrow (Tue,
> 16 May 2023) between 13:00–15:00 UTC[1] due to network switch upgrades in
> codfw row D[2].
>
> During this window, git reads from the replica will not work.
>
> To my knowledge, this affects bots which rely on the replica for git read
> operations.
>
> Apologies for any inconvenience.
>
> Tyler Cipriani (he/him)
> Engineering Manager, Release Engineering
> Wikimedia Foundation
>
> [0]: 
> [1]: 
> [2]: 
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Word embeddings / vector search

[Amir Sarabadani]

On top of the ones mentioned,
ores topic detection model (the
one that says what wikiproject an article belongs to, an example
) has been
using word embedding since 2018-ish.

HTH

Am Di., 9. Mai 2023 um 22:10 Uhr schrieb Isaac Johnson :

> +1 to the suggestion to connect with the Search team. Also a few more
> thoughts about vector / natural-language search and its relevance to
> Wikimedia from my perspective in Research:
>
>- The common critique of lexical / keyword-based search and why folks
>point to vector / embedding-based search is handling more natural-language
>queries (e.g., "What are the different objectives of the United Nations
>Sustainable Development Goals?" vs. "UN SDG"). The former has a lot of
>words in it that lead to keyword overlap with less-relevant pages so
>keyword-based search doesn't do as well. The latter is much more direct and
>even matches an existing redirect on Wikipedia to the article on UN
>Sustainable Development Goals, so our existing keyword-based search handles
>it very well.
>- Most existing users of Wikimedia's search are probably doing
>something closer to the latter above -- i.e. using pretty exact keywords to
>navigate to a specific page (or find it exists). This is backed up by the
>data: 80% of searches on Wikipedia are auto-completed directly to
>article pages
>
> .
>In that sense, the system is working quite well! The Search team also has
>added quite a bit of normalization into the pipeline (see
>
> https://diff.wikimedia.org/2023/04/28/language-harmony-and-unpacking-a-year-in-the-life-of-a-search-nerd/
>for a fun overview). For the more complicated natural-language queries to
>find relevant Wikipedia articles, my sense is that folks using natural
>language searches are probably doing that within external search engines,
>which have huge teams/infrastructure to support this, and then clicking
>through to Wikipedia.
>- That said, there are probably use-cases where natural-language
>search would be more valuable. For example, within new interaction domains
>such as chat-bots or for new editors / developers who don't yet know the
>exact terminology to search for but want to do generic things like get
>access to Toolforge or find out how to add a link to a page. I've been
>putting together an example of this for Wikitech for the upcoming Hackathon
>(details ) and others have
>proposed e.g., this for Project pages to help editors find answers to
>questions about editing (details
>).
>- Finally, there's a second, related aspect to this which is the size
>and diversity of a given document. Within the Wikipedia article namespace,
>documents are generally about a single, constrained topic. So the fact that
>lexical search systems like Elasticsearch operate at the document-level is
>a very good fit -- i.e. index all the keywords for a given article
>together. When thinking about other namespaces like Project/Help pages or
>Wikitech documentation, a single page can be much larger and be about far
>more diverse topics. This presents further challenges to finding good
>keyword-overlap because often the search would ideally find a very specific
>paragraph in a much larger document about many other things. Vector search
>doesn't directly solve this but in practice, folks tend to learn embeddings
>for smaller passages than an entire doc -- e.g., sections or even
>paragraphs within the section. For that reason alone, I suspect vector
>search will do better for namespaces outside of the article namespace on
>Wikipedia. Whether it's worth the cost is a separate question as it also
>introduces substantial new challenges in keeping the embeddings up-to-date
>:)
>
> Hope that helps.
>
> Best,
> Isaac
>
> On Tue, May 9, 2023 at 2:10 PM Dan Andreescu 
> wrote:
>
>> I encourage you to reach out to the search team, they're lovely folks and
>> even better engineers.
>>
>> On Tue, May 9, 2023 at 1:53 PM Lars Aronsson  wrote:
>>
>>> On 2023-05-09 09:27, Thiemo Kreuz wrote:
>>> > I'm curious what the actual question is. The basic concepts are
>>> > studied for about 60 years, and are in use for about 20 to 30 years.
>>>
>>> Sorry to hear that you're so negative. It's quite obvious that this is
>>> not
>>> currently used in Wikipedia, but is presented everywhere as a novelty
>>> that has not been around for 20 or 30 years.
>>>
>>> >
>>> https://www.elastic.co/de/blog/introducing-approximate-nearest-neighbor-search-in-elasticsearch-8-0
>>> > 

[Wikitech-l] Re: Python requests broken by urllib3 version 2.x

[Amir Sarabadani]

You can also create an empty virtual env, install all requirements and then
do
pip freeze > requirements.txt

That should take care of pinning

Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
lucas.werkmeis...@wikimedia.de>:

> For the general case of Python projects, I’d argue that a better solution
> is to adopt the lockfile pattern (package-lock.json, composer.lock,
> Cargo.lock, etc.) and pin *all* dependencies, and only update them when
> the new versions have been tested and are known to work. pip-tools
>  can help with that, for example (
> requirements.in specifies “loose” dependencies; pip-compile creates a
> pinned requirements.txt; pip-sync installs it; pip-compile -U upgrades
> requirements.txt later; you check both requirements.in and
> requirements.txt into version control.) But I don’t know if that applies
> in your integration/config case.
>
> Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso :
>
>> Hello,
>>
>> This is for python projects.
>>
>> Today, May 4th, urllib3  has
>> released a new major version 2.0.2 which breaks the extremely popular
>> requests  library.
>>
>> The fix is to pin urllib3<2 to prevent the new major version from being
>> installed (example
>> 
>> ).
>>
>> https://phabricator.wikimedia.org/T335977
>>
>> Upstream issue: https://github.com/psf/requests/issues/6432
>>
>>
>> Antoine "hashar" Musso
>> Wikimedia Release Engineering
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Lucas Werkmeister (he/er)
> Software Engineer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Phone: +49 (0)30-577 11 62-0
> https://wikimedia.de
>
> Imagine a world in which every single human being can freely share in the
> sum of all knowledge. Help us to achieve our vision!
> https://spenden.wikimedia.de
>
> Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> Körperschaften I Berlin, Steuernummer 27/029/42207.
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Two breaking changes related to Databases

[Amir Sarabadani]

Hello,

We have two breaking changes to announce.
Breaking change to Rdbms Database subclasses

There are no Database subclasses currently known in Codesearch (besides
those built-in to core). But, since the class is documented as "stable to
extend" in API, an announcement is in order.

First, an explanation on what is happening. Currently, queries flow in the
Rdbms library as follows: When Database::select() or similar is called
(directly, or via the query builders), the Rdbms library internally builds
the SQL text and then calls the general-purpose function Database::query().
Subsequently, ::query() then runs several regexes over the SQL text to
figure out information about this query (which tables are affected, whether
it’s a write query, what the query “verb” is, and so on). This is taxing
and slow. The appservers at WMF, for example, spend 0.34% of CPU on web
requests just to extract information from already-formatted SQL queries for
logging purposes. This logic drastically reduces readability and
maintainability of Rdbms internals.

To fix this, we've internally introduced a new class called Query, to
represent the SQL text in a more structured way in the first place. This
avoids having to first format structure data into SQL text only to
reverse-engineer it right afterwards.

This change requires changing the protected Database::executeQuery() method
to no longer accept a string argument and instead require a Query object.
Additionally IDatabase::getTempTableWrites() has become a private method.
See the patch <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/910756> or the
ticket <https://phabricator.wikimedia.org/T326181> for more information.

The public Database::query() method continues to support a string argument,
but now also supports a Query object. When given a string, it falls back to
the old regexes to extract and create a Query object. With the newly
introduced UnionQueryBuilder, we believe there are no longer cases where
MediaWiki extensions have to call Database::query, and can instead adopt
query builders for all their queries. Direct calls to Database::query
outside of MediaWiki core are now highly discouraged.

Breaking change to typehints of IDatabase

Many parts of core typehint to IDatabase, but most times these always
return a replica database. For example, ApiBase::getDB(). This means the
interface technically allows calling insert(), and this is indeed valid,
but would also immediately throw a fatal error once the method is entered.
Going forward, most of these will be typehinted to IReadableDatabase
instead, which is a narrower interface (34 public methods instead of 83).

This is not a breaking change in practice, because calls to those methods
would already produce an error, but the typehint makes it official now. This
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/911269> is the patch
changing ApiBase::getDB(), and we expect more such changes to other parts
of MediaWiki over the coming months. Those will not be individually
announced.

Best
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] New query builders and IConnectionProvider

[Amir Sarabadani]

Hello,

As part of ongoing refactors and improvements to the Rdbms component in
MediaWiki, we have made some changes that might affect your work. Read
the February
announcement
<https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/YNLVJVTYTK3IGQF4HY7ITGMIJ2W5Q7CG/>
for previous changes in this area.
New DeleteQueryBuilder and UnionQueryBuilder

We introduced the chainable SelectQueryBuilder
<https://www.mediawiki.org/wiki/Manual:Database_access#SelectQueryBuilder>
in 2020 to create SELECT queries. This "fluent" API style allows each
segment of the query to be set through a discoverable method name with rich
documentation and parameter types. Last February we introduced
UpdateQueryBuilder in the same vein.

The new DeleteQueryBuilder similarly supersedes use of IDatabase::delete()
for building DELETE operations. Check change 913646
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/913646/1/includes/user/UserOptionsManager.php#417>
for how this is used in practice.

Union queries are fairly rare in our codebases. Previously, one would have
to bypass the query builders (by calling the underlying SQL text formatters
like Database::selectSqlText) and pass several raw SQL strings through
Database::unionQueries and then to Database::query. As you can see, this is
not optimal. The new UnionQueryBuilder enables you to combine several
SelectQueryBuilder objects and then use the familiar fetchResultSet()
method on the resulting UnionQueryBuilder object. You can see an example of
this in change 906101
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/906101/5/maintenance/findOrphanedFiles.php>
.
IConnectionProvider typehint

In February, we introduced the LBFactory->getPrimaryConnection() and
LBFactory->getReplicaConnection() methods to improve ergonomics around
getting database connections, and the IReadableDatabase typehint for
replica connections.

We're now introducing the IConnectionProvider typehint as a stable
interface holding the above methods without the rest of LBFactory. Meaning,
if you all you need is a connection, you can typehint to
IConnectionProvider. This is an extremely narrow interface (four public
methods!) compared to LBFactory (42 public methods). This will reduce
coupling and should ease testing as well. We already adopted
IConnectionProvider in several MediaWiki core components, which made a
notable dent in MediaWiki's PhpMetrics complexity score
<https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/complexity.html>
.

This backwards-compatible change can be adopted through only a change in
typehint. MediaWiki’s LBFactory service implements IConnectionProvider.
Dependency injection remains unchanged.

You can see an example in change 913649
<https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DiscussionTools/+/913649/2/includes/SubscriptionStore.php>.
We recommend changing variable names like $lbFactory to $dbProvider for
consistency in usage.

Future changes

We will continue transforming MediaWiki core to adopt query builders,
IConnectionProvider, and IReadableDatabase. And, we'll continue helping
various core components and extensions to phase out their direct calls to
LoadBalancer::getConnection(). This should improve code health,
readability, testability, and overall coupling scores in MediaWiki.

Our next announcement will introduce InsertQueryBuilder, and we're
considering a query builder for "UPSERT" queries as well. Lastly, we are
planning to discourage remaining use of raw SQL in "where" conditions by
introducing an expression builder (T210206
<https://phabricator.wikimedia.org/T210206>).
You can help

Nearly every feature of MediaWiki requires retrieving or storing
information from a database at one point or another. Reworking these is a
long tail of effort. Any help in adopting query builders, and updating code
away from ILoadBalancer and ILBFactory would be really appreciated.
Together we can improve readability and code health (now with less
footguns!).

Until the next update,

--

Amir Sarabadani (he/him), Staff Database Architect

Timo Tijhof, Principal Performance Engineer
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] [CoC] Proposed amendment: Ban on soliciting tech support for hate groups

[Amir Sarabadani]

Hello,

In accordance with CoC's amendment policy
. The committee
proposes an amendment on behalf of one of the community members to be added
in unacceptable behavior section
:

Soliciting help, support or technical assistance for websites whose
predominant activity or content is behaviour that explicitly and
significantly violates provisions of the code of conduct is forbidden in
Wikimedia technical spaces. For example, websites advocating violence or
hate speech are not welcome to ask for support on Wikimedia support forums.

This is similar to w:en:Wikipedia:No Nazis


This will be open for three months for community feedback in
https://www.mediawiki.org/wiki/Topic:Xgg6dvg2athdb1h7 and if the community
reaches consensus and the committee won't oppose it, It will be added to
the policy.
Best
Amir (he/him),
On behalf of the CoC committee
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Reflecting on my listening tour

[Amir Sarabadani]

This is not really about Selena's email but it's so nerdy I still want to
talk about what tech debt is and what it isn't.

The word tech debt at the beginning meant a very specific thing: When an
engineer takes a shortcut to deliver a feature faster. Daniel has a pretty
good essay on tech debt: https://www.mediawiki.org/wiki/The_Tech_Debt_Trap

But the term slowly became so overused that engineers used it to refer to
any invisible work they couldn't tie into a feature. At this point, people
refer to tech debt as the general maintenance practically. And that led to
non-tech people to slowly disregard maintenance work because everything is
tech debt now.

What tech debt is not (this is not a MECE list):

   - Keep the lights on: OS upgrades, security updates, hw maintenance,
   incidents, ...
   - Bitrot: The code stays the same but the underlying technology changes,
   the usage pattern changes, requirements changes, etc.
   - Over-engineering: This is way more common than you think. Building an
   overly complex palace that no one can maintain anymore because no one
   understands it and everyone is afraid to remove anything. This is sorta the
   opposite of tech debt, people spent more resources than they should and
   it's still hurting us.
   - Plain bad engineering: People make mistakes, typos are easily fixable
   while architectural mistakes are not, sometimes we know it because of
   hindsight, sometimes it was obvious from the beginning.
   - Limiting architecture: Many architectural decisions come in the form
   of trade-offs, a classic example is performance vs. security. If someone
   wants something and it's not possible due to architectural limitations, it
   doesn't necessarily mean a bad thing. We sacrificed something in favor of
   the other.
   - Code hygiene: Improvements on readability and maintainability of the
   code.
   - Scale/Performance/Security considerations: I have seen that people see
   a prototype and want it in Wikipedia right now. I understand it but any
   solution on such a large scale comes with all sorts of edge cases and
   hidden costs. People bash MediaWiki for being too complex and messy but
   anything you build at the scale of Wikipedia is going to be complex and
   messy. If a solution takes years, sometimes that's the only option.

What I want to say is that while we do have a tech debt problem in
Wikimedia, we also have a lot of bitrot problems too, or any other "slowing
down development" kind of problem. Some cases it's fixable, in some cases
it is not. What is needed is more resources on maintenance which is
happening and is making me extremely happy. Whether we call that paying
back tech debt or not.


Am Fr., 14. Apr. 2023 um 23:44 Uhr schrieb Brian Wolff :

> Perhaps i am hyperfocused on technical debt in the sense of improving the
> abstractions used in mediawiki. The phrasing around sustainability
> especially leads me in that direction. However, technical debt is certainly
> a broad concept and can mean a lot of things.
>
> The common thread in the examples you cited seem to be things that have
> fallen through the ownership cracks. I'm not sure its the case that
> engineers aren't incentivized to fix these, so much as there are no natural
> engineers to be incentivized due to team structure (scribunto is an
> exception but i would disagree with that task's inclusion for reasons that
> get off topic). On the other hand, perhaps a different incentivization
> structure would encourage people to branch out more.
>
> I think it is especially telling that 3 (or 4 even) of these tasks are
> multimedia related, given that wmf hasn't had a multimedia team in a very
> long time [SDC does not count], and definitely not one focused on the
> backend. There are quite a few multimedia related areas in desperate need
> of love (it is 2023 and video uploads are limited to 4gb with the flakiest
> upload process known to man).
>
>
> It was also pointed out to me on irc, that many critical workflows in the
> community depend on toolforge tools that have very limited volunteer
> maintainership. A sort of https://xkcd.com/2347/ situation. Just because
> they're not "production" we often pretend they don't exist. Regardless of
> how we label them, in the end it doesn't make a difference to the end user,
> and the fragility of that ecosystem is a form of technical debt that is
> often overlooked.
>
>
> So i guess it all depends on what is meant by "technical debt"
> --
> Brian
>
> On Friday, April 14, 2023, Andre Klapper  wrote:
>
>> On Thu, 2023-04-13 at 20:06 -0700, bawolff wrote:
>> > > "I think there are lots of promising opportunities to incentivise
>> > > people to pay off technical debt and make our existing stack more
>> > > sustainable. Right now there are no incentives for engineers in
>> > > this regard."
>> >
>> > Interesting. Personally to me, it can sometimes feel like we never
>> > stop talking about technical debt. While I think paying off technical
>> > 

[Wikitech-l] Re: March 2023 Datacenter Switchover

[Amir Sarabadani]

It's a bit complicated.
When SRE sets the read-only mark, they start counting from that time and it
starts propagating which takes a while to be actually shown to all users
but some users might still see the RO error while some actual writes are
happening somewhere else because the cache is not invalidated yet (I think
it has a TTL of 5 seconds but I need to double check). We still consider
that as RO time because it's affecting users regardless.

HTH



Am Mi., 1. März 2023 um 18:06 Uhr schrieb Dušan Kreheľ <
dusankre...@gmail.com>:

> Clément Goubert and everybody,
>
> I analyzed https://stream.wikimedia.org/v2/stream/recentchange and i
> have the another results.
>
> Last change (before migration): 2023-03-01T14:00:30
> First change (after migration):2023-03-01T14:02:05
> Result: Down time (14:00:31 to 14:02:05) is 94s.
>
> I think that analysis is more authoritative. I think it analyzes based
> on something like REQUEST_TIME in PHP.
>
> Dušan Kreheľ
>
>
> 2023-03-01 16:30 GMT+01:00, Clément Goubert :
> > Dear Wikitechians,
> >
> > Dear colleagues,
> >
> > The switchover process requires a *brief read-only period for all
> > Foundation-hosted wikis*, which started at *14:00 UTC on Wednesday March
> > 1st*, and lasted *119 seconds*. All our public and private wikis
> continued
> > to be available for reading as usual. Users saw a notification of the
> > upcoming maintenance, and anyone still editing was asked to try again in
> a
> > few minutes.
> >
> > As a side note, with other SREs we have been trying to discern the effect
> > of the Switchover in many of the graphs we have to monitor the
> > infrastructure in https://grafana.wikimedia.org during Switchover. In
> many,
> > it's impossible to tell the event. The most discernible graph we have is
> of
> > the edit rate, which can be viewed here: Grafana
> > <
> https://grafana-rw.wikimedia.org/d/00208/edit-count?from=167767380=1=167768100
> >.
> > Can you spot it? See the attached picture to help:
> >
> > I am extending thanks to everyone that was also present on IRC, helping
> out
> > in any way that they could. Thanks as well to Community Relations who
> > notified communities of the read-only window ahead of time. And thanks to
> > everyone that contributed to MultiDC
> > ,
> > especially Performance for pushing forward with the last parts of it,
> > allowing us to perform this Switchover faster and with more confidence
> than
> > ever before.
> >
> > If you wanna relive through the Switchover, here's a link to a capture
> > of Listen
> > to Wikipedia  during
> the
> > Switchover: Listen to the Switchover
> > <
> https://drive.google.com/file/d/1jqQUVCq3ksjOM5bKoIfCZ5Zt9RRW1Nl_/view?usp=share_link
> >
> > (spoiler:
> > the part with no sounds is the switchover)
> >
> > A similar event will follow a few weeks later, when we move back to
> > Virginia. This is currently scheduled for *Wednesday, April 26th*.
> > Thank you,
> >
> > On Tue, Feb 21, 2023 at 1:55 PM Clément Goubert 
> > wrote:
> >
> >> Dear Wikitechians,
> >>
> >> I would like to remind you that the datacenter switchover will happen on
> >> *Wednesday
> >> March 1st* starting at *14:00 UTC.*
> >>
> >> Please refer to the original email for any additional information. As
> >> always, you can reach out to me directly or the SRE team in
> >> #wikimedia-sre
> >> on IRC with any question, or through Phabricator.
> >>
> >> Thank you,
> >>
> >> On Tue, Feb 14, 2023 at 1:58 PM Clément Goubert  >
> >> wrote:
> >>
> >>> Dear Wikitechians,
> >>>
> >>> On *Wednesday March 1st*, the SRE team will run a planned data center
> >>> switchover, moving all wikis from our primary data center in Virginia
> to
> >>> the secondary data center in Texas. This is an important periodic test
> >>> of
> >>> our tools and procedures, to ensure the wikis will continue to be
> >>> available
> >>> even in the event of major technical issues in our primary home. It
> also
> >>> gives all our SRE and ops teams a chance to do maintenance and upgrades
> >>> on
> >>> systems in Virginia that normally run 24 hours a day.
> >>>
> >>> The switchover process requires a *brief read-only period for all
> >>> Foundation-hosted wikis*, which will start at *14:00 UTC on Wednesday
> >>> March 1st*, and will last for a few minutes while we execute the
> >>> migration as efficiently as possible. All our public and private wikis
> >>> will
> >>> be continuously available for reading as usual, but no one will be able
> >>> to
> >>> save edits during the process. Users will see a notification of the
> >>> upcoming maintenance, and anyone still editing will be asked to try
> >>> again
> >>> in a few minutes.
> >>>
> >>> CommRel has already begun notifying communities of the read-only
> window.
> >>> A similar event will follow a few weeks later, when we move back to
> >>> Virginia. This is currently scheduled 

[Wikitech-l] New DB connect/query methods for MediaWiki

[Amir Sarabadani]

lica connection objects. It’ll also make the coupling between the rest
of MediaWiki and extension with the rdbms library looser
<https://en.wikipedia.org/wiki/Loose_coupling#In_programming>.

Note that the IDatabase::query() is not part of IReadableDatabase. This
method is discouraged as it executes raw SQL that may include write
queries. If you call this method, continue to typehint IDatabase until you
transition to safe methods like select() or SelectQueryBuilder.



Example adoption:


   -


   
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/892031


== 3. Introducing UpdateQueryBuilder ==

What changed:

In 2020, we introduced a chainable SelectQueryBuilder
<https://www.mediawiki.org/wiki/Manual:Database_access#SelectQueryBuilder>
for SELECT queries. To complement this, we now added a class called
UpdateQueryBuilder, for UPDATE queries. The following existing code uses
update() with unnamed positional parameters:

$db->update(
'image',
[ 'img_name' => 'foo' ],
[ 'img_name' => 'bar' ],
__METHOD__
);


You may now use the fluent style like so:

$db->newUpdateQueryBuilder()

 ->update( 'image' )

 ->set( [ 'img_name' => 'foo' ] )

 ->where( [ 'img_name' => 'bar' ])

 ->caller( __METHOD__ )
->execute();

Reasoning:

Similar to the SelectQueryBuilder, the fluent style is more readable. It
simplifies building of conditions, is less prone to mistakes, and is more
aligned with widely-used frameworks such as Doctrine (PHP) and Jooq (Java).

We have had at least two data corruption incidents in production due to
passing the wrong order of $set and $where condition (swapping them by
mistake). This reduces the chance of such data corruptions happening, which
alone was worth developing the builder for, before we get to the improved
ergonomics.

Example adoption:


   -


   
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/889627/5/includes/Category/Category.php

Adaption is being tracked in T330640
<https://phabricator.wikimedia.org/T330640>.

What is next:

We will continue to develop query builders for DELETE and INSERT queries.

The work on migrating calls from ::select() to SelectQueryBuilder is
on-going (T311866 <https://phabricator.wikimedia.org/T311866>).

Thank you
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Avoid invisible characters in page titles

[Amir Sarabadani]

Disallowing invisible characters or cleaning them is a bad idea.

Invisible characters are actually heavily used in many languages including
Persian (and part of the official manual of style of the language taught in
schools) it is downright wrong to check and fix those in many wikis in
those languages.

Also many wikis have titles in other languages such as wiktionaries or
redirects in a different languages (For example:
https://en.wikipedia.org/w/index.php?title=%D8%AA%D9%87%D8%B1%D8%A7%D9%86=no)
which means removing ZWNJ or similar characters would be also unacceptable
in English Wiktionary or English Wikipedia as well.

There are some exemptions though: Two invisible characters are wrong, or an
invisible character at the end or beginning. But all of these are cases in
Persian language and another language might actually allow that as well.

Best

Am Di., 17. Jan. 2023 um 12:48 Uhr schrieb Martin Domdey :

> Thank you,
>
> it seems that there's nobody working on it anymore, right?
>
> Kind regards,
> Martin ...
>
>
>
> Am Di., 17. Jan. 2023 um 12:28 Uhr schrieb Andre Klapper <
> aklap...@wikimedia.org>:
>
>> Hi,
>>
>> On Tue, 2023-01-17 at 12:03 +0100, Martin Domdey wrote:
>> > isn't it better to avoid invisible characters in page titles
>> > while creating the pages?
>> >
>> > Please look here, there has been problems with invisible characters
>> > working with it when parsing or page linking those page titles with
>> > invisible unicode
>> > characters: https://de.wikipedia.org/wiki/Benutzer_Diskussion:Wurgl#L
>> > iste_der_Biografien/Ci
>>
>> See also
>>
>> https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#UTF-8_ZERO_WIDTH_SPACE_in_page_title
>>
>> > Instead of this there will never be a problem when invisible
>> > characters within the page title name will be deleted when
>> > creating the page.
>> >
>> > What do you think about it and what technical approaches do
>> > already exist? How are LTR and RTL marks dealt if creating pages with
>> > them?
>>
>> See https://phabricator.wikimedia.org/maniphest/query/GDxAs4QdEDTG/#R
>> for related bugs, and a ticket about improving cleanupTitles.php.
>>
>> Cheers,
>> andre
>> --
>> Andre Klapper (he/him) | Bugwrangler / Developer Advocate
>> https://blogs.gnome.org/aklapper/
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Movereason-dropdown

[Amir Sarabadani]

Hi,
I suggest creating a ticket so it can be implemented in MediaWiki itself,
it sounds like a rather easy and useful feature.

Best

Am Mi., 30. Nov. 2022 um 03:03 Uhr schrieb Bináris :

> As I got no answer, I was forced to solve it myself with JS:
> https://hu.wikipedia.org/wiki/Szerkeszt%C5%91:Bin%C3%A1ris/Renamereasons.js
>
> Bináris  ezt írta (időpont: 2022. nov. 21., H,
> 14:15):
>
>> Hi,
>>
>> we have a MediaWiki:Deletereason-dropdown which appears as a dropdown on
>> Special:Delete. How can I create a similar dropdown for frequent
>> renaming/moving reasons?
>>
>> --
>> Bináris
>>
>
>
> --
> Bináris
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: No Commons Uploads

[Amir Sarabadani]

Hi,
It is known and people are working on it. You can check
https://www.wikimediastatus.net/ for the status of the incident.

Best

Am Di., 1. Nov. 2022 um 23:33 Uhr schrieb Martin Domdey :

> Hi,
>
> is it known, that there cannot be uploaded a file to Commons right now?
>
> Martin ...
>
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Feedback wanted: PHPCS in a static types world

[Amir Sarabadani]

Hi,
A great initiative, thank you!

I am generally in favor of this proposal but just want to give a cautionary
tale. It's a bit off-topic but important.
Given that there is no actual enforcing mechanism for the documentation
typehints, some of them have actually drifted from reality. I caused a UBN
bug once by relying on the documentation for the type of a variable. So my
request is to avoid mass migration of documentation type hints to php type
declaration.

Best

Am So., 30. Okt. 2022 um 14:02 Uhr schrieb Daniel Kinzler <
dkinz...@wikimedia.org>:

> Thank you for suggesting this!
> I agree that type declaration is preferable to type documentation, and
> that type documentation is often redundant if type declaration is present.
>
> However, we can't always use type declarations. For instance, union types
> are quite useful, since PHP doesn't have method overloading. And union type
> declarations will only become available in PHP 8. So we'll have a mix of
> declared and un-declared parameters and fields for a while. I think we
> should still require type documentation if there is no type declaration -
> and of course, if a method has any @param tags, it needs to have all of
> them.
>
> Also there is the notable exception of the array type. Saying that
> something is an array is generally insufficient, we should say at least
> whether it's a list or an associative array, and document the type of the
> array elements or and/or well-known keys.
> And we should be careful that we don't end up discouraging documentation
> of the meaning of a parameter. The barrier to adding some explanation of
> the meaning of a parameter is lower if there is already a @param string
> $name line. If I'd first have to create a doc block, I may just not add the
> documentation at all. We should still encourage having doc blocks in all
> but the most trivial cases (simple constructors, getters and setters
> probably don't need one).
>
> -- daniel
>
> PS: I'm not sure I like constructor argument property promotion... For
> very simple value objects that might be nice, but generally, I fear it will
> make it harder to see all fields declared on an object.
> Am 28.10.2022 um 16:03 schrieb Lucas Werkmeister:
>
> Hi all!
>
> In my opinion, MediaWiki’s PHPCS ruleset feels largely rooted in an older
> version of PHP, where static type declarations (formerly known as “type
> hints”) did not exist. As we move towards more modern code, I think some
> rules should be relaxed, and others adjusted. More specifically, I’d like
> to know if most people agree with the following propositions and conclusion:
>
> Proposition 1: *Some code is sufficiently documented by names and types*,
> and does not require additional documentation. Cases where additional
> documentation is required do certainly exist, but they can only be
> identified by human reviewers, not by automated tools.
>
> You can see this in our existing code wherever a doc comment specifies
> only a type (with @var, @param, or @return), but no additional text. For
> example, in CreditsAction
> ,
> nobody needs to be told that the LinkRenderer will be used to render
> links, or that the UserFactory creates User objects:
>
> class CreditsAction extends FormlessAction {
>
>   /** @var LinkRenderer */
>
>   private $linkRenderer;
>
>   /** @var UserFactory */
>   private $userFactory;
>
> Likewise, it’s not necessary to explain in great detail that the string
> returned by LinksTable::getTableName()
> 
> is the table name, that the $actor parameter of ActorCache::remove(
> UserIdentity $actor )
> 
> represents the actor to remove from the cache, or what the meaning of the 
> Message
> $m and returned MessageValue are in Message\Converter::convertMessage()
> 
> :
>
> /**
>
> * Convert a Message to a MessageValue
>
> * @param Message $m
>
> * @return MessageValue
>
> */
> public function convertMessage( Message $m ) {
>
> (I want to clarify that in this last example I’m only talking about the
> @param and @return tags that already don’t have any prose text. While the
> method comment “Convert a Message to a MessageValue” might also be
> considered redundant, I think this would be more contentious, and I’m not
> advocating for removing that today.)
>
> Proposition 2: *Adding types as static types is generally preferable.*
> Unlike doc comments, static types are checked at runtime and thus
> guaranteed to be correct (as long as the code runs at all); the small
> runtime cost should be partially offset by performance improvements in
> newer PHP versions, and otherwise considered to be worth it. New code
> 

[Wikitech-l] Request for feedback: Two changes related to databases in MediaWiki

[Amir Sarabadani]

Hello,
Two changes is being made on MediaWiki databases:

*First:*
We are reworking how extensions manage their database schema. The idea is
to basically deprecate LoadExtensionSchemaUpdates hook[1] and replace it
with structured values in extension.json.

This is needed to avoid arbitrary php code execution during database update
which can cause problems and was a contributing factor in one of our major
outages [2]. The other reason for this change is to be able to wire
abstract schema of extensions to core so it can be tested in CI (for
example for having a valid json, having prefixes set correctly, etc.) You
can see a POC of that in [3]

Given that once the change is merged to core, breaking changs to extension
registry schema would require a lot of work (updating the maniphest version
and so on), it's better to get it right before merge.

I have proposed something like [4]. For any feedback on this, please go to
https://phabricator.wikimedia.org/T237839,

*Second:*
We are working to provide a replacement for raw SQL for query conditions.
When the query condition is exact equality condition (e.g. rev_id = 123),
you don't need to provide raw SQL but for most other cases you need to
either provide it or build it via IDatabase methods. For example:
'rev_timestamp > ' . $db->addQuotes( $db->timestamp( $since ) ). You can
see an example of this in [5] (being changed to something much better via a
newly introduced method).

This is not optimal and not very developer friendly, to solve that, we are
proposing to have something like ExpressionBuilder::greater(
'rev_timestamp', $db->timestamp( $since ) ) supported in
SelectQueryBuilder::where().

You can see the POC in [6]. For any feedback on this please go to
https://phabricator.wikimedia.org/T210206

[1] https://www.mediawiki.org/wiki/Manual:Hooks/LoadExtensionSchemaUpdates
[2]
https://wikitech.wikimedia.org/wiki/Incidents/2020-04-07_Wikidata%27s_wb_items_per_site_table_dropped
[3]
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/816262/10/tests/phpunit/structure/AbstractSchemaValidationTest.php
[4] https://phabricator.wikimedia.org/T237839#8268173
[5]
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/832625/6/includes/api/ApiQueryDeletedrevs.php
[6] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/841940/

Thank you
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Proposed change to the MediaWiki database policy

[Amir Sarabadani]

Hello,
Here is the last call. If no objection comes in by the end of this week,
I'll add it to the policy.

Best

Am Fr., 15. Juli 2022 um 14:21 Uhr schrieb Amir Sarabadani <
asarabad...@wikimedia.org>:

> Hello,
> We have a proposal to make small adjustments to the MediaWiki database
> policy which would make abstract schema an explicit requirement.
>
> Abstract schema was approved by TechCom in
> https://phabricator.wikimedia.org/T191231 and now core and all
> WMF-deployed extensions have migrated to abstract schema.
>
> You can find the proposal in here. Please take a look and comment:
> https://www.mediawiki.org/wiki/Talk:MediaWiki_database_policy
>
> If there are no major objections by the end of August 2022, I will codify
> this into the policy.
>
> Best
> --
> *Amir Sarabadani (he/him)*
> Staff Database Architect
> Wikimedia Foundation <https://wikimediafoundation.org/>
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Deletion of 5000+ pages forbidden

[Amir Sarabadani]

Solving the ticket MatmaRex mentioned (T20493) would remove the 5,000 
limitation and address several other major tech debt in our databases. It's on 
the roadmap of the database work (T297633) and I hope to start it later in this 
year or early next. It will take a while but we will get there and the problem 
is on our radar.

HTH
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] MediaWiki core schema drifts with production databases

[Amir Sarabadani]

Hello,

After many years of work, I'm happy to announce a milestone in addressing
one of our major areas of tech debt in database infrastructure: we have
eliminated all schema drifts between MediaWiki core and production.

It all started six years ago when users in English Wikipedia reported that
checking history of some pages is quite slow *at random*. More in-depth
analysis showed the revision table in English Wikipedia was missing an
important index in some of the replicas. An audit of the schema of the
revision table revealed much bigger drifts in the revision table of that
Wiki. You can read more in its ticket: T132416
<https://phabricator.wikimedia.org/T132416>

Lack of schema parity between expectation and reality is quite dangerous.
Trying to force an index in code assuming it would exist in production
(under the same name) would cause fatal error every time it’s attempted.
Trying to write to a field that doesn’t exist is similar. Such changes
easily pass tests and work well in our test setups (such as beta cluster)
just to cause an outage in production.

If only one table in one Wiki had this many drifts, looking at all Wikis
and all tables became of vital importance. We have around ~1,000 wikis,
~200 hosts (each one hosting on average ~100 Wikis), and each Wiki has
around ~130 tables (half of them being tables from MediaWiki core) and each
table can have multiple drifts.

We slowly started looking for and addressing schema drifts five years ago
and later automated the discovery by utilizing abstract schema (before
that, the tool had to parse SQL) and discovered an overwhelming number of
drifts. You can look at the history of the work in T104459
<https://phabricator.wikimedia.org/T104459>.

Around fifty tickets addressing the drifts have been completed and they are
collected in T312538 <https://phabricator.wikimedia.org/T312538>. I suggest
checking some of them to see the scale of the work done. Each one of these
tickets took days to months of work to finish. Large number of them also
existed in primary databases, requiring a primary switchover and read-only
time for one or more Wikis. Each drift was different, in some cases, you
needed to change the code and not production so it needed a thorough
investigation.

Why do such drifts happen? The most common reason was when a schema change
happened in code but it was never requested to be applied in production.
For example, a schema change in code in 2007 led to having any wiki created
before that date to have a different schema than wikis created after it. We
introduced processes
<https://wikitech.wikimedia.org/wiki/Schema_changes#Workflow_of_a_schema_change>
and tooling to make sure this doesn’t happen anymore in 2015 but we still
needed to address previous drifts. The second common reason was when a host
didn’t get the schema change for various reasons (was out of rotation when
the schema was being applied, a shortcoming of the manual process). By
automating <https://wikitech.wikimedia.org/wiki/Auto_schema> most of the
schema change operational work we reduced the chance of such drifts from
happening as well.

After finishing core, we now need to look at WMF-deployed extensions,
starting with FlaggedRevs <https://phabricator.wikimedia.org/T313253> that,
while being deployed to only 50 wikis and having only 8 tables, has ~7,000
drifts. Thankfully, most other extensions are in a healthier state.

I would like to personally thank Manuel Arostegui and Jaime Crespo for
their monumental dedication to fix these issues in the past years. Also a
big thank you to several of our amazing developers, Umherirrender, James
Forrester and Sam Reed who helped on reporting, going through the history
of MediaWiki to figure out why these drifts happened, and helping build the
reporting tools.

Best
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Changes in schema of MediaWiki links tables

[Amir Sarabadani]

Hello,
Progress report:
The data migration has been everywhere except on English Wikipedia. So if
you use templatelinks for English Wikipedia, you still need to rely on the
old schema. The data migration will finish in English Wikipedia in around
two weeks.

We have stopped writing to the old columns in s5 and s2 this week (list of
wikis on s5
<https://noc.wikimedia.org/conf/highlight.php?file=dblists/s5.dblist> and list
of wikis on s2
<https://noc.wikimedia.org/conf/highlight.php?file=dblists/s2.dblist>) and
we will drop those columns in the aforementioned sections next week (August
15-21). The rest of the sections (except s1, English Wikipedia) will follow
the week after (August 22-28).

If you're relying on the old templatelinks schema, you are already getting
incorrect data and soon your code or queries will start to error due
querying missing fields.

We measured the impact of this normalization on the health of our
databases. Here are the reports for s5
<https://phabricator.wikimedia.org/T299417#8143953> and s2
<https://phabricator.wikimedia.org/T314041#8146798>.

Best

Am Di., 31. Mai 2022 um 15:42 Uhr schrieb Amir Sarabadani <
asarabad...@wikimedia.org>:

> An update on migration of templatelinks:
>
> With the exception of English Wikipedia and Wikimedia Commons, where the
> migration is still ongoing, every other wiki is now fully migrated to the
> normalized schema and is being read using the linktarget table in
> production.
>
> Please migrate your tools to use the new schema as we will slowly stop
> writing to tl_namespace and tl_title fields and drop them in the next month.
>
> Best
>
> Am Di., 29. März 2022 um 15:37 Uhr schrieb Amir Sarabadani <
> asarabad...@wikimedia.org>:
>
>> (If you don’t work with links tables such as templatelinks, pagelinks and
>> so on, feel free to ignore this message)
>>
>> TLDR: The schema of links tables (starting with templatelinks) will
>> change to have numeric id pointing to linktarget table instead of repeating
>> namespace and title.
>>
>> Hello,
>>
>> The current schema and storage of most links tables are: page id (the
>> source), namespace id of the target link and title of the target. For
>> example, if a page with id of 1 uses Template:Foo, the row in the database
>> would be 1, 6, and Foo (Template namespace has id of 6)
>>
>> Repeating the target’s title is not sustainable, for example more than
>> half of Wikimedia Commons database is just three links tables. The sheer
>> size of these tables makes a considerable portion of all queries slower,
>> backups and dumps taking longer and taking much more space than needed due
>> to unnecessary duplication. In Wikimedia Commons, on average a title is
>> duplicated around 100 times for templatelinks and around 20 times for
>> pagelinks. The numbers for other wikis depend on the usage patterns.
>>
>> Moving forward, these tables will be normalized, meaning a typical row
>> will hold mapping of page id to linktarget id instead. Linktarget is a new
>> table deployed in production and contains immutable records of namespace id
>> and string. The major differences between page and linktarget tables are:
>> 1- linktarget values won’t change (unlike page records that change with
>> page move) 2- linktarget values can point to non-existent pages (=red
>> links).
>>
>> The first table being done is templatelinks, then pagelinks, imagelinks
>> and categorylinks will follow. During the migration phase both values will
>> be accessible but we will turn off writing to the old columns once the
>> values are backfilled and switched to be read from the new schema. We will
>> announce any major changes beforehand but this is to let you know these
>> changes are coming.
>>
>> While the normalization of all links tables will take several years to
>> finish, templatelinks will finish in the next few months and is the most
>> pressing one.
>>
>> So if you:
>>
>>-
>>
>>… rely on the schema of these tables in cloud replicas, you will need
>>to change your tools.
>>-
>>
>>… rely on dumps of these tables, you will need to change your scripts.
>>
>>
>> Currently, templatelinks writes to both data schemes for new rows in most
>> wikis. This week we will start backfilling the data with the new schema but
>> it will take months to finish in large wikis.
>>
>> You can keep track of the general long-term work in
>> https://phabricator.wikimedia.org/T300222 and the specific work for
>> templatelinks in https://phabricator.wikimedia.org/T299417. You can also
>> read more on the reasoning in https://phabricator.wikimedia.org/T24.
>>
>> Thanks
>> --
>> *Amir Sarabadani (he/him)*
>> Staff Database Architect
>> Wikimedia Foundation <https://wikimediafoundation.org/>
>>
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Proposal on redesigning externallinks table: Request for feedback

[Amir Sarabadani]

Hello,
The externallinks table
<https://www.mediawiki.org/wiki/Manual:Externallinks_table> in MediaWiki is
among the largest in Wikimedia production. It's the second largest database
table in Wikimedia Commons (and will soon claim the first place, after
templatelinks
normalization <https://phabricator.wikimedia.org/T299417> completes).

There is a proposal to redesign this table. You can read more about it in
T312666 <https://phabricator.wikimedia.org/T312666>. If you use this table
or have some feedback about this proposal, please comment on the ticket.

Also, this means its schema and data will change soon. Be prepared to
update your tools and reports if you depend on the externallinks table.

Providing technical support to Wikimedia Commons is one of the official
goals of Wikimedia Foundation for this fiscal year, [1] and this redesign
will help address long-standing storage and database capacity issues of
this important project.

[1]: From the annual plan
<https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2022-2023>:
"Deepen our commitment to Knowledge as a Service by strengthening how we
prioritize and allocate product and tech support to 740+ Wikimedia
projects, starting with Wikimedia Commons and Wikidata."

Best
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Proposed change to the MediaWiki database policy

[Amir Sarabadani]

Hello,
We have a proposal to make small adjustments to the MediaWiki database
policy which would make abstract schema an explicit requirement.

Abstract schema was approved by TechCom in
https://phabricator.wikimedia.org/T191231 and now core and all WMF-deployed
extensions have migrated to abstract schema.

You can find the proposal in here. Please take a look and comment:
https://www.mediawiki.org/wiki/Talk:MediaWiki_database_policy

If there are no major objections by the end of August 2022, I will codify
this into the policy.

Best
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Abstract Schema: Milestone reached for WMF Deployed Extensions

[Amir Sarabadani]

Thank you so much for taking the lead on this Der Umherirrende!
Congratulations on such a massive achievement.

This will have an everlasting impact and I'm planning to use this work to
start fixing drifts of extensions soon (once we are done with fixing core
drifts)

Awesome work.

On Fri, Jul 15, 2022 at 3:49 AM Dan Andreescu 
wrote:

> Thank you so much for this work.  There are additional benefits as we all
> have a clearer understanding of how data moves around our various systems.
> My team (Data Engineering) is standing up a data catalog and we'll see
> about ingesting these schemas directly to centralize the information.
>
> On Thu, Jul 14, 2022 at 17:00 Der Umherirrende 
> wrote:
>
>> Hello,
>>
>> read previous update news about the abstract schema on the mediawiki-l
>> list under <
>> https://lists.wikimedia.org/hyperkitty/list/mediawik...@lists.wikimedia.org/thread/LLP6GJEGXJH2S62MU2UXXL4XVVPTYRMJ/
>> >
>>
>> The migration of mediawiki-core from sql schema file to the abstract
>> schema finished in May 2021, while the migration of WMF Deployed Extensions
>> is still ongoing on the goal task T261912[1] where 38 extensions with own
>> database tables are listed.
>>
>> But I am happy to write:
>> The migration of all WMF Deployed Extensions is now finished!
>>
>> At the end there are 35 extensions affected and all now shipes a json
>> file with the schema defintion and the sql file is generated from the json
>> file.
>> The most patch sets were merged this week and will be released with
>> REL1_39.
>>
>> To be able to provide the abstract schema file some extensions needs
>> updates beforehand, like standardise of timestamp columns or remove of
>> foreign keys.
>> The new json files makes it easier to analyse the schemas now and find
>> more optimisation with automation, like duplicate indexes or missing
>> primary keys.
>>
>> A benefit of the abstract schema is also to provide a postgres and sqlite
>> schema for free for all these extensions.
>> It also easier for schema changes in the future to keep the schema file
>> in sync and hopefully gives a better support for third party users with
>> postgres or sqlite.
>>
>> [Disclaimer: Do not expect that the extension will work directly with
>> postgres or sqlite. There are some RDBMS differences needs to handle in the
>> code, like different timestamp formats on postgres, but feel free to patch
>> the extension or fill bugs if you find issues, where the differences are
>> not addressed in the code.]
>>
>> There are still tasks to fix for better integration of the abstract
>> schema, like testing in CI if the schema file and the generated sql files
>> are in sync to avoid issues on partial updates[2]. This is only done for
>> core at the moment, please take a deeper look, when reviewing such patches
>> on extensions, thanks.
>> Knowing that all extensions have one of the three supported rdbms schema
>> makes it easier to enable voting CI jobs on merge for postgres and sqlite
>> in the future, but that needs fixing of tests first. Help is welcome here.
>>
>> Looking ahead:
>> In the future the abstract schema of the extension could also be used to
>> detect schema drifts on extension tables on wmf wikis, this happens at the
>> moment only for core[3].
>>
>> Hopefully some of the long standing requests to bundle popular extensions
>> with the tarball can be addressed[4], because the extensions are now
>> fulfill the requirement to support all rdbms. Testing is welcome for this
>> as well!
>>
>> It is possible that there are some areas needs updates, like
>> documentation or guidelines, feel free to improve if you find something
>> outdated.
>>
>>
>> I would like to thanks Sam Reed to start with the convert of extensions,
>> as well as various other developer converting extensions in the last
>> years/months! Great work I could benefit from while creating my own patch
>> sets.
>> Also thanks to James Forrester for the review of many of my patch sets.
>>
>> Greetings,
>> Der Umherirrende
>>
>> [1] https://phabricator.wikimedia.org/T261912
>> [2] https://phabricator.wikimedia.org/T261057
>> [3] https://drift-tracker.toolforge.org/
>> [4] https://phabricator.wikimedia.org/T305072
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org

[Wikitech-l] Re: Changes in schema of MediaWiki links tables

[Amir Sarabadani]

An update on migration of templatelinks:

With the exception of English Wikipedia and Wikimedia Commons, where the
migration is still ongoing, every other wiki is now fully migrated to the
normalized schema and is being read using the linktarget table in
production.

Please migrate your tools to use the new schema as we will slowly stop
writing to tl_namespace and tl_title fields and drop them in the next month.

Best

Am Di., 29. März 2022 um 15:37 Uhr schrieb Amir Sarabadani <
asarabad...@wikimedia.org>:

> (If you don’t work with links tables such as templatelinks, pagelinks and
> so on, feel free to ignore this message)
>
> TLDR: The schema of links tables (starting with templatelinks) will change
> to have numeric id pointing to linktarget table instead of repeating
> namespace and title.
>
> Hello,
>
> The current schema and storage of most links tables are: page id (the
> source), namespace id of the target link and title of the target. For
> example, if a page with id of 1 uses Template:Foo, the row in the database
> would be 1, 6, and Foo (Template namespace has id of 6)
>
> Repeating the target’s title is not sustainable, for example more than
> half of Wikimedia Commons database is just three links tables. The sheer
> size of these tables makes a considerable portion of all queries slower,
> backups and dumps taking longer and taking much more space than needed due
> to unnecessary duplication. In Wikimedia Commons, on average a title is
> duplicated around 100 times for templatelinks and around 20 times for
> pagelinks. The numbers for other wikis depend on the usage patterns.
>
> Moving forward, these tables will be normalized, meaning a typical row
> will hold mapping of page id to linktarget id instead. Linktarget is a new
> table deployed in production and contains immutable records of namespace id
> and string. The major differences between page and linktarget tables are:
> 1- linktarget values won’t change (unlike page records that change with
> page move) 2- linktarget values can point to non-existent pages (=red
> links).
>
> The first table being done is templatelinks, then pagelinks, imagelinks
> and categorylinks will follow. During the migration phase both values will
> be accessible but we will turn off writing to the old columns once the
> values are backfilled and switched to be read from the new schema. We will
> announce any major changes beforehand but this is to let you know these
> changes are coming.
>
> While the normalization of all links tables will take several years to
> finish, templatelinks will finish in the next few months and is the most
> pressing one.
>
> So if you:
>
>-
>
>… rely on the schema of these tables in cloud replicas, you will need
>to change your tools.
>-
>
>… rely on dumps of these tables, you will need to change your scripts.
>
>
> Currently, templatelinks writes to both data schemes for new rows in most
> wikis. This week we will start backfilling the data with the new schema but
> it will take months to finish in large wikis.
>
> You can keep track of the general long-term work in
> https://phabricator.wikimedia.org/T300222 and the specific work for
> templatelinks in https://phabricator.wikimedia.org/T299417. You can also
> read more on the reasoning in https://phabricator.wikimedia.org/T24.
>
> Thanks
> --
> *Amir Sarabadani (he/him)*
> Staff Database Architect
> Wikimedia Foundation <https://wikimediafoundation.org/>
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re:  Fresh 22.05 released!

[Amir Sarabadani]

Thank you for making and maintaining fresh. It's really useful.

On Tue, May 10, 2022 at 10:18 PM Krinkle  wrote:

> This release promotes Node.js 14 to be the default for the fresh-node
> command.
>
> Get started by installing, updating, or learning more, at:
> https://gerrit.wikimedia.org/g/fresh#fresh-environment
>
> Changelog:
> https://gerrit.wikimedia.org/g/fresh/+/22.05.1/CHANGELOG.md
>
> Node.js 14 is now the default environment for the fresh-node command, this
> follows WMF CI (Thanks James Forrester!). [1] Node.js 12 remains available
> via fresh-node12. Node.js 10 has been removed after being deprecated since
> September of last year.
>
> This release also adds support for detecting and installing to your home
> directory instead of system-wide, which is often preferred on Linux. The
> installer automatically selects one of $HOME/bin or $HOME/.local/bin if it
> exists and is in your "PATH". (Thanks Antoine Musso!)
>
> If you encounter problems with Node.js 14 or experience other issues, let
> us know on Phabricator at https://phabricator.wikimedia.org/tag/fresh/.
> This is also where you can previous tasks.
>
> Fresh is a fast way to create isolated environments from your terminal.
> These can be used to work more responsibly with 'npm' developer tools such
> as ESLint, QUnit, Grunt, Selenium, and more. Example guide:
> https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing
>
> --
> Timo Tijhof
>
> [1] https://phabricator.wikimedia.org/T267890
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Removal of revision_actor_temp table

[Amir Sarabadani]

Hello,

(If you don't query revision_actor_temp table, feel free to ignore this
email)

We will remove revision_actor_temp in two weeks, this table was only built
for temporary use inside mediawiki.

Cloud replicas provide a view that you can query the revision table
directly and instead of joining the revision table with
revision_actor_temp, you can simply get the value of rev_actor field.

We have finally backfilled the value of rev_actor in production and that
can be used directly (thus we will remove the view soon).

If you query this table in your tools, reports, etc. You need to change it
ASAP. Hopefully this should make your queries much simpler. Keep in mind a
similar work will happen on revision_comment_temp table in the future.

You can follow the work in https://phabricator.wikimedia.org/T275246

Best
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Changes in schema of MediaWiki links tables

[Amir Sarabadani]

(If you don’t work with links tables such as templatelinks, pagelinks and
so on, feel free to ignore this message)

TLDR: The schema of links tables (starting with templatelinks) will change
to have numeric id pointing to linktarget table instead of repeating
namespace and title.

Hello,

The current schema and storage of most links tables are: page id (the
source), namespace id of the target link and title of the target. For
example, if a page with id of 1 uses Template:Foo, the row in the database
would be 1, 6, and Foo (Template namespace has id of 6)

Repeating the target’s title is not sustainable, for example more than half
of Wikimedia Commons database is just three links tables. The sheer size of
these tables makes a considerable portion of all queries slower, backups
and dumps taking longer and taking much more space than needed due to
unnecessary duplication. In Wikimedia Commons, on average a title is
duplicated around 100 times for templatelinks and around 20 times for
pagelinks. The numbers for other wikis depend on the usage patterns.

Moving forward, these tables will be normalized, meaning a typical row will
hold mapping of page id to linktarget id instead. Linktarget is a new table
deployed in production and contains immutable records of namespace id and
string. The major differences between page and linktarget tables are: 1-
linktarget values won’t change (unlike page records that change with page
move) 2- linktarget values can point to non-existent pages (=red links).

The first table being done is templatelinks, then pagelinks, imagelinks and
categorylinks will follow. During the migration phase both values will be
accessible but we will turn off writing to the old columns once the values
are backfilled and switched to be read from the new schema. We will
announce any major changes beforehand but this is to let you know these
changes are coming.

While the normalization of all links tables will take several years to
finish, templatelinks will finish in the next few months and is the most
pressing one.

So if you:

   -

   … rely on the schema of these tables in cloud replicas, you will need to
   change your tools.
   -

   … rely on dumps of these tables, you will need to change your scripts.


Currently, templatelinks writes to both data schemes for new rows in most
wikis. This week we will start backfilling the data with the new schema but
it will take months to finish in large wikis.

You can keep track of the general long-term work in
https://phabricator.wikimedia.org/T300222 and the specific work for
templatelinks in https://phabricator.wikimedia.org/T299417. You can also
read more on the reasoning in https://phabricator.wikimedia.org/T24.

Thanks
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: 錄 Diving Into Wikimedia Deployment Data

[Amir Sarabadani]

On Thu, Feb 17, 2022 at 1:51 AM Risker  wrote:

> Thank you very much for sharing this data, Tyler (and to the team that
> researched and analysed it, as well).  I think it shows that the train has
> been pretty successful in mitigating the issues it was intended to
> improve.
>
> I note the data points that show there has been a significant and clear
> trend toward fewer comments per patch.  This would be worth investigating
> further. Iis the total number of reviews pretty consistent, or is it
> increasing or decreasing?  Is it possible that developers have become more
> proficient at writing patches to standard, and thus fewer comments are
> required?  Or could it be that, because more time is invested in writing
> patches (assuming that more patches = more time writing them), there is
> less time for review?
>

There can be other reasons as well, three reasons that I have seen
anecdotally and not in any way scientific are:
 - Reducing average size of patches. This is a good practice in software
engineering and slowly being adapted by developers. Smaller patches leading
to more straightforward review.
 - Rise of simple refactor patches. With lots of refactoring work being
done (specially breaking down monster classes: https://monmon.toolforge.org),
we have a lot of patches fixing deprecations (in core and extensions) and
other cleanup work that are useful but really don't require much discussion.
 - Better test coverage. I remember back then when test coverage was much
lower, most of the review comments were something along the lines of "I
tested it locally and it broke twenty million things". With better test
coverage we have now (specially browser tests, api tests, the rest of e2e
tests), now these are caught by jenkins before a human reviewer finds them
(or worse, show up in production) which is good :)

The reality is probably a combination of all three plus other reasons.

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: ClassCrawler – extremely fast and structured code search engine

[Amir Sarabadani]

I co maintain codesearch with Kunal and I have similar notes. I hope
instead of duplicating the work, we could join forces to improve the
development productivity infrastructure.

Codesearch has been working fine in the past couple of years. There is a
new frontend being built and I hope we can deploy it soon to provide a
better user experience and I personally don't see a value in
re-implementing codesearch. Especially using non-open source software.

In a rather long-term solution, I hope/dream we could implement what Google
has in automating refactoring. It's called LSC [1] (Large Scale Changes)
and we can even piggy back to the library upgrader tool to automate easy
depreciation fixes so developers could focus on complex cases. It's
disheartening to me to see the valuable time of our volunteer developers
being spent on something that could be automated. (For example see the
sheer number of patches made for this deprecation:
https://gerrit.wikimedia.org/r/q/bug:T286694). It doesn't have to be able
to parse php code and do complex magic at first. We can start with simple
regex replacements and then add using rector (a really nice library for
doing refactors in php) and its equivalent in other languages.

[1] For more information see "Software Engineering at Google: Lessons
Learned from Programming Over Time" book:
https://www.goodreads.com/book/show/48816586-software-engineering-at-google

Best

On Sat, Feb 5, 2022 at 2:41 AM Kunal Mehta  wrote:

>
> Hi,
>
> On 2/4/22 08:58, Adam Baso wrote:
> > Thanks for sharing this. I was wondering, might it be possible to help
> > bring this sort of functionality into Code Search (
> > https://codesearch.wmflabs.org  ) ? I
> > noticed the presentation of the search UI looked similar, but I see how
> > the symbol resolution might be something useful for Code Search and
> > upstream Hound.
>
> Indeed, we've been discussing and exploring symbol-based search for a
> while now: . There are some
> pretty neat upstream projects that do this like 
> and zoekt, which is designed for Gerrit integration. I would also note
> that things are likely to change whenever we migrate to GitLab, which
> has its own search functionality built-in
> (). My assumption is that
> GitLab will add symbol-based search eventually to compete with GitHub,
> hopefully that ends up in the CE version someday...
>
> While I very much disagree with the opening proposition that "Working
> with Wikimedia code is time-consuming and risky", I think symbol search
> of the MediaWiki codebase would be incredibly powerful and unlock a new
> level of tooling, just like Codesearch did when it was first introduced,
> so I'm glad to see people looking into it! For example we could do stuff
> like  with it.
>
> There were two main principles in building MediaWiki Codesearch, first
> that everything be licensed as free software[1] (which Bryan covered as
> well) and that we try to use upstream as much as possible. Our
> modifications are injected by a proxy rather than patch the upstream
> code...which has turned out to be incredibly stable over the years.
>
> If you want to collaborate, all the code and setup is in Git and can add
> you to the project, but I see little to no value in building proprietary
> tools or reinventing what other projects have done pretty well rather
> than building on top of them.
>
> [1] https://mako.cc/writing/hill-free_tools.html
>
> -- Legoktm
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>


-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Wikimedia-l] Re: Re: Uplifting the multimedia stack (was: Community Wishlist Survery)

[Amir Sarabadani]

(Speaking in my volunteer capacity)
I doubt there is any malicious intent by WMF. I personally think the
underlying problem is time. Let me explain.

Fixing a big issue in software takes time (I wrote a long essay about it in
this thread) so it makes sense WMF annual planning to focus on issues
before they get to a level that hinders community's work. The problem is
that an issue doesn't get enough attention if it's not severe enough to
affect users so the cycle of frustration continues. For example, I sent an
email in February 2021, at the start of annual planning, to one of the
directors at product outlining all of the issues of multimedia stack.
Because at that point, it wasn't this bad, it didn't make it to FY21-22
plans. Now I feel like a cassandra. We have similar issues in lots of other
places that will lead to frustration. Load balancers (pybal), dumps, beta
cluster, flagged revs, patrolling tools, etc. etc.

On Tue, Jan 11, 2022 at 8:21 AM bawolff  wrote:

> Honestly, I find the "not in the annual plan" thing more damning than the
> actual issue at hand.
>
> The core competency of WMF is supposed to be keeping the site running. WMF
> does a lot of things, some of them very useful, others less so, but at its
> core its mission is to keep the site going. Everything else should be
> secondary to that.
>
> It should be obvious that running a 300 TB+ media store servicing 70
> billion requests a month requires occasional investment and maintenance
>
> And yet, this was not only not in this year's annual plan, it has been
> ignored in the annual plan for many many years. We didn't get to this state
> by just 1 year of neglect.
>
> Which raises the question - If wmf is not in the business of keeping the
> Wikimedia sites going, what is it in the business of?
>
> On Tue, Jan 11, 2022 at 6:01 AM Kunal Mehta  wrote:
>
>> Hi,
>>
>> On 1/1/22 12:10, Asaf Bartov wrote:
>> > It seems to me there are *very few* people who could change status quo,
>> > not much more than a handful: the Foundation's executive leadership (in
>> > its annual planning work, coming up this first quarter of 2022), and
>> the
>> > Board of Trustees.
>>
>> If the goal is to get paid WMF staff to fix the issues, then you're
>> correct. However, I do not believe that as a solution is healthy
>> long-term. The WMF isn't perfect and I don't think it's desirable to
>> have a huge WMF that tries to do everything and has a monopoly on
>> technical prioritization.
>>
>> The technical stack must be co-owned by volunteers and paid staff from
>> different orgs at all levels. It's significantly more straightforward
>> now for trusted volunteers to get NDA/deployment access than it used to
>> be, there are dedicated training sessions, etc.
>>
>> Given that the multimedia stack is neglected and the WMF has given no
>> indication it intends to work on/fix the problem, we should be
>> recruiting people outside the WMF's paid staff who are interested in
>> working on this and give them the necessary access/mentorship to get it
>> done. Given the amount of work on e.g. T40010[1] to develop an
>> alternative SVG renderer, I'm sure those people exist.
>>
>> Take moving Thumbor to Buster[2] for example. That requires
>> forward-porting some Debian packages written Python, and then testing in
>> WMCS that there's no horrible regressions in newer imagemagick, librsvg,
>> etc. I'm always happy to mentor people w/r to Debian packaging (and have
>> done so in the past), and there are a decent amount of people in our
>> community who know Python, and likely others from the Commons community
>> who would be willing to help with testing and dealing with whatever
>> fallout.
>>
>> So I think the status quo can be changed by just about anyone who is
>> motivated to do so, not by trying to convince the WMF to change its
>> prioritization, but just by doing the work. We should be empowering
>> those people rather than continuing to further entrench a WMF technical
>> monopoly.
>>
>> [1] https://phabricator.wikimedia.org/T40010
>> [2] https://phabricator.wikimedia.org/T216815
>>
>> -- Legoktm
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Changes to maximum allowed time for RecentChanges and several other special pages

[Amir Sarabadani]

Hello,
As of today, all requests to several special pages (and their API
counterparts) are subject to a maximum database query execution time of 30
seconds. These special pages are: RecentChanges, Watchlist, Contributions,
and Log. This limit has already been in place for one third of all requests
accessing these page types since December 16th.

This threshold is based on a sampling of half a million requests done to
these special pages by users (excluding crawlers) in three largest wikis
(English Wikipedia, Wikidata and Wikimedia Commons). Out of 500,000
requests only 64 were above thirty seconds and 38 (out of that 64) were
above 60s. This meant that while the database returned the result of the
query in those cases, due to the 60 second limit of the webservice workers,
no results were shown to the user. Our logs show that we have around 1000
requests taking more than thirty seconds (in database query time) per day
so the ratio is extremely small (We get around 10 billion requests per day).

Putting things in context, these queries, while being only 0.01% of the
total number of requests to those special pages, were responsible for 2.9%
of the load. In a few scenarios where such queries appeared at a higher
volume, they were a contributing factor in outages that made one or more
Wikimedia projects unavailable. If left as-is the current configuration is
a potential DDoS vector (and likely was weaponized as one).

Even though ~1000 requests a day is not much, I understand that it is an
inconvenience and it might break useful workflows for some people (for
example patrollers). Query timeout doesn’t mean your request is bad, it
means our database schemas need improvements. We are aware of some of these
needed improvements and we are working on it but migrating terabytes of
data that needs to be replicated to twenty different servers while serving
millions of queries per second is not easy and will take time.

In the meantime, if this is breaking your workflow, please use
https://quarry.wmcloud.org or build a tool in Toolforge (it has access to
database replicas with a much higher timeout) to accommodate. For more
information, see:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database

If you need advice on writing the queries. Feel free to create a ticket
with the “Data-Persistence (Consultation)” tag and we would be happy to
help.

For similar reasons the DynamicPageList extension has had a query limit of
10 seconds (due to its history of causing major issues
<https://phabricator.wikimedia.org/T287380>) since December 16.
Unfortunately, this doesn’t mean it’s safe to enable DPL on more wikis.

For more information, you can take a look at the ticket:
https://phabricator.wikimedia.org/T297708

Thank you
-- 
*Amir Sarabadani (he/him)*
Staff Database Architect
Wikimedia Foundation <https://wikimediafoundation.org/>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Wikimedia-l] Uplifting the multimedia stack (was: Community Wishlist Survery)

[Amir Sarabadani]

On Fri, Dec 31, 2021 at 12:42 AM Strainu  wrote:

> > So where is the best current place to discuss scaling Commons, and all
> that entails?
>
> My impression is that we don't have one. All we hear is "it needs to be
> planned", but there is no transparency on what that planning involves or
> when it actually happens.
>
> > I'd be surprised if the bottleneck were people or budget
>
> The main problem I see is that we end up in this kind of situation.
> Scaling and bug fixing critical features should be part of the annual
> budget. Each line of code deployed to production wikis should have an owner
> and associated maintenance budget each year. Without this, the team will
> not even commit reviews - see the thread on wikitech a few months back
> where a volunteer programmer willing to work on Upload Wizard was basically
> told "We will not review your code. Go fork."
>

There is "code stewardship program" and its goal is to find owners for
components that don't have an owner (or undeploy them). Sometimes it's
successful, sometimes it's not. I have been asking for a maintainer for
FlaggedRevs for four years now, beta cluster is suffering from a similar
situation, etc. etc. It takes time to find an owner for everything, to fill
the gaps in places we don't have a team to handle those (e.g. Multimedia,
you can't just hand over that to team responsible for security for
example). More info at
https://www.mediawiki.org/wiki/Code_stewardship_reviews

>
> > Some examples from recent discussions
>
> Also improvements to the Upload Wizard. There are quite a few open items
> in Phab on this.
>
+1

>
> I really hope you will have better luck than others with bringing this
> issue up in the priority list for next year - multimedia support is growing
> more outdated by the minute.
>
Honestly, the situation is more dire than you think. For example, until a
couple months ago, we didn't have backups for the media files. There was a
live copy in the secondary datacenter but for example if due to a software
issue, we lost some files, they were gone. I would like to thank Jaime
Crespo for pushing for it and implementing the backups.

But I beat my drum again, it's not something you can fix overnight. I'm
sure people are monitoring this mailing list and are aware of the problem.

Best

>
> Strainu
>
> Pe joi, 30 decembrie 2021, Samuel Klein  a scris:
> > Separate thread.  I'm not sure which list is appropriate.
> > ... but not all the way to sentience.
> >
> > The annual community wishlist survey (implemented by a small team,
> possibly in isolation?) may not be the mechanism for prioritizing large
> changes, but the latter also deserves a community-curated priority queue.
> To complement the staff-maintained priorities in phab ~
> > For core challenges (like Commons stability and capacity), I'd be
> surprised if the bottleneck were people or budget.  We do need a shared
> understanding of what issues are most important and most urgent, and how to
> solve them. For instance, a way to turn Amir's recent email about the
> problem (and related phab tickets) into a family of persistent,
> implementable specs and proposals and their articulated obstacles.
> > An issue tracker like phab is good for tracking the progress and
> dependencies of agreed-upon tasks, but weak for discussing what is
> important, what we know about it, how to address it. And weak for
> discussing ecosystem-design issues that are important and need persistent
> updating but don't have a simple checklist of steps.
> > So where is the best current place to discuss scaling Commons, and all
> that entails?  Some examples from recent discussions (most from the wm-l
> thread below):
> > - Uploads: Support for large file uploads / Keeping bulk upload tools
> online
> > - Video: Debugging + rolling out the videojs player
> > - Formats: Adding support for CML and dozens of other common high-demand
> file formats
> > - Thumbs: Updating thumbor and librsvg
> > - Search: WCQS still down, noauth option wanted for tools
> > - General: Finish implementing redesign of the image table
> >
> > SJ
> > On Wed, Dec 29, 2021 at 6:26 AM Amir Sarabadani 
> wrote:
> >>
> >> I'm not debating your note. It is very valid that we lack proper
> support for multimedia stack. I myself wrote a detailed rant on how broken
> it is [1] but three notes:
> >>  - Fixing something like this takes time, you need to assign the budget
> for it (which means it has to be done during the annual planning) and if
> gets approved, you need to start it with the fiscal year (meaning July
> 2022) and then hire (meaning, write JD, do recruitment, interview lots of
> people, get them hire

[Wikitech-l] Re:  Summary of 1.37.0-wmf.17 train deployment

[Amir Sarabadani]

Indeed. James is one of the best things that has happened to the movement.

Kunal Mehta  schrieb am Do., 12. Aug. 2021, 09:04:

> On 8/11/21 5:56 PM, Mukunda Modell wrote:
> > Yes I know James is mentioned twice. There really aren't enough
> > thank-yous for all that James does.
>
> This, 100%. <3
>
> -- Legoktm
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Phasing out old jQuery versions

[Amir Sarabadani]

I can't reproduce it. Also, metrics are not high either. Can you check and
say where are you getting the deprecations from? Preferably in the ticket.
Thanks!

On Sat, Aug 7, 2021 at 3:07 PM Max Semenik  wrote:

> Huh, I see warnings from it on enwiki even when loading pages anonymously.
> Gonna be fun)
>
> On Sat, Aug 7, 2021 at 3:52 PM Amir Sarabadani 
> wrote:
>
>> Hi,
>> (If you don't use jQuery in your code or gadgets, you can ignore this
>> email)
>>
>> in 2017 jQuery 3 got deployed to production and a backward compatibility
>> layer was introduced to prevent old codes from breaking. Since then it has
>> been emitting warnings in your console. All start with "JQMIGRATE". Some of
>> these old jquery breaking changes have been deprecated in versions released
>> in 2006 or 2007.
>>
>> We are slowly pulling the plug and removing that compatibility layer
>> (which would reduce the default payload and make Wikipedia faster for
>> everyone). Yesterday, that layer was removed from mediawiki CI, tests in
>> gated extensions have passed but there is a chance that your extension
>> might be broken in CI now if it's not part of gated extensions. In that
>> case, please fix the errors.
>>
>> We will move forward to deploy this in Wikimedia wikis soon meaning
>> really old gadgets will also break if they don't fix their code. Please
>> check your console and if you see jquery migration deprecation warnings,
>> attend to them because they will break soon. Firefox doesn't give trace to
>> warnings anymore but in Chrome (and using ?debug=1) you can easily find the
>> source of deprecation warnings.
>>
>> We already fixed many usages in code and gadgets and we will make sure to
>> fix heavily used gadgets and user scripts but we can't possibly fix
>> everything and it's the maintainers' responsibility to fix deprecation
>> warnings that is being emitted for years.
>>
>> Another aspect is that if you use jquery.ui, this library has been
>> deprecated as well and I recommend moving away from it but if you still use
>> it and migration can be hard, we patched our jquery.ui fork to make sure it
>> doesn't break. Still, if you find a part of jquery.ui that has not been
>> fixed, let me know.
>>
>> You can track the work in https://phabricator.wikimedia.org/T280944 and
>> the metrics of deprecations can be found in
>> https://grafana.wikimedia.org/d/00037/mw-js-deprecate?orgId=1=1m=now-90d=now=24h
>>
>> Thank you and sorry for the inconvenience.
>> --
>> Amir (he/him)
>>
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Best regards,
> Max Semenik
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Phasing out old jQuery versions

[Amir Sarabadani]

Hi,
(If you don't use jQuery in your code or gadgets, you can ignore this email)

in 2017 jQuery 3 got deployed to production and a backward compatibility
layer was introduced to prevent old codes from breaking. Since then it has
been emitting warnings in your console. All start with "JQMIGRATE". Some of
these old jquery breaking changes have been deprecated in versions released
in 2006 or 2007.

We are slowly pulling the plug and removing that compatibility layer (which
would reduce the default payload and make Wikipedia faster for everyone).
Yesterday, that layer was removed from mediawiki CI, tests in gated
extensions have passed but there is a chance that your extension might be
broken in CI now if it's not part of gated extensions. In that case, please
fix the errors.

We will move forward to deploy this in Wikimedia wikis soon meaning really
old gadgets will also break if they don't fix their code. Please check your
console and if you see jquery migration deprecation warnings, attend to
them because they will break soon. Firefox doesn't give trace to warnings
anymore but in Chrome (and using ?debug=1) you can easily find the source
of deprecation warnings.

We already fixed many usages in code and gadgets and we will make sure to
fix heavily used gadgets and user scripts but we can't possibly fix
everything and it's the maintainers' responsibility to fix deprecation
warnings that is being emitted for years.

Another aspect is that if you use jquery.ui, this library has been
deprecated as well and I recommend moving away from it but if you still use
it and migration can be hard, we patched our jquery.ui fork to make sure it
doesn't break. Still, if you find a part of jquery.ui that has not been
fixed, let me know.

You can track the work in https://phabricator.wikimedia.org/T280944 and the
metrics of deprecations can be found in
https://grafana.wikimedia.org/d/00037/mw-js-deprecate?orgId=1=1m=now-90d=now=24h

Thank you and sorry for the inconvenience.
-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Goto for microoptimisation

[Amir Sarabadani]

It might be interesting to point out to discussions around use of goto in
Linux Kernel:
https://koblents.com/Ches/Links/Month-Mar-2013/20-Using-Goto-in-Linux-Kernel-Code/
https://www.kernel.org/doc/html/v4.10/process/coding-style.html#centralized-exiting-of-functions

IIRC (from a presentation by Greg KH I watched but I can't find it now), in
Linux, goto forward is fine to use everywhere but goto backwards is only
used in the scheduler. And the scheduler is special (to put it mildly). So
I think using goto backwards should be highly discouraged and maybe
downright banned but forward should be okay.

HTH

On Sun, Aug 1, 2021 at 8:05 AM rupert THURNER 
wrote:

> On Sat, Jul 31, 2021 at 6:10 AM Tim Starling 
> wrote:
> >
> > For performance sensitive tight loops, such as parsing and HTML
> construction, to get the best performance it's necessary to think about
> what PHP is doing on an opcode by opcode basis.
> ...
> > I am proposing
> >
> > if ( $x == 1 ) {
> > action1();
> > goto not_2; // avoid unnecessary comparison $x == 2
> > } else {
> > action_not_1();
> > }
> > if ( $x == 2 ) {
> > action2();
> > } else {
> > not_2:
> > action_not_2();
> > }
> ...
> > I am requesting that goto be considered acceptable for
> micro-optimisation.
>
> ha, what question. the single goto and its target are 5 lines apart,
> even me php incompetent person can understand it.
>
> you triggered me reading more about it though. the commit comment
> states it takes 30% less instructions:
>   Measuring instruction count per iteration with perf stat, averaged over
>   10M iterations, PS1. Test case:
>   Html::openElement('a', [ 'class' => [ 'foo', 'bar' ] ] )
>
>   * Baseline: 11160.7265433
>   * in_array(): 10390.3837233
>   * dropDefaults() changes: 9674.1248824
>   * expandAttributes() misc: 9248.1947500
>   * implode/explode and space check: 8318.9800417
>   * Sanitizer inline: 8021.7371794
>
> does this mean these changes bring 30% speed improvement? that is
> incredible! how often is this part called to retrieve one article?
>
> now i understand why legoktm is prepared to rewrite mediawiki in rust
> (https://www.mediawiki.org/wiki/Template:User_Rust), and why proposals
> exist to extend php with rust (https://github.com/rethinkphp/php-rs ,
> https://docs.rs/solder/0.1.6/solder/ ). tempted i was to use legoktm's
> template on my user page, when i finally saw that php is amazing with
> regular expressions by including pcre c library:
>
> https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/regexredux.html
> .
>
> rupert
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>


-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Stream of recent changes diffs

[Amir Sarabadani]

Note on ORES as one of its maintainers:
ORES doesn't use recent changes for getting content and scoring edits. It
hits the API.

HTH

On Thu, Jul 1, 2021 at 3:18 PM Robin Hood  wrote:

> I’m no expert, but I believe the only way to get a diff via the API is
> through https://www.mediawiki.org/wiki/API:Compare. I haven’t worked with
> it to any great degree, though, so I’m afraid I can’t help beyond pointing
> you in that direction.
>
>
>
> *From:* Physikerwelt 
> *Sent:* July 1, 2021 8:17 AM
> *To:* Wikimedia developers 
> *Cc:* andre.greiner-petter ; Aaron
> Halfaker 
> *Subject:* [Wikitech-l] Stream of recent changes diffs
>
>
>
> Dear all,
>
>
>
> we have developed a tool that is (in some cases) capable of checking if
> formulae in -tags in the context of a wikitext fragment are likely
> to be correct or not. We would like to test the tool on the recent changes.
> From
>
>
>
> https://www.mediawiki.org/wiki/API:Recent_changes_stream
>
>
>
> we can get the stream of recent changes. However, I did not find a way to
> get the diff (either in HTML or Wikitext) to figure out how the content was
> changed. The only option I see is to request the revision text manually
> additionally. This would be a few unnecessary requests since most of the
> changes do not change -tags. I assume that others, i.e., ORES
>
>
>
> https://www.mediawiki.org/wiki/ORES,
>
>
>
> compute the diffs anyhow and wonder if there is an easier way to get the
> diffs from the recent changes stream without additional requests.
>
>
>
> All the best
>
> Physikerwelt (Moritz Schubotz)
>
>
>
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Changes to storage of files metadata

[Amir Sarabadani]

Hello,
If you don't do anything with metadata fields of file tables (image table
for example) in replicas, you can ignore this email.

"image" table in Wikimedia Commons is extremely big (more than 380GB
compressed) and has been causing multiple major issues (including an
incident recently). Deep inspections revealed that more than 80% of this
table is metadata of PDF files, around 10% is metadata of DjVu files and
the 10% left is the rest of the information. This clearly needs fixing.

The work has been done on this by Tim Starling and we are slowly rolling
out two major changes:
First, format of metadata in the database (for example img_metadata field
in image table) will change for all files. It used to be php serialization
but it will be changed to json. You can see an example of before and after
in https://phabricator.wikimedia.org/T275268#7178983 Keep it in mind that
for some time this will be a hybrid mode that some files will have it in
json format and some will have it in php serialization. You need to support
both formats for a while if you parse this value.

Second, some parts of metadata for PDF and later DjVu files won't be
accessible in Wikimedia Cloud anymore. Since these data will be moved to
External Storage and ES is not accessible to the outside. It's mostly OCR
text of PDF files. You can still access them using API
(action=query=imageinfo).

Nothing to the outside users will change, the API will return the same
result, the user interface will show the same thing but it would make all
of Wikimedia Commons more reliable and faster to access (by indirect
changes such as improving InnoDB buffer pool efficiency), improves time to
take database backups, enables us to make bigger changes on image table and
improve its schema and much more.

I hope no one heavily relies on the img_metadata field in the cloud
replicas but if you do, please let me know and reach out for help.

You can keep track of the work in https://phabricator.wikimedia.org/T275268

Thank you for understanding and sorry for any inconvenience.
-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Thank you Tuesday

[Amir Sarabadani]

Hi,
Lots of people thanked me for deploying mailman3 but I want to mention that
it would have not been possible without Wikimedia Cloud Services team
giving a lot of resources to me so I could have a test setup and puppetize
mailman3 easily which in turn made deployment to production much easier.

Thank you for providing such a critical infrastructure to us! Keep up the
great work.

Best
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Why does the train start on Tuesday?

[Amir Sarabadani]

Jon, I think you're misunderstanding the point of the "No Deployment on
Friday" policy.

Let's look from far, Why is a work day a no-deploy day? Why are we limiting
ourselves to a four-day week while we can use our full potential? The
reason is that if we deploy something to production on Friday and if it
breaks production, then there is no one around to fix the issue over the
weekend. So far it's rather obvious.

In other words, the requirement for having a time to deploy changes is to
have some buffer until the weekend *to fix issues caused by changes
deployed*. That buffer is Friday. It's made a no deployment day so you
could push urgent fixes (including train blockers). Of course, the fix
should not be too big to cause issues later.

Fridays are not "no deployment day" in the same of sense Sunday is a "no
deployment day". It's the buffer to fix UBN issues, not a long weekend. If
you're fixing an UBN issue, then please go ahead and deploy on Friday. The
ultimate goal of RelEng policies is to have major issues live in production
for the shortest possible time. Refusing to deploy a major fix on a Friday
does the exact opposite of that.

Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen
them finding major issues before they hit all Wikipedia languages many
times, more than I can count.

HTH

On Tue, Jun 22, 2021 at 11:11 PM Scott Bassett 
wrote:

> On Tue, Jun 22, 2021 at 3:03 PM Jon Robson  wrote:
>
>> A few questions to provoke discussion/share knowledge better:
>> * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
>>
>
> I'd note here that the standard security deployment window is Monday
> between 21:00 and 23:00 UTC.  That date and time is not a hard requirement
> by any means, but having such a window exist early in the week, prior to
> the start of the train, has worked out well for a few reasons.  It's both
> convenient and less risky to only  deploy security patches to a single wmf
> production branch, which is the case most Mondays.  It's also less risky
> having the space to monitor patches and roll them back or re-patch during
> the week, as opposed to say, on a Friday, with substantially reduced
> coverage going into most weekends.  Of course there are times when critical
> security issues need to be dealt with on a Friday or even over the weekend,
> but in general, the Security Team likes to avoid this.  Moving the train to
> a Mon, Tue, Wed cadence would imply the security window be moved to the
> previous Friday or possibly Thursday, which is doable, but not desired for
> the aforementioned reasons.
>
> --
> Scott Bassett
> sbass...@wikimedia.org
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Mailman 3 - is there an API available?

[Amir Sarabadani]

There have been requests to build something similar (for example automated
tools for checking only users with certain rights would have access to the
internal mailing list of that right) and the good thing it's now much
easier to do in mailman3. I hope to get around to doing it soon. I make
sure to keep your use case in mind and at least make it slightly easier for
you. Stay tuned.

Best


On Sat, Jun 19, 2021 at 9:33 PM Martin Urbanec 
wrote:

> Hello Kunal,
>
> Thanks for your answer. WMCZ has a private self-hosted wiki that can be
> used as a data source for the maillist.
>
> Unfortunately, who is a member is private information, so I'm not sure how
> it would be possible to sync it with WMF service. An alternative would be
> to somehow (privately) expose the subscribers in a machine-readable form,
> so I could create a bot to indicate subscribeness status in a wiki table,
> but I'm also not sure how feasible that would be.
>
> Martin
>
> so 19. 6. 2021 v 18:20 odesílatel Kunal Mehta  napsal:
>
>> Hi,
>>
>> On 6/19/21 5:32 AM, Martin Urbanec wrote:
>> > does Mailman 3 (as-installed by Wikimedia) have an API I could use?
>>
>> Yes it has an API, but it's not available for public access. Everything
>> in Mailman3 is driven by a REST API, however that API has a single
>> user/password and has access to do *everything*, so we can't hand it out.
>>
>> > I'm an administrator of WMCZ's internal maillist, and it would help me
>> > if I could sync the addresses with evidence of members with a script
>> > rather than having to do it manually each time a new member is admitted
>> > (or quits).
>>
>> So...depending on what you plan on syncing it with, this could be
>> doable. In  it's been
>> requested to auto-subscribe people to the Cloud-announce list, which
>> might require making some part of the REST API accessible in the
>> internal network or having scripts run on the lists server.
>>
>> If you could file a task explaining a bit more detail where you want to
>> sync from, how often it needs to sync, etc. then we could probably
>> figure something out.
>>
>> -- Kunal
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Upstream shutdowns

[Amir Sarabadani]

That's pretty good but I feel something specific to Wikimedia is needed.
Like analytics cluster, wmcs, ... and It should be in wikitech. It'll be so
big that I think it would need subpages, etc. Just thinking out loud.

On Sat, Jun 5, 2021 at 5:22 PM Kunal Mehta  wrote:

> On 6/5/21 10:56 AM, Amir Sarabadani wrote:
> > I think one part of the problem is that we don't have a comprehensive
> > catalog of our dependencies and their dependencies (and so on).
>
> Do you mean something different than
> <https://www.mediawiki.org/wiki/Upstream_projects>? (which is missing a
> lot of items and could use lots of expansion :))
>
> -- Legoktm
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>


-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Upstream shutdowns

[Amir Sarabadani]

I think one part of the problem is that we don't have a comprehensive
catalog of our dependencies and their dependencies (and so on). Having such
list helps us understand hidden patterns in our infra ("we are depending
too much on facebook products") or try to reduce complexities by
harmonizing them (http://boringtechnology.club/)

At least having a structured catalog is a good starting point IMHO.
Best

On Thu, Jun 3, 2021 at 6:53 PM Dan Andreescu 
wrote:

> I suspect there's much more than just this list, even if we just restrict
> it to "major" components.  On Analytics, we've had to adjust as the
> following systems EOL-ed or changed licensing:
>
> * Pivot (moved to Turnilo, a fork)
> * Camus (moving to Gobblin)
> * Kafka Connect
>
> On Wed, Jun 2, 2021 at 5:16 AM Antoine Musso  wrote:
>
>> Le 02/06/2021 à 03:35, K. Peachey a écrit :
>>
>> There is also one of the search systems migrating to non O/S from
>> memory... elastic?
>>
>> Hello,
>>
>> Yes that is ElasticSearch. They have recently announced they are now
>> licensing code under Server Side Public License (SSPL)
>>  it is a viral
>> license which in short requires that any service based on such code get
>> licensed with the same license.  I am not a lawyer, but I guess that would
>> imply that our whole stack switch to it as well.  As such it is not
>> considered free by Debian or the Open Source Initiative (OSI).
>>
>> As I understand it, the move has been made possible since all
>> contributions were subject to a Contributor License Agreement (CLA)
>> . So that
>> although the code was placed under a free license, the CLA effectively
>> grants unrestricted publishing right to an organization, they can thus
>> relicense the code however they want.   If I get it right, the old code is
>> still under a free license but it can also be used under the new non free
>> license.As I get it the move has been done due to frictions with Amazon
>> which is providing a search service.   Amazon announced they would be
>> behind the free community fork: https://www.opensearch.org/
>>
>>
>> Antoine "hashar" Musso
>>
>>
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Mailman2 is now shut down (T52864)

[Amir Sarabadani]

That's it. That's the email.
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: New Code of Conduct committee candidates

[Amir Sarabadani]

Hello,
Since no objections have been raised by the community in the given period,
the committee is now in effect.

Welcome Ariel, Effie and Kunal, and thank you Matanya, Tonina and Tpt for
your service.

Best

On Wed, Apr 14, 2021 at 2:24 AM Amir Sarabadani  wrote:

> Hello,
> The committee has finished selecting new members and the new committee
> candidates are (In alphabetical order):
>
>- Amir Sarabadani
>- Kunal Mehta
>- Martin Urbanec
>- MusikAnimal
>- Tony Thomas
>
> And auxiliary members will be (Also in alphabetical order):
>
>- Ariel Glenn
>- Effie Mouzeli
>- Huji
>- Jayprakash12345
>- Nuria Ruiz
>
> You can read more about the members in the mediawiki.org page
> <https://www.mediawiki.org/wiki/Code_of_Conduct/Committee/Members/Candidates>
> .
>
> The changes compared to last term are:
>
>- Kunal Mehta is joining the main CoC committee
>- Ariel Glenn is coming back to auxiliary committee
>- Effie Mouzeli is joining auxiliary committee for the first time
>- Tonina Zhelyazkova is leaving the committee
>- Matanya and Tpt are leaving the auxiliary committee
>- I won't run for the chair this year.
>
>
> This is not the final structure. According to the CoC
> <https://www.mediawiki.org/wiki/Code_of_Conduct/Committee#Selection_of_new_members>,
> the current committee publishes the new members and calls for public
> feedback for *six weeks* and after that, the current committee might
> apply changes to the structure based on public feedback.
>
> Please let the committee know if you have any concern regarding the
> members and its structure until *26 May 2021* and after that, the new
> committee will be in effect and will serve for a year.
>
> Amir, On behalf of the Code of Conduct committee
> Best
>
>

-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Abstract Schema: An Update

[Amir Sarabadani]

Hello,
If you missed the previous updates, there's the first
and
the second
one.
This is the last one I'm sending but in a good way.

We now finished migrating all 57 core tables to abstract schema. Now tables.sql
is empty for MySQL
.
We will soon remove the tables.sql files and links to them.

We have also cleaned more than hundreds of old schema change files.
 That enabled us to actually
look for unused sql files and drop tens of unused ones that have not been
used since 2002 ,
2004 , or 2005
 (and much more)
and they got lost in the sheer number of our sql patch files.

The next update is exciting for me. With abstraction in place, we can now
have a proper tracking of drifts between schema in paper and our production
(This is a follow up from a major incident in 2018). Now we have
https://drift-tracker.toolforge.org/ that keeps track of these drifts. Our
schema has been around for more than twenty years and we have hundreds of
database hosts, making sure everything is using the right database schema
(and stays correct) is impossible manually and we have been finding and
fixing these drifts since 2018, see this comment onwards
.). Of course more work
in improving the tracker is welcome (here's the list
)

If you want to enjoy the benefits of abstract schema [1] in extension(s)
you or your team maintains, Please abstract the schema of your extension.
There's a long list of WMF deployed extensions that are not using abstract
schema  and some even already
have a patch that only needs reviewing. Once that's done, we can add that
to drift tracking and have a more comprehensive list of potential issues.
If you need help with the abstraction work, just ping me.

This also helped us resolve several long-standing tickets like T104459
 (5.5 years old), T62962
 (7 years old), and T42626
 (soon reaching its ninth
birthday) and will help us to address even more tech debt in future.

There is more to be done, improving the abstract schema *change* system,
finding a home of schema documentation, improving the drift tracker and
making it more automated, so much more. But the biggest chunk of work is
now finally done.

I really would like to thank Ammarpad for great work on abstracting the
tables and handling all sorts of edge cases, James Forrester and Tgr for
their reviews which without them this wouldn't be possible and Sam Reed who
wrote a script to speed up migration
.
This was a team work to its core.

[1] In more details, by abstracting you will have automated checks for
dirfits of the schema of extension(s) you maintain and production. You will
have Postgres support for free. Also, you can have automated documentation
generation, ability to test the schema itself, and have better consistency
of your data types (like one datatype for timestamps).

Until the next adventure.
-- 
Amir (he/him)
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Re: [Wikitech-l] [pywikibot] Thank you Xqt

[Amir Sarabadani]

Seconding that. Thank you for all your hard work!

On Sat, May 8, 2021 at 12:29 AM Kunal Mehta  wrote:

> I was looking at Pywikibot's commit stats for something and noticed that
> as of earlier today, `git shortlog -s` reports that Xqt hit 4,000
> commits in the pywikibot/core repository!
>
> For comparison, the next human is at 822 commits. In total the
> repository has a little over 9,800 commits, of which ~40% are from Xqt.
>
> Thank you Xqt for all your work maintaining Pywikibot for so many years
> and for helping power so many bots across Wikimedia.
>
> -- Legoktm
>
> ___
> pywikibot mailing list
> pywiki...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Mailman3 is now deployed to production, mailing lists are upgrading

[Amir Sarabadani]

Hello,

TLDR: Mailman3 is now available for general use, all mailing lists will be
migrated in the next couple of weeks, providing everyone with a much better
mailing list experience. You will notice some changes, let us know if you
run into issues.

Long version:
We're happy to announce that Mailman3 is available for general use and some
have already been migrated. You can find the current mailing lists that
have been migrated at [1] and their archives in "hyperkitty"[2].

Mailman3 is a full rewrite of our previous mailing list software
(Mailman2), and the migration is long overdue. Some key new features that
we want to highlight:
* One user account (no more monthly password reminder emails or list
passwords)
* Ability to search through archives
* Posting through a web interface
* A web interface that doesn't look like its from the early 2000s
* Better security of accounts and messages

The first mailing list migrated was LGBT@ and you can see its mailing list
page in [3]. We are going to slowly migrate the rest of mailing lists (all
+700 of them), you can track the work in [4]. All new mailing lists from
now on will be only on Mailman3.

This means:
* We will send an email to admins of any mailing list right before
starting the upgrade process, and once it's finished.
* The link to subscribe to lists will change, please update your wiki
pages, documentation, etc. We will provide redirects though.
* Links to old archives for public mailing lists won't break. It will
stay at it is now and will become redirects shortly. But URLs of archived
emails of private mailing lists will break. This is necessary for improving
security of mailing lists. Keep it in mind that in the new system you can
easily search in the archives.

Given that in Mailman3, you can simply have one central account for all of
your mailing lists. We highly encourage you to make one [1], that way, you
can easily control what mailing lists you are subscribed to, easily join
new mailing lists and much more. This also makes administrating and
moderating mailing lists much easier.

If you have any questions or you encounter any issues, let us know! You can
create a Phabricator ticket ("Wikimedia-Mailing-lists" project) or ping us
on IRC in #wikimedia-tech.

We hope that Mailman3 brings much needed love to our mailing lists without
breaking your workflows (like reading mails by piping telnet into less or
something like that).

The umbrella ticket for the work: https://phabricator.wikimedia.org/T52864

[1] https://lists.wikimedia.org/postorius/lists/
[2] https://lists.wikimedia.org/hyperkitty/
[3] https://lists.wikimedia.org/postorius/lists/lgbt.lists.wikimedia.org/
[4] https://phabricator.wikimedia.org/T280322

Best,
Kunal (Legoktm) and Amir (Ladsgroup)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikitech logo

[Amir Sarabadani]

Hi,
Reminder: You have only four days to vote on the new logo.

Best

On Fri, Apr 16, 2021 at 6:28 PM Amir Sarabadani  wrote:

> Hello,
> I'm once again asking you to vote on a logo. This time is the new Wikitech
> logo since its original one was derived from the old MediaWiki logo that
> has now changed. You can see this ticket
> <https://phabricator.wikimedia.org/T279087> for previous discussion and
> proposals of the new wikitech logo.
>
> This time, it's much simpler. Only one round (as they are just variants of
> the same logo), For two weeks only and effective immediately afterwards.
>
> You can vote in https://wikitech.wikimedia.org/wiki/Wikitech_logo and
> voting is open until 30 April 2021. You can only support up to two logos
> (out of five proposed).I recommend reading the ticket before voting.
>
> Thanks
> --
> Amir (he/him)
>
>

-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Herald rules

[Amir Sarabadani]

Hi,
I'm really sorry for sending email to such a large venue but I couldn't
find a better mailing list. Feel free to ignore this email if you don't do
anything with Herald rules.

Herald rules, a set of rules in phabricator to automate the work, are
expensive and slowly making saving any change on phabricator slower and
slower. You can take a look at this ticket.


As the result, we have been migrating these rules to maintenance bot
. Which means changes
won't be immediately applied anymore (and it'll take up to an hour and they
will be post-change).

If you see your Herald rule has been disabled, please don't enable it. If
you want to change it, you can make a PR in the maintenance bot source code.
 Please avoid
introducing new Herald rules if there's a similar functionality supported
by the bot. Just add it to the work list of the bot. An exception would be
on time-sensitive tickets. Like handling UBN ones. They will stay as Herald
rules.

Any sort of change to improve documentation, code health, adding new
functionality so we can migrate more Herald rules, or migrating existing
ones would be greatly appreciated. If there are bugs, feel free to create a
ticket for it
.

You can also create email filters to ignore emails triggered by maintenance
bot (which its activity will increase).

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Wikitech logo

[Amir Sarabadani]

Hello,
I'm once again asking you to vote on a logo. This time is the new Wikitech
logo since its original one was derived from the old MediaWiki logo that
has now changed. You can see this ticket
 for previous discussion and
proposals of the new wikitech logo.

This time, it's much simpler. Only one round (as they are just variants of
the same logo), For two weeks only and effective immediately afterwards.

You can vote in https://wikitech.wikimedia.org/wiki/Wikitech_logo and
voting is open until 30 April 2021. You can only support up to two logos
(out of five proposed).I recommend reading the ticket before voting.

Thanks
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] New Code of Conduct committee candidates

[Amir Sarabadani]

Hello,
The committee has finished selecting new members and the new committee
candidates are (In alphabetical order):

   - Amir Sarabadani
   - Kunal Mehta
   - Martin Urbanec
   - MusikAnimal
   - Tony Thomas

And auxiliary members will be (Also in alphabetical order):

   - Ariel Glenn
   - Effie Mouzeli
   - Huji
   - Jayprakash12345
   - Nuria Ruiz

You can read more about the members in the mediawiki.org page
<https://www.mediawiki.org/wiki/Code_of_Conduct/Committee/Members/Candidates>
.

The changes compared to last term are:

   - Kunal Mehta is joining the main CoC committee
   - Ariel Glenn is coming back to auxiliary committee
   - Effie Mouzeli is joining auxiliary committee for the first time
   - Tonina Zhelyazkova is leaving the committee
   - Matanya and Tpt are leaving the auxiliary committee
   - I won't run for the chair this year.


This is not the final structure. According to the CoC
<https://www.mediawiki.org/wiki/Code_of_Conduct/Committee#Selection_of_new_members>,
the current committee publishes the new members and calls for public
feedback for *six weeks* and after that, the current committee might apply
changes to the structure based on public feedback.

Please let the committee know if you have any concern regarding the members
and its structure until *26 May 2021* and after that, the new committee
will be in effect and will serve for a year.

Amir, On behalf of the Code of Conduct committee
Best
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Logo of MediaWiki has changed

[Amir Sarabadani]

Hello,
After more than one year of design, discussion, vote, iteration, and months
of legal work, I’m happy to announce that the logo of MediaWiki has been
officially changed. This applies to both the software and logo of
https://mediawiki.org.

The old logo of MediaWiki was adopted slightly more than fifteen years ago.
This logo was featuring the nice concept of a sunflower representing
diversity, constant growth and also wilderness.

However, with years, the logo became outdated and we realized that it had
several problems, including but not limited to:

   - It was a bitmap picture so it’s unusable in large sizes
   - Its high details (“too realistic”) made it unusable in small sizes
   - Its fixed and realistic style made it hard to have variations or
   adaptations

Most, virtually all, software products use a simpler and more abstract form
following basic logo design guidelines and best-practices to avoid above
(and more) issues. For example, docker, kubernetes, Ubuntu, Vue.js, React,
Apache Kafka and many more.

You can find the discussion of changing the logo in
https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_MediaWiki,_2020
. As you can see on this page, a lot of interesting practical and
theoretical exchanges happened, leading to the final vote and decision.

The new logo represents a collection of projects built on our engine: each
petal is one of the many wikis that we support, and the lack of an explicit
core shows that we are part of these projects, as well as and they are part
of MediaWiki. The new logo also reflects the fact that evolution never
stops, and like the petals of a flower, the development of each project,
the growth of each community built on our engine allows everyone else to
grow.

The designer of the new logo is [[User:Serhio Magpie]]. With the nice
abstraction baked-in, you can use it in large or small sizes or you can
adapt it for different usecases (there’s one already for mediawiki on
docker: https://phabricator.wikimedia.org/T274678). There is a logo
guideline for MediaWiki now:
https://www.mediawiki.org/wiki/Manual:MediaWiki_logo_guidelines

We already deployed changes to mediawiki.org and landed related patches on
master, meaning from 1.36 release onwards, it’ll come with the new logo.
You can follow the work of rolling it out in
https://phabricator.wikimedia.org/T268230.

I humbly ask Wikimedians to update their wikis, for example usages on the
main pages, Wikipedia articles, templates, and more. You can use the logos
in this category on Commons. The files are already protected against upload
vandalism. https://commons.wikimedia.org/wiki/Category:MediaWiki_logo_(2020)

A big thank you to all who helped this project to finish. From designers,
community members, people who voted and discussed it intensively for
months, Wikimedia Legal for doing all the necessary work for transferring
the rights, clearing it and filing it for trademark. And many many more
people.

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikimedia-l] Test Mailman3 in production

[Amir Sarabadani]

I think the subject line is the only thing that matters but I also want to
emphasize that unlike mailman2, you can just have a central account for all
of your mailing lists. Confirm your email when registering a new user
(which you can just click on a link) and everything will be pretty
straightforward afterwards.

HTH


On Fri, Mar 26, 2021 at 2:11 PM Deb Tankersley 
wrote:

> Hi,
>
> I've started to play with the interface, but questioning a bit the
> instructions given (for clarity purposes) when you're signing up for a list:
>
> Before you can start using GNU Mailman at this site, you must first confirm
>> that this is your email address.  You can do this by replying to this
>> message,
>> keeping the Subject header intact.
>>
>
> Should there be additional instructions on if something is needed in the
> body of that reply email, or that it can be empty and just keep the subject
> line the same?
>
> Cheers,
>
> Deb
>
> --
>
> deb tankersley (she/her)
>
> senior program manager, engineering
>
> Wikimedia Foundation
>
>
>
> On Fri, Mar 26, 2021 at 3:40 AM Pankaj Jangid 
> wrote:
>
>> Amir Sarabadani  writes:
>>
>> > tl;dr: https://lists-next.wikimedia.org is running mailman3. Please
>> help us
>> > test the software before we upgrade the real mailing list server.
>>
>> Indeed this is great news. I have subscribed to a couple of mailing
>> lists that are already using v3. And it really makes it easy to
>> manage. Thanks for working on this.
>>
>> One question. I subscribe to this list via the NNTP bridge hosted at
>> gmane.io. Will the change make any difference at that end? I mean do the
>> administrators of Gmane require any change?
>>
>> --
>> Regards,
>> Pankaj Jangid
>>
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
> ___
> Wikimedia-l mailing list, guidelines at:
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and
> https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: wikimedi...@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> <mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe>
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Test Mailman3 in production

[Amir Sarabadani]

Hello,

tl;dr: https://lists-next.wikimedia.org is running mailman3. Please help us
test the software before we upgrade the real mailing list server.

Kunal and I have been working on deploying the new mailman (version 3) to
replace mailman2 serving https://lists.wikimedia.org and powering all of
our mailing lists.

Mailman2 is a dinosaur that should have gone extinct years ago. Pretty old
user interface (especially for admins and moderators), storing passwords in
plain text, lack of any database (everything is file on disk), pretty old
code, lack of ability to search in archives or send email from web
interface, running on EOL python (python2), encoding issues with non-Latin
languages, hard to redact archives, and the list goes on and on.

The new version has been developed/puppetized/tested in the Cloud and is
now ready for proper testing! Give it a try:
https://lists-next.wikimedia.org. We have created some mailing lists you
can join and can test. If you want to test the experience as a list
administrator/moderator, we can give those permissions out as well.

WARNING: All data on the lists-next server will be deleted after the test
period is over.

We will also need help updating documentation on wikis and elsewhere.

If you find any bugs/issues (yay!), please file a ticket in the
“Wikimedia-Mailing-lists” Phabricator project and we’ll check it out.

In the coming days/weeks will also import some public mailing lists from
the old version to the new version to check archive size, search index
size, and other aspects. There are other TODOs left as well like
monitoring, logging, anti-abuse, etc.

Slowly and after testing (hopefully soon), we expect to deploy this on
lists.wikimedia.org and mailing lists one by one or in batches can be
upgraded to the 21st century.

The overall task tracking this project is T52864
 and a big thank you people who
are helping this move forward.

Regards,

Kunal and Amir
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] FlaggedRevs (Pending changes) extension and its future

[Amir Sarabadani]

Hello,
This extension is one of the oldest extensions we still have in production
and currently does not have a maintainer. FlaggedRevs has been the cause of
several incidents and visible regressions specially since software decays
and our environment constantly changes.

Another problem with this extension is its scope, while most of its
functionalities are not enabled in WMF production or enabled in a very
small set of wikis (e.g. multiple dimensions was enabled only in Hebrew
Wikisource but they agreed to disable it). This has made maintaining the
extension a tall order (more of a nightmare). In other words, this
extension does too many things and does none well.

To move forward, we are removing barely used functionalities of this
extension. Including support for multiple dimensions, multiple tiers,
several one of its special pages and more to make it less of a burden to
maintain and improve. It would simplify its logic drastically and will
enable us to rework its old interface, fix several deprecated code that
this extension is the last to block their removal (like action=ajax),
reduce the number of issues/incidents/regressions caused by this extension
and much more.

Users in most wikis that have this extension enabled (including English
Wikipedia and German Wikipedia) won't see a difference but for example,
users won't be able to mark a revision as "good quality" or "average
quality" anymore in English Wikibooks or Finnish Wikipedia. The
"checked/stable" status will stay though.

This is also to notify third party installations of this extension that
this will happen from release 1.36 so if you need to have some of its
features we are removing, please fork.

To see the discussion around this, take a look at
https://phabricator.wikimedia.org/T185664

Thank you for understanding and sorry for any inconvenience.
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] about jawp mailing list

[Amir Sarabadani]

Hi,
It's https://phabricator.wikimedia.org/T271123
Feel free to subscribe. At the current state, if you fix that, something
else breaks. The proper solution would be to migrate to mailman3 (which is
slowly making progress: https://phabricator.wikimedia.org/T52864)

Best

On Wed, Mar 10, 2021 at 10:48 AM  wrote:

> Good evening. I am Rightandlight and I am working on the mailing list for
> global unlocking and unblocking application including conversation page
> lock on jawp.
> Now, on the jawp mailing list, the characters are garbled and I can barely
> see it in the email, but I can't see it in the archive and it's not seen by
> many people. Is it possible to eliminate the garbled characters in the jawp
> mailing list archive?
> https://lists.wikimedia.org/pipermail/wikija-l/
> Rightandlight
> * I am always looking for someone to unlock or help me with global
> unlocking.
>
> https://ja.wikipedia.org/wiki/%E5%88%A9%E7%94%A8%E8%80%85:%E3%82%A2%E3%83%AB%E3%83%88%E3%82%AF%E3%83%BC%E3%83%AB/%E3%82%B0%E3%83%AD%E3%83%BC%E3%83%90%E3%83%AB%E3%83%AD%E3%83%83%E3%82%AF%E3%81%A8%E3%82%B0%E3%83%AD%E3%83%BC%E3%83%90%E3%83%AB%E3%83%96%E3%83%AD%E3%83%83%E3%82%AF
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Issue with jitsi - Anyone knows why?

[Amir Sarabadani]

Hello,
I usually wouldn't bother people with my issues but I'm sorta desperate
here. This is about https://meet.wmcloud.org. WM Cloud jitsi instance. It's
on a bigram VM and on docker
.

Users report that when using this "a session with three people today and it
was rather poor. Bad grainy video from my side even though I have 100 Mbps
both ways. After about 15 minutes the session froze and the other two
dropped out." or "we were 4 persons and it was very unstable (no screen
sharing). people's connection got lost, so they could often still hear but
could not participate in the call anymore." and you can reproduce the issue
too if you stay long enough in a meeting with another connection (your
phone for example).

I can't find any reason why this is happening. I wrote some of my
investigations here  but we
checked the cloud's infra. CPU, memory, etc. all look fine as well as the
network throughput. It doesn't happen with a new VM but quickly (after one
meeting) builds up to have the same issues again. I added a regular restart
of the docker containers (it even destroys them and recreates them again,
also the docker service itself gets restarted) but nothing changes (maybe I
should add a restart of the network manager too?). I assume the iptables
being busy because of docker can contribute to the issue but not this much.

I'm running out of ideas. If anyone has worked with such setup and feels
comfortable debugging this, let me know and I give permission to check the
VM.

Thank you
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Code of conduct committee call for new members

[Amir Sarabadani]

Hello all,

It's coming close to time for annual appointments of community members to
serve on the Code of Conduct (CoC) committee. The Code of Conduct Committee
is a team of five trusted individuals plus five auxiliary members with
diverse affiliations responsible for general enforcement of the Code of
conduct for Wikimedia technical spaces. Committee members are in charge of
processing complaints, discussing with the parties affected, agreeing on
resolutions, and following up on their enforcement. For more on their
duties and roles, see
https://www.mediawiki.org/wiki/Code_of_Conduct/Committee

This is a call for community members interested in volunteering for
appointment to this committee. Volunteers serving in this role should be
experienced Wikimedians or have had experience serving in a similar
position before.

The current committee is doing the selection and will research and discuss
candidates. Six weeks before the beginning of the next Committee term,
meaning 9 April 2021, they will publish their candidate slate (a list of
candidates) on-wiki. The community can provide feedback on these
candidates, via private email to the group choosing the next Committee. The
feedback period will be two weeks. The current Committee will then either
finalize the slate, or update the candidate slate in response to concerns
raised. If the candidate slate changes, there will be another two week
feedback period covering the newly proposed members. After the selections
are finalized, there will be a training period, after which the new
Committee is appointed. The current Committee continues to serve until the
feedback, selection, and training process is complete.

If you are interested in serving on this committee or like to nominate a
candidate, please write an email to techconductcandidates AT wikimedia.org
with details of your experience on the projects, your thoughts on the code
of conduct and the committee and what you hope to bring to the role and
whether you have a preference in being auxiliary or constant member of the
committee. The committee consists of five members plus five auxiliary
members and they will serve for a year; all applications are appreciated
and will be carefully considered. The deadline for applications is the end
of day on 26 March 2021.

Please feel free to pass this invitation along to any users who you think
may be qualified and interested.

Best,
Amir on behalf of the CoC committee
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Update on abstract schema and schema changes

[Amir Sarabadani]

Hello,
There has been a lot of progress in abstract schema and abstract schema
changes initiative since last time

I gave an update on it. So here's another one.

*Abstract Schema*
So far, more than 90% (51 out 56) of tables of mediawiki core are now
migrated to abstract schema.

This means much smaller schema drifts between MySQL and Postgres. We have
done more than 250 schema changes in Postgres to fix these drifts.
Including 56 index rename, 66 data type change, setting default to 43
fields and changing nullability of 29 fields. To compare, that's more
schema changes done on Postgres from 2014 until 2020. Once we have migrated
all tables, we can close this four-year old ticket
.

Similar improvement has happened on standardizing timestamp fields in MySQL
, once all tables are migrated,
we can call this eight-year old ticket done too.

One nice thing about having an abstract schema is that you can generate
documentation automatically, This page is completely made

automatically from tables.json. We can make it generated in
doc.wikimedia.org on every merge. And also we can make the database layout
diagram

created automatically.

Another nice thing. When you have an abstract schema, you can easily write
tests and enforce database conventions. For example, you can write a test
to make sure all tables have exactly five columns (because five is your
lucky number). We haven't written such a test but now there's a test that
enforces a uniform prefix for columns and indexes of tables in core
. We are currently fixing its
violations to standardize our schema even more.

I'm planning to make reporting on drifts between the abstract schema and
our production completely automated and make it accessible to DBAs for
further investigations which is now much easier thanks to abstract schema.
You can follow the progress of that work in this ticket.


*Abstract Schema Changes*
Now we have a new maintenance script, it produces schema change sql files
(aka ALTER TABLE files) based on snapshot of abstract schema of before and
after of a table. Here's an example of an index rename.
 It would make
creating schema change patches much easier (a little bit of work but you
don't need to know internals of Postgres anymore, it's also less prone to
mistakes)

With approval of RFC to drop support of upgrading from versions older than
two LTS releases, we can now drop hundreds and hundreds of sql files. It
would give us room to breath and audit our sql files to find orphan ones
and improve abstract schema change work. That is currently blocked on this
patch landing. 

We will work on reshaping the schema changes in general since its current
checks system is less than optimal, its tests are not very updated and so
much more to do.

*What can we do?*
Glad you asked :D The biggest request I have from people is to migrate
their extensions to abstract schema. There's a list of WMF-deployed
extensions that their schema has not migrated yet
. This is doubly important as we
want to build a reporting system for drifts in production and it's not
possible to report these drifts for extensions that their schema has not
migrated yet. So if you or your team maintain an extension from that list,
prioritize migrating that please. Reedy wrote a great script

that takes a sql file and produces its equivalent abstract schema and it
gives you a good starting point (PR is welcome!). Feel free to add me as a
reviewer to patches of migrating extensions to abstract schema.

Another thing is that if you use postgres for mediawiki, you help testing
our postgres schema by trying master (make sure to take a backup first) and
see if everything is alright.

*Thank you!*
I would really like to thank Ammarpad for migrating lots of tables of core
to abstract schema and handling all sorts of edge cases and doing most of
the work of using uniform prefix tests and fixes. Thanks to James Forrester
for reviewing lots of patches. Thanks Reedy for the script and also
abstracting lots of tables in extensions and also Tgr for helping in
reviews and getting the project going. Also a big thank you to DBAs for doing
a lot more schema changes in production
. You rock!
An apology is also warranted for breaking update.php on master twice
(caused by yours truly).

Until next update!
-- 
Amir 

Re: [Wikitech-l] "Fixing redirects" to "improve performance"

[Amir Sarabadani]

Hey,
In general, the performance implications of an edit for fixing redirects
heavily outweighs the small improvement of removing a redirect.

The biggest exception to that is main pages of wikis (which redirecting
en.wikipedia.org to en.wikipedia.org/wiki/Main_page actually wastes 27
years of Wikipedia readers time every month) [1] so if there is lots of
clicks coming for an article from another article (you can find those in
clickstream datasets [2]) this might be a good idea to fix but in general I
don't think so and it's even more work for the system to re-render the
page, update links table, etc.

[1] https://phabricator.wikimedia.org/T120085
[2] https://dumps.wikimedia.org/other/clickstream/

On Mon, Dec 28, 2020 at 3:06 AM Thibaut Payet 
wrote:

> Thank you very much for your detailed reply.
>
> We tried to convince our friend with your message and WP:PERF but he's
> still not convinced, he even created a RfC on frwiki which is currently
> being discussed[1] and we need some more information.
>
> He created these two pages:
> *  (list of
> redirections)
> *  (list of direct
> links)
>
> Previewed them and argue that the parser profiling data table shown
> below the page indicate that there's less CPU time/real time usage with
> the page using direct links than the one using redirects:
> "As surprising as it may seem, the page that gives the best result is
> the one with the direct links. Exactly the opposite of those who want to
> prohibit the fixes [note: we don't], it is precisely the direct links
> that give the best results. Go figure."[2]
>
> Does that kind of benchmark makes sense? Is the parser profiler data
> even relevant here? What exactly is calculated?
>
> Sorry for all these questions but we would be grateful if you could shed
> some light on this.
>
> Happy holidays,
> Thibaut Payet
> [[User:Thibaut120094]]
>
> [1] 
> [2] 
>
> On 24.11.2020 05:36, AntiCompositeNumber wrote:
> > While the specifics may have changed over time, the general point of
> > the page is correct -- "fixing" a redirect will use significantly more
> > server time than clicking on one. When you click a redirect, MediaWiki
> > queries the redirect table to see where you should go instead. Reads
> > from the database like this are fast. However, when you edit the page
> > to "fix" a redirect, MediaWiki has to save the edit, update the links
> > tables, re-render the HTML, invalidate the cache, and serve the new
> > HTML, among other things.
> >
> >  From a reader's perspective, there's no performance difference between
> > clicking a redirect and clicking any other internal link. We're
> > talking milliseconds here, if that. If you wanted to really improve
> > the performance of a page, remove all the images. (Note: do not do
> > this.)
> >
> > This falls under
> >  >:
> > let the developers and the operations folk worry about performance. If
> > you're already making an edit to an article and you feel like
> > bypassing a few redirects, go ahead and knock yourself out. But going
> > out of your way to "fix" links to redirects is just a waste of your
> > time.
> >
> > ACN
> >
> > On Mon, Nov 23, 2020 at 11:23 PM Andreas Papacharalampous 
> wrote:
> >>
> >> the page was created in 2006 [1], but i don't think it's up to date for
> 2020. the last couple of "constructive" edits were from 2006-2011 but i'm
> not sure.
> >>
> >> [1]:
> https://xtools.wmflabs.org/articleinfo/en.wikipedia.org/Wikipedia:Tools/Navigation%20popups/About%20fixing%20redirects
> >> --andreas
> >>
> >>
> >>
> >> On Mon, Nov 23, 2020 at 11:11 PM Thibaut Payet <
> thibaut120...@outlook.fr> wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> On the French Wikipedia we're currently reworking our help page about
> >>> redirects and some of us would like to include a section about common
> >>> misconceptions, especially those described at
> >>> <
> https://en.wikipedia.org/w/index.php?title=Wikipedia:Tools/Navigation_popups/About_fixing_redirects=988508313
> >.
> >>>
> >>> However one user who's used to "fix" redirects is strongly opposed
> >>> because "this page is from 2006" and it's "unsourced".[1]
> >>>
> >>> So I would like to ask the sysadmins from Wikimedia and the MediaWiki
> >>> developers who are following this mailing list: is this page created in
> >>> 2006 still true and relevant in 2020?
> >>>
> >>> Thank you.
> >>>
> >>> --
> >>> Kind regards,
> >>> Thibaut Payet
> >>> [[User:Thibaut120094]]
> >>>
> >>> [1]
> >>> <
> https://sigma.toolforge.org/usersearch.py?name=Alaspada=Discussion_aide%3ARedirection=frwiki=
> >
> >>>
> >>> ___
> >>> Wikitech-l mailing list
> >>> Wikitech-l@lists.wikimedia.org
> >>> 

[Wikitech-l] MediaWiki logo

[Amir Sarabadani]

Hello people!
So the voting period is done now and the second proposal is the clear
winner.
https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_MediaWiki,_2020/Round_2

163 people participated and these are the top winners using Schulze method:
* 2: Gradient (translucent - CamelCase)
* 8: Yellow (solid - CamelCase)
* 6: Yellow (translucent - CamelCase)

Comparing 2 and 8, 92 people preferred 2 over 8 while 71 people preferred
the other way around.

Now we are in the legal clearance period (which hopefully will take a
month) and once it's over, we will start implementing the change, if 2
doesn't get cleared, 8 will be implemented and so on.

Thank you for everyone who participated (except myself, thanking myself
would be weird)
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The second round of voting for mediawiki logo - one week left

[Amir Sarabadani]

Hello,
This is a gentle reminder that there's one week left from the voting period
of choosing the next logo of mediawiki.

If you haven't voted yet, you can vote in
https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_MediaWiki,_2020/Round_2

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Thank you Tuesday

[Amir Sarabadani]

It has been a while since we had Thank you Tuesdays but this one needs a
shout out.

Thank you Manuel for epic work of cleaning up more than one thousand schema
drifts in s3 (small wikis) from missing indexes to uncleaned fields and
much more:
https://phabricator.wikimedia.org/T260111

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Update on abstract schema and schema changes

[Amir Sarabadani]

Hello,
It has been a while since I gave an update on the state of abstracting
schema and schema changes in mediawiki
. So here's a really long one.

So far around half of the mediawiki core tables have been migrated to
abstract schema (plus lots of extensions lika Wikibase, Babel, Linter,
BetaFeatures, etc.). Special thanks to Tgr for reviewing most of the
patches and Sam Reed and James Forrester for doing the extensions.

With the growing number of schemas being abstracted, this is going to
affect your development if you work on schema and schema changes in core or
any of the extensions. So If you do, please read Manual:Schema changes
 in mediawiki.org

You might think that abstraction is just migrating SQL to JSON but it's
much more, we are making the database schema of mediawiki much more
consistent, We are basically addressing several long standing issues like
T164898  and T42626
 as well.

*Improvement aspects*

First aspect is drifts between different DBMSes. Sqlite schema is being
produced by regex replacement (this code
)
which is less than great but at least it comes from one place. For
Postgres, its schema and MySQL/Sqlite has drifted so drastically, that
fixing it so far required 76 schema changes fixing issues ranging from
missing indexes to missing PKs, extra AUTO_INCREMENT where it shouldn't be,
missing DEFAULT values, drifting data types and much more.  You can follow
the fixes of Postgres in here .

The second aspect is the inconsistency in the schema itself. How do we
model strings? VARCHAR? VARBINARY()? VARCHAR() BINARY? (all three are
different things). You'd be surprised how inconsistent our MySQL is. So
far, we are migrating all VARCHAR() BINARY fields to VARBINARY() (so far
ten schema changes).

Another inconsistency is timestamps. In MySQL, around half of them are
BINARY(14) and the other half VARBINARY(14) (but in Postgres all are
TIMESTAMPTZ), there is even a ticket
 about it. It makes sense to
migrate all of them to BINARY(14) but not all timestamps are 14 characters,
e.g. expiry fields accept "infinity" as value and it's a valid timestamp in
Postgres ¯\_(ツ)_/¯ When you turn an expiry field to BINARY(14), "infinity"
becomes "  infinity" and as the result mediawiki doesn't recognize it
as infinity ("infinity" != "  infinity"). There are several ways to
move forward handling expiry fields, you can follow the discussion in this
gerrit patch .

Another fun aspect: Booleans. MySQL doesn't have boolean, it translates
them to TINYINT(1) but other DBMSes don't have TINYINT, they have SMALLINT
and BOOL though (and we mostly use SMALLINT for them), we decided to go
with SMALLINT for these cases (which is different than what Doctrine DBAL
does, it uses BOOL, so we introduced our own custom type for booleans).

Last but not least: ENUMs. MySQL and Postgres support that but Sqlite
doesn't. Doctrine DBAL doesn't support ENUM at all (as it's an
anti-pattern) while core has eight fields that are ENUM. There's an RFC to
discourage using it in general. Feel free to comment on it.


A miscellaneous note: The directories that hold the archive of sql patches
of schema change are exploding (some of the sql patches are even orphan but
we can't find them because there are so many of them). So I started a RFC
to clean that mess up: Drop support for database upgrade older than two LTS
releases 

*What's next?*

   -  We continue to migrate more tables, hopefully we will get two third
   of them by the end of the year (fingers crossed). You can follow the
   progress in its ticket .
   -  We will support abstract schema changes, really soon, like in a
   couple of weeks. Basically you start a json file containing snapshots of
   before and after of a table and then a maintenance script will produce the
   needed sql patches for you for different schemas. This will increase the
   developer productivity drastically, since 1- Schema change sql files become
   more reliable and consistent and less prone to errors like adding the
   column to the wrong table in some DBMSes
   

   2- You don't need to know Postgres or Sqlite peculiarities to make patches
   against it. The reason you need to proved the whole table for adding like
   an index is that sqlite doesn't support all types of ALTER TABLES, 

[Wikitech-l] RFC: Drop support for database upgrade of mediawiki older than two LTS releases

[Amir Sarabadani]

Hello,
If you're not maintaining a mediawiki instance or don't work with
mediawiki's installer/updater, feel free to ignore this email (sorry for
spam).

The aforementioned RFC has reached phase 3, which means we need to reach
out to stakeholders and since this RFC is going to impact third party
installations, hence this email.

Currently and in theory, mediawiki supports upgrading from 1.2 (released in
2004) to its current version but if this RFC is approved, you wouldn't be
able to upgrade from versions older than two LTS releases. So for example
for upgrading to 1.35, you can't upgrade to it from 1.26 or lower, 1.26 was
released in end of 2015 and EOL'd in end of 2016, if you want to upgrade
from 1.26, you have to first upgrade to 1.34 and then you would be able to
upgrade to 1.35 [Note that this is an example, 1.35 is already released and
thus not affected by this RFC, this is going to affect future releases].

Here's the RFC: https://phabricator.wikimedia.org/T259771 Feel free to
chime in and spread the word to other stakeholders.

Thanks
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The Second round of voting for mediawiki logo just started!

[Amir Sarabadani]

(Sorry for late response, this email fell into cracks of my messy inbox)

On Mon, Sep 28, 2020 at 1:53 PM bawolff  wrote:

> TBH, I was under the impression that the second round was going to be
> narrowing down to top contenders (maybe the 3 or so top designs), not
> choosing the top contender (I guess that's my fault though, it wasn't
> stated anywhere that that was going to be the case or anything).
>

Actually the plan originally was to put anything that passed the basic wiki
thresholds (70%) but nothing beside the first proposal did. I could add the
proposal one and the other ones as runner ups but honestly it feels weird
advancing a logo design that had an opposition for each support.


> It was kind of hard to follow the first round with 20 something proposals
>

It was only 17, didn't even reach 20.

with some of them benefiting from showing up earlier than others, and most
> of the votes taking place during the time period where votes were allegedly
> not going to count yet.
>

That's true, I accept the mess up on my side (I have been planning this and
asking around for more than a year now but it's not like you coordinate
such changes on a monthly basis, I'm definitely learning), I tried to
compensate by giving a full month for the voting period which is pretty
long plus giving periodic reminders.


> I did notice that some of the people voting had never previously edited
> mediawiki.org (Or made very few previous edits). It kind of feels a
> little weird to treat this as a "vote" (and not a "consensus" building
> exercise) if we don't have eligibility criteria.
>

This is the part that changing the mediawiki logo is different from the
usual wikimedia decision making process. The reason is that in for example
English Wikipedia, the biggest venue of contribution is en.wikipedia.org
but the biggest venue for contributing to mediawiki is not mediawiki.org,
you make patches, you report bugs, you help people in IRC, and so on. If
you counted that, I assume a huge portion of the voters would be considered
eligible if we count venues like phabricator and gerrit. Also for example
for "picture of the year competition" in Commons, the eligibility is not
the number of edits in commons. It's the number of edits in any wiki and if
we want to count the number of edits in any wiki too, then I'm pretty sure
virtually every voter would be considered eligible.

>
> I do kind of wish there was a none of the above option.
>

Proposal four was the status quo, it clearly didn't pass (with 39%)


> Looking through the votes, I definitely see some people saying things like
> "Least bad option", which is not exactly an inspiring show of support.
>

Well, for each neutral or weak support in the proposal six, there was one
person who showed "strong support".  And this is the thing with logos, it's
not like a voting on a policy change, supporting or not supporting a logo
is a very subjective matter, for a similar situation look at elections,
there are people who go crazy about a candidate and people who are just
like "less horrible than the other candidate" and this is normal, people
are different with different perspective, If I wanted to force my
perspective, I would have advanced the first proposal too and even in the
variants for the current proposal, my favorites are not getting anywhere
but that's okay. Logos have lots of oobjective factors (like accessibility,
proper abstraction, color consistency, simplicity, brand awareness, etc.)
but the biggest one is the general look and feel and it differs from person
to person. That's why companies do extensive A/B testing on design, it
caused backlash too, for example a lead designer who left Google in protest
that they were doing A/B testing on forty different shades of blue which
basically destroyed the artistic freedom of designers (we are not going in
that direction but we need to acknowledge the subjectivity of designs)


HTH

>
> --
> Brian
>
> On Mon, Sep 28, 2020 at 8:50 AM Amir Sarabadani 
> wrote:
>
>> Hey,
>> The first round was using the standard voting process in wikis (using
>> support/oppose and the thresholds like 70%) and this is the way we elect
>> admins, checkusers or other user rights, or change policies in Wikis. I
>> don't recall that there has ever been anyone elected as admin with below
>> 70% or we have ever changed any policies with below 70% (not to mention the
>> runner up logos are 56% and 61%, basically for any support, they had an
>> opposition). Our logo is similar, no logo except proposal six could reach
>> seventy percent and while there were good designs that almost made it but
>> clearly none of them has enough support (and percentage of support) to
>> reach the next round. That's a pity (one of the runner ups was actually by
>&g

Re: [Wikitech-l] The Second round of voting for mediawiki logo just started!

[Amir Sarabadani]

Hey,
The first round was using the standard voting process in wikis (using
support/oppose and the thresholds like 70%) and this is the way we elect
admins, checkusers or other user rights, or change policies in Wikis. I
don't recall that there has ever been anyone elected as admin with below
70% or we have ever changed any policies with below 70% (not to mention the
runner up logos are 56% and 61%, basically for any support, they had an
opposition). Our logo is similar, no logo except proposal six could reach
seventy percent and while there were good designs that almost made it but
clearly none of them has enough support (and percentage of support) to
reach the next round. That's a pity (one of the runner ups was actually by
me) but if that's what the community wants, I happily accept it.

The second round has always been
<https://www.mediawiki.org/w/index.php?title=Project:Proposal_for_changing_logo_of_MediaWiki,_2020/Round_1=4006263=3997205>
about different variants of the logos that pass the first round.

HTH

On Mon, Sep 28, 2020 at 9:30 AM Adam Wight  wrote:

> Hi, thanks for helping coordinate this process!
>
> I have concerns about what happened between round 1 and round 2, it seems
> that we're no longer left with a real choice.  It's unclear what method was
> used to tally the round 1
> <https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_MediaWiki,_2020/Round_1>
> votes, was this a "support percentage"?  Whenever a vote is taken, it's
> important to stick to democratic norms, basically "one person, one vote".
> Round 2 is entirely variations on a single proposal, which disenfranchises
> everyone who didn't prefer that design.  Is it too late to discuss?
>
> Kind regards,
> Adam
> On 9/25/20 11:42 PM, Amir Sarabadani wrote:
>
> Hello,
> The subject line is self-explanatory, you can go to the voting page
> <https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_MediaWiki,_2020/Round_2>
> and cast your vote.
>
> This is going to continue for a month and it's about different variants of
> the top contender (different colors, different wordmarks, etc.). You need
> to order logos based on your preference (the most preferred one first, the
> least preferred one the last) and then cast your vote. The final winner
> will be chosen using Schulze method
> <https://en.wikipedia.org/wiki/Schulze_method>.
>
> If you have mistakenly voted in the test phase, you can just copy your
> vote from the test page
> <https://www.mediawiki.org/wiki/User:Ladsgroup/Round_2/votes> to the
> actual voting page
> <https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_MediaWiki,_2020/Round_2/Votes>
> (the numbers of logos haven't changed).
>
> Special thank you to Chuck Roslof from WMF legal for doing the preliminary
> clearance of the proposal.
>
> Have a nice weekend!
> --
> Amir (he/him)
>
>
> ___
> Wikitech-l mailing 
> listWikitech-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] The Second round of voting for mediawiki logo just started!

[Amir Sarabadani]

Hello,
The subject line is self-explanatory, you can go to the voting page

and cast your vote.

This is going to continue for a month and it's about different variants of
the top contender (different colors, different wordmarks, etc.). You need
to order logos based on your preference (the most preferred one first, the
least preferred one the last) and then cast your vote. The final winner
will be chosen using Schulze method
.

If you have mistakenly voted in the test phase, you can just copy your vote
from the test page
 to the actual
voting page

(the numbers of logos haven't changed).

Special thank you to Chuck Roslof from WMF legal for doing the preliminary
clearance of the proposal.

Have a nice weekend!
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Second round of voting for the new logo of mediawiki - testing the voting gadget

[Amir Sarabadani]

Hey,
Thanks for the feedback. It's now random.

On Sat, Sep 19, 2020 at 9:29 PM QEDK  wrote:

> Hello Amir,
>
> I think that the ordering should be randomized to rule out bias, as most
> people will choose to drag the least since dragging is one of the most
> high-effort interactions. Otherwise, it looks pretty good.
>
> Best,
> QEDK
>
> [1] https://en.wikipedia.org/wiki/Default_effect
> [2] https://en.wikipedia.org/wiki/Above_the_fold
> [3] https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)
>
> On Sat, 19 Sep 2020 at 23:00, Amir Sarabadani  wrote:
>
> > Hello,
> > The second round of voting for the new logo will start soon. In this
> round,
> > we are going to vote on different variants of the top design (Proposal
> > six). Given that 16 different variants of this logo exist (with different
> > wordmarks, colors, etc.), we need preferential voting instead of the
> > classic support/oppose. We will use Schulze method [1] for choosing the
> > final winner.
> >
> > I hacked together a simple voting gadget that you can test in
> >
> >
> https://www.mediawiki.org/wiki/User:Ladsgroup/Round_2?withJS=MediaWiki:Gadget-logo-voting.js
> >
> > Using that, you just order logos based on how much you like them (with
> drag
> > and drop), and then click save. The votes will be saved in another page.
> > Please test and let me know of any issues by the next couple of days (by
> > September 25). Any edits on the gadget is also welcome. Please put your
> > suggestions on its talk page if you don't have rights to edit it.
> >
> > Of course, voting without the gadget will be also possible, it'll be just
> > slightly cumbersome.
> >
> > One known issue so far: The gadget on mobile doesn't work because mobile
> > browsers don't listen to drag events.
> >
> > [1] https://en.wikipedia.org/wiki/Schulze_method
> > Best
> > --
> > Amir (he/him)
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Second round of voting for the new logo of mediawiki - testing the voting gadget

[Amir Sarabadani]

Hello,
The second round of voting for the new logo will start soon. In this round,
we are going to vote on different variants of the top design (Proposal
six). Given that 16 different variants of this logo exist (with different
wordmarks, colors, etc.), we need preferential voting instead of the
classic support/oppose. We will use Schulze method [1] for choosing the
final winner.

I hacked together a simple voting gadget that you can test in
https://www.mediawiki.org/wiki/User:Ladsgroup/Round_2?withJS=MediaWiki:Gadget-logo-voting.js

Using that, you just order logos based on how much you like them (with drag
and drop), and then click save. The votes will be saved in another page.
Please test and let me know of any issues by the next couple of days (by
September 25). Any edits on the gadget is also welcome. Please put your
suggestions on its talk page if you don't have rights to edit it.

Of course, voting without the gadget will be also possible, it'll be just
slightly cumbersome.

One known issue so far: The gadget on mobile doesn't work because mobile
browsers don't listen to drag events.

[1] https://en.wikipedia.org/wiki/Schulze_method
Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] CoC amendment proposal: Appeals process

[Amir Sarabadani]

Hello,
This is Amir from the CoC committee. Currently, there are two open
amendment proposals that haven't had much progress in the past years and I
would like to close them or at least make some progress. One of them in
particular is really important.

There are two major issues with the current appeal process:
* According to the CoC, the appeal body is a team in WMF that doesn't exist
anymore (for years now) [1] People in this team (who all are still working
for WMF) have moved to other teams and some haven't done anything with
technical communities for years now. We need to find an appeal body to
replace this. One suggestion is to have CRC [2] and its successor review
these cases but WMF legal should give the green light on this. Any other
suggestion is more than welcome.
* It's not clear that the appeal body would have the authority to override
the outcome of cases or they are just advisory or they are peers (they need
to discuss until there's a consensus among both).

Given the confidential nature of TCoC cases, having a robust appeal body
ensures a fair and just environment for technical contributions. Please
take a look and participate in the discussion:
https://www.mediawiki.org/wiki/Topic:Uay0vz6no8ayac3m

Thank you very much.

[1] https://meta.wikimedia.org/wiki/Community_Relations/Community_health
[2] https://meta.wikimedia.org/wiki/Trust_and_Safety/Case_Review_Committee
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikimedia Chat

[Amir Sarabadani]

Hey!

On Mon, Aug 31, 2020 at 9:56 AM Count Count 
wrote:

> Hi!
>
> Thanks for providing another communication tool.
>
> Some questions:
>
>- Is there a reason why we are not using Wikimedia account SSO for the
>chat?
>
Due to the privacy policy of Wikimedia Cloud, we are not allowed to use SUL
or LDAP of Wikimedia or Wikitech respectively, but I have been looking into
improving the user integration because it has been asked in multiple
places. You can follow it in https://phabricator.wikimedia.org/T261631

Mattermost doesn't have a native support for Wikimedia OAuth but it has
support for GitLab OAuth and funnily enough, since everything is
customizable in settings, I assume it should workTM, if yes, it would look
weird but it would work. If not, we have to file a bug in upstream
(mattermost) and ask for a general OAuth support.

   - The Terms link leads directly to the Tech Code of Conduct. Shouldn't
>it link to  https://foundation.wikimedia.org/wiki/Terms_of_Use/en
> instead?
>After all terms are usually more than just a Code of Conduct.
>

I don't have many preferences in that regard, I have one choice for terms
of use and can't decide between this or that.

>- The linked privacy policy
><https://wikitech.wikimedia.org/wiki/Nova_Resource:Chat/Privacy_Policy>
> is
>a bit scary. It states that there is no expectation of privacy as my
> chats
>could be published at any time and that the administrators might have
>access to my username/password combination, which should not at all be
>necessary. Is that really the intent here?
>

As said before, this is the standard privacy policy that we must use in
order to comply with the Cloud terms of use
https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use

Not that I won't publish anything private, also the mattermost itself
protects private data (like passwords) by hashing them.

>
>
> Best regards,
>
> Count Count
>
> On Sun, Aug 30, 2020 at 2:00 AM Amir Sarabadani 
> wrote:
>
> > Hello,
> > Due to the current situation, there are more and more collaborations
> > happening online instead. and now you can see Wikimedia-related
> discussion
> > groups in Slack, Discord, Telegram, Facebook, and many more. Besides
> being
> > scattered and inaccessible to people who don't have accounts in those
> > platforms (for privacy reasons for example), these platforms use
> > proprietary and closed-source software, are outside Wikimedia
> > infrastructure and some harvest our personal data for profit.
> >
> > IRC on freenode is a good alternative but it lacks basic functionalities
> of
> > a modern chat platform. So we created Wikimedia Chat, a mattermost
> instance
> > in Wikimedia Cloud. Compared to IRC, you have:
> > * Ability to scrollback and read messages when you were offline
> > * Push notification and email notification
> > * You don't need to get a cloak to hide your IP from others
> > * Proper support for sharing media
> > * Two factor authentication
> > * A proper mobile app support
> > * Ability to add custom emojis (yes, it's extremely important)
> > * Profile pictures
> > * Ability to ping everyone with @here
> > * much much more.
> >
> > You can use Wikimedia Chat by going to https://chat.wmcloud.org, anyone
> > can
> > make an account. This is part of Wikimedia Social suite [1], the oher
> > similar project is "Wikimedia Meet". [2]
> >
> > Some notes:
> > * This is done in my volunteer capacity and has been maintained by a
> group
> > of volunteers. If you're willing to join the team (either technical or
> > enforcing CoC, kicking out spammers, other daily work), drop me a
> message.
> > * Privacy policy of Wikimedia Cloud applies: https://w.wiki/aQW
> > * As a result, all messages older than 90 days get automatically deleted.
> > * As a Wikimedia Cloud project, all of discussions, private and public
> are
> > covered by Code of conduct in technical spaces:  https://w.wiki/AK$
> >
> > Hope that would be useful for you, if you encounter any technical issues,
> > file a bug in the phabricator.
> >
> > [1] https://meta.wikimedia.org/wiki/Wikimedia_Social_Suite
> > [2] https://meta.wikimedia.org/wiki/Wikimedia_Meet
> >
> > Best
> > --
> > Amir (he/him)
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Logo of mediawiki

[Amir Sarabadani]

Hello everyone,
I just wanted to remind you that only one week left from the voting period
of the first round (general design and look of proposals).

Best

On Mon, Aug 10, 2020 at 2:42 PM Amir Sarabadani  wrote:

> Hey again.
> I just wanted to remind people that the first round of voting has started,
> you can vote on the general look and design of the proposals (but you can't
> add more anymore). This will continue for a month and after that there will
> be another round of voting for variants of the top designs (like different
> wordmarks, different colors, etc.)
>
> Thank you for your contributions on this project.
>
> Best
>
> On Sun, Jul 19, 2020 at 3:44 PM Amir Sarabadani 
> wrote:
>
>> Hello everyone,
>> Thank you for your enthusiasm around the logo. It made me so happy, and
>> also thank you for all of the feedback around the process, I'm certainly
>> learning.
>> Based on the feedback I want to say some notes:
>>
>> With all other discussions happening at the same time this can be
>> draining and having a global pandemic going on is not going to help either.
>> So the deadlines are pretty relaxed.
>>
>> It's already been a month (almost) and let's add three weeks for adding
>> and discussing proposals, meaning until 10th of August you can add and
>> discuss proposals. After that, we will have a month for the first round of
>> voting, In this round, you can vote on the logo designs in general, meaning
>> until 10th of September you can vote on logos general look and design (not
>> the colors, wordmark, number of pedals, etc). The second round of voting
>> will be a month afterwards (meaning until 10th of October) which you can
>> vote on different variants of the top logos of the first round (like
>> different wordmarks, different colors, and so on) and after that we will
>> move forward with implementing the top logo and variant.
>>
>> In short, there will be three phases:
>>  - Current phase (until 10th of August): Suggest proposal and discuss
>> proposals.
>>  - Second phase (until 10th of September): Vote on general design of all
>> proposals
>>  - Third phase (until 10th of October): Vote on different variants of the
>> top proposals.
>>
>> If anyone wants to help with facilitating discussions, I'd be more than
>> happy to have some help. Send me a private message and we can form a group.
>>
>> Also, I would like to humbly ask everyone to be more mindful in their
>> comments, each proposal is a product of someone's creative thinking and
>> work. Especially avoid criticizing people. In other words, please be
>> kinder, we all can use a little bit of kindness these days.
>>
>> Thank you.
>>
>> On Mon, Jun 22, 2020 at 2:39 AM Amir Sarabadani 
>> wrote:
>>
>>> Hello,
>>> The current logo of MediaWiki was adapted slightly more than fifteen
>>> years ago and hasn’t changed since. This logo despite having the nice
>>> concept of sunflower, is old. The sunflower represents the diversity, the
>>> constant growth and also the wildness.
>>>
>>> Among its biggest issues I can point out that it’s a bitmap picture so
>>> it’s unusable in large sizes (like large posters) and it’s too realistic
>>> making it unusable in small sizes.
>>>
>>> Most, virtually all, software products use a simpler and more abstract
>>> form. For example, docker, kubernetes, Ubuntu, VueJs, React, Apache Kafka,
>>> and many more. It’s a good time for MediaWiki to follow suit.
>>>
>>> My request is for changing the logo of MediaWiki and I have no plans or
>>> interest in changing logo of any other project.
>>>
>>> Please show your support, oppose or your comments in the discussion
>>> page. You can also add more suggestions.
>>>
>>> The discussion page:
>>> https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_mediawiki,_2020
>>>
>>> Best
>>> --
>>> Amir (he/him)
>>>
>>>
>>
>> --
>> Amir (he/him)
>>
>>
>
> --
> Amir (he/him)
>
>

-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikimedia Chat

[Amir Sarabadani]

I don't have any issues atm (tried again with different connections, etc.).
Is it common, does others have this issue too?

On Sun, Aug 30, 2020 at 3:21 PM QEDK  wrote:

> I'm only getting 408 (connection timed out) errors.
>
> Best,
> QEDK
>
> On Sun, Aug 30, 2020, 06:26 Esteban Zarate  wrote:
>
> > Thanks, it works!!
> >
> > El sáb., 29 de ago. de 2020 a la(s) 21:40, Amir Sarabadani (
> > ladsgr...@gmail.com) escribió:
> >
> > > Hey, can you try? https://chat.wmcloud.org/wikimedia
> > >
> > > On Sun, Aug 30, 2020 at 2:37 AM Esteban Zarate 
> > wrote:
> > >
> > > > I just installed but there is nothing, can you create a channel as
> > test?,
> > > > please
> > > >
> > > > El sáb., 29 de ago. de 2020 a la(s) 21:00, Amir Sarabadani (
> > > > ladsgr...@gmail.com) escribió:
> > > >
> > > > > Hello,
> > > > > Due to the current situation, there are more and more
> collaborations
> > > > > happening online instead. and now you can see Wikimedia-related
> > > > discussion
> > > > > groups in Slack, Discord, Telegram, Facebook, and many more.
> Besides
> > > > being
> > > > > scattered and inaccessible to people who don't have accounts in
> those
> > > > > platforms (for privacy reasons for example), these platforms use
> > > > > proprietary and closed-source software, are outside Wikimedia
> > > > > infrastructure and some harvest our personal data for profit.
> > > > >
> > > > > IRC on freenode is a good alternative but it lacks basic
> > > functionalities
> > > > of
> > > > > a modern chat platform. So we created Wikimedia Chat, a mattermost
> > > > instance
> > > > > in Wikimedia Cloud. Compared to IRC, you have:
> > > > > * Ability to scrollback and read messages when you were offline
> > > > > * Push notification and email notification
> > > > > * You don't need to get a cloak to hide your IP from others
> > > > > * Proper support for sharing media
> > > > > * Two factor authentication
> > > > > * A proper mobile app support
> > > > > * Ability to add custom emojis (yes, it's extremely important)
> > > > > * Profile pictures
> > > > > * Ability to ping everyone with @here
> > > > > * much much more.
> > > > >
> > > > > You can use Wikimedia Chat by going to https://chat.wmcloud.org,
> > > anyone
> > > > > can
> > > > > make an account. This is part of Wikimedia Social suite [1], the
> oher
> > > > > similar project is "Wikimedia Meet". [2]
> > > > >
> > > > > Some notes:
> > > > > * This is done in my volunteer capacity and has been maintained by
> a
> > > > group
> > > > > of volunteers. If you're willing to join the team (either technical
> > or
> > > > > enforcing CoC, kicking out spammers, other daily work), drop me a
> > > > message.
> > > > > * Privacy policy of Wikimedia Cloud applies: https://w.wiki/aQW
> > > > > * As a result, all messages older than 90 days get automatically
> > > deleted.
> > > > > * As a Wikimedia Cloud project, all of discussions, private and
> > public
> > > > are
> > > > > covered by Code of conduct in technical spaces:
> https://w.wiki/AK$
> > > > >
> > > > > Hope that would be useful for you, if you encounter any technical
> > > issues,
> > > > > file a bug in the phabricator.
> > > > >
> > > > > [1] https://meta.wikimedia.org/wiki/Wikimedia_Social_Suite
> > > > > [2] https://meta.wikimedia.org/wiki/Wikimedia_Meet
> > > > >
> > > > > Best
> > > > > --
> > > > > Amir (he/him)
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > >
> > > > ___
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > >
> > >
> > >
> > > --
> > > Amir (he/him)
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikimedia Chat

[Amir Sarabadani]

Hey, can you try? https://chat.wmcloud.org/wikimedia

On Sun, Aug 30, 2020 at 2:37 AM Esteban Zarate  wrote:

> I just installed but there is nothing, can you create a channel as test?,
> please
>
> El sáb., 29 de ago. de 2020 a la(s) 21:00, Amir Sarabadani (
> ladsgr...@gmail.com) escribió:
>
> > Hello,
> > Due to the current situation, there are more and more collaborations
> > happening online instead. and now you can see Wikimedia-related
> discussion
> > groups in Slack, Discord, Telegram, Facebook, and many more. Besides
> being
> > scattered and inaccessible to people who don't have accounts in those
> > platforms (for privacy reasons for example), these platforms use
> > proprietary and closed-source software, are outside Wikimedia
> > infrastructure and some harvest our personal data for profit.
> >
> > IRC on freenode is a good alternative but it lacks basic functionalities
> of
> > a modern chat platform. So we created Wikimedia Chat, a mattermost
> instance
> > in Wikimedia Cloud. Compared to IRC, you have:
> > * Ability to scrollback and read messages when you were offline
> > * Push notification and email notification
> > * You don't need to get a cloak to hide your IP from others
> > * Proper support for sharing media
> > * Two factor authentication
> > * A proper mobile app support
> > * Ability to add custom emojis (yes, it's extremely important)
> > * Profile pictures
> > * Ability to ping everyone with @here
> > * much much more.
> >
> > You can use Wikimedia Chat by going to https://chat.wmcloud.org, anyone
> > can
> > make an account. This is part of Wikimedia Social suite [1], the oher
> > similar project is "Wikimedia Meet". [2]
> >
> > Some notes:
> > * This is done in my volunteer capacity and has been maintained by a
> group
> > of volunteers. If you're willing to join the team (either technical or
> > enforcing CoC, kicking out spammers, other daily work), drop me a
> message.
> > * Privacy policy of Wikimedia Cloud applies: https://w.wiki/aQW
> > * As a result, all messages older than 90 days get automatically deleted.
> > * As a Wikimedia Cloud project, all of discussions, private and public
> are
> > covered by Code of conduct in technical spaces:  https://w.wiki/AK$
> >
> > Hope that would be useful for you, if you encounter any technical issues,
> > file a bug in the phabricator.
> >
> > [1] https://meta.wikimedia.org/wiki/Wikimedia_Social_Suite
> > [2] https://meta.wikimedia.org/wiki/Wikimedia_Meet
> >
> > Best
> > --
> > Amir (he/him)
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Wikimedia Chat

[Amir Sarabadani]

Hello,
Due to the current situation, there are more and more collaborations
happening online instead. and now you can see Wikimedia-related discussion
groups in Slack, Discord, Telegram, Facebook, and many more. Besides being
scattered and inaccessible to people who don't have accounts in those
platforms (for privacy reasons for example), these platforms use
proprietary and closed-source software, are outside Wikimedia
infrastructure and some harvest our personal data for profit.

IRC on freenode is a good alternative but it lacks basic functionalities of
a modern chat platform. So we created Wikimedia Chat, a mattermost instance
in Wikimedia Cloud. Compared to IRC, you have:
* Ability to scrollback and read messages when you were offline
* Push notification and email notification
* You don't need to get a cloak to hide your IP from others
* Proper support for sharing media
* Two factor authentication
* A proper mobile app support
* Ability to add custom emojis (yes, it's extremely important)
* Profile pictures
* Ability to ping everyone with @here
* much much more.

You can use Wikimedia Chat by going to https://chat.wmcloud.org, anyone can
make an account. This is part of Wikimedia Social suite [1], the oher
similar project is "Wikimedia Meet". [2]

Some notes:
* This is done in my volunteer capacity and has been maintained by a group
of volunteers. If you're willing to join the team (either technical or
enforcing CoC, kicking out spammers, other daily work), drop me a message.
* Privacy policy of Wikimedia Cloud applies: https://w.wiki/aQW
* As a result, all messages older than 90 days get automatically deleted.
* As a Wikimedia Cloud project, all of discussions, private and public are
covered by Code of conduct in technical spaces:  https://w.wiki/AK$

Hope that would be useful for you, if you encounter any technical issues,
file a bug in the phabricator.

[1] https://meta.wikimedia.org/wiki/Wikimedia_Social_Suite
[2] https://meta.wikimedia.org/wiki/Wikimedia_Meet

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upgrading mailman (the software behind mailing lists)

[Amir Sarabadani]

I think it's because mailman3 doesn't need something like listadmin
anymore. It provides a REST API that you can use to achieve the same
functionality in a couple lines of code:
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/rest.html

HTH

On Mon, Aug 24, 2020 at 1:30 AM Michael Schönitzer <
michael.schoenit...@wikimedia.de> wrote:

> A sad side effect of the upgrade will probably be that listadmin[1] will
> break. Does anyone know an alternative?
>
> [1] https://sourceforge.net/projects/listadmin/
>
> Am So., 9. Aug. 2020 um 23:13 Uhr schrieb David Gerard  >:
>
> > In fairness, pipermail archives were always a bit shaky - on the
> > occasions when an email's had to be removed from the archive
> > previously, it's messed up the URLs of all the other emails in that
> > month's archive.
> >
> > But let's say it would be *nice* not to mess up the public archive
> > URLs if feasible :-)
> >
> >
> > - d.
> >
> >
> > On Sun, 9 Aug 2020 at 21:19, Amir Sarabadani 
> wrote:
> > >
> > > hmm, Links of archived discussions in private mailing lists are not as
> > > important as the ones in public lists, we definitely should migrate
> > public
> > > mailing lists first and after that we can migrate any private mailing
> > list
> > > that is okay with their links being broken (and then we remove those
> old
> > > archives to make it unaccessible to public). It probably means we need
> to
> > > keep mailman2 around for a while.
> > >
> > > On Sun, Aug 9, 2020 at 9:38 PM AntiCompositeNumber <
> > > anticompositenum...@gmail.com> wrote:
> > >
> > > > I would agree that it would be a good solution, except for the next
> > > > bullet in the same document:
> > > > "The above mechanism won’t work for private archives since the
> > > > archives are gated with password and without a Mailman 2 list, there
> > > > is no password. You can however import them to Mailman 3."
> > > >
> > > > Since keeping private lists private is also a requirement, that
> pretty
> > > > much means rolling our own auth system on top of Mailman 3 or
> creating
> > > > a bunch of HTTP redirect rules.
> > > >
> > > > On Sun, Aug 9, 2020 at 3:31 PM Amir Sarabadani 
> > > > wrote:
> > > > >
> > > > > According to the upgrade guide (
> > > > >
> > https://docs.mailman3.org/en/latest/migration.html#other-considerations
> > > > ):
> > > > > "If you need your URLs for Mailman 2 archives to work, you can keep
> > the
> > > > > HTML files generated for the archives around and your web server
> > > > > configuration for the archives intact (possibly with a notice to
> > viewers
> > > > > that it is now a read-only archive, see this list
> > > > > <https://mail.python.org/pipermail/security-sig/> for example)."
> > > > >
> > > > > Here's an example: https://mail.python.org/pipermail/security-sig/
> > > > (with a
> > > > > notice).
> > > > >
> > > > > It means, the old archives will stay the same (and
> > accessible/searchable
> > > > > with the new interface as well) but new mails won't get added there
> > to
> > > > the
> > > > > old archives. I think that's a good compromise.
> > > > >
> > > > >
> > > > > On Sun, Aug 9, 2020 at 9:19 PM David Gerard 
> > wrote:
> > > > >
> > > > > > yes - those links are thrown around as if they're archival. How
> > will
> > > > > > the change affect links to past messages? Will someone need to
> > > > > > construct a redirect farm?
> > > > > >
> > > > > >
> > > > > > - d.
> > > > > >
> > > > > > On Sun, 9 Aug 2020 at 19:54, AntiCompositeNumber
> > > > > >  wrote:
> > > > > > >
> > > > > > > Glad to hear this is moving forward!
> > > > > > >
> > > > > > > Keeping archive links working, for both public and private
> lists,
> > > > > > > should be a requirement. There's a lot of institutional
> knowledge
> > > > > > > stored in the mailing list archives, and it's very important to
> > keep
> > > > > > > that around.
> > >

Re: [Wikitech-l] Logo of mediawiki

[Amir Sarabadani]

Hey again.
I just wanted to remind people that the first round of voting has started,
you can vote on the general look and design of the proposals (but you can't
add more anymore). This will continue for a month and after that there will
be another round of voting for variants of the top designs (like different
wordmarks, different colors, etc.)

Thank you for your contributions on this project.

Best

On Sun, Jul 19, 2020 at 3:44 PM Amir Sarabadani  wrote:

> Hello everyone,
> Thank you for your enthusiasm around the logo. It made me so happy, and
> also thank you for all of the feedback around the process, I'm certainly
> learning.
> Based on the feedback I want to say some notes:
>
> With all other discussions happening at the same time this can be draining
> and having a global pandemic going on is not going to help either. So the
> deadlines are pretty relaxed.
>
> It's already been a month (almost) and let's add three weeks for adding
> and discussing proposals, meaning until 10th of August you can add and
> discuss proposals. After that, we will have a month for the first round of
> voting, In this round, you can vote on the logo designs in general, meaning
> until 10th of September you can vote on logos general look and design (not
> the colors, wordmark, number of pedals, etc). The second round of voting
> will be a month afterwards (meaning until 10th of October) which you can
> vote on different variants of the top logos of the first round (like
> different wordmarks, different colors, and so on) and after that we will
> move forward with implementing the top logo and variant.
>
> In short, there will be three phases:
>  - Current phase (until 10th of August): Suggest proposal and discuss
> proposals.
>  - Second phase (until 10th of September): Vote on general design of all
> proposals
>  - Third phase (until 10th of October): Vote on different variants of the
> top proposals.
>
> If anyone wants to help with facilitating discussions, I'd be more than
> happy to have some help. Send me a private message and we can form a group.
>
> Also, I would like to humbly ask everyone to be more mindful in their
> comments, each proposal is a product of someone's creative thinking and
> work. Especially avoid criticizing people. In other words, please be
> kinder, we all can use a little bit of kindness these days.
>
> Thank you.
>
> On Mon, Jun 22, 2020 at 2:39 AM Amir Sarabadani 
> wrote:
>
>> Hello,
>> The current logo of MediaWiki was adapted slightly more than fifteen
>> years ago and hasn’t changed since. This logo despite having the nice
>> concept of sunflower, is old. The sunflower represents the diversity, the
>> constant growth and also the wildness.
>>
>> Among its biggest issues I can point out that it’s a bitmap picture so
>> it’s unusable in large sizes (like large posters) and it’s too realistic
>> making it unusable in small sizes.
>>
>> Most, virtually all, software products use a simpler and more abstract
>> form. For example, docker, kubernetes, Ubuntu, VueJs, React, Apache Kafka,
>> and many more. It’s a good time for MediaWiki to follow suit.
>>
>> My request is for changing the logo of MediaWiki and I have no plans or
>> interest in changing logo of any other project.
>>
>> Please show your support, oppose or your comments in the discussion page.
>> You can also add more suggestions.
>>
>> The discussion page:
>> https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_mediawiki,_2020
>>
>> Best
>> --
>> Amir (he/him)
>>
>>
>
> --
> Amir (he/him)
>
>

-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upgrading mailman (the software behind mailing lists)

[Amir Sarabadani]

hmm, Links of archived discussions in private mailing lists are not as
important as the ones in public lists, we definitely should migrate public
mailing lists first and after that we can migrate any private mailing list
that is okay with their links being broken (and then we remove those old
archives to make it unaccessible to public). It probably means we need to
keep mailman2 around for a while.

On Sun, Aug 9, 2020 at 9:38 PM AntiCompositeNumber <
anticompositenum...@gmail.com> wrote:

> I would agree that it would be a good solution, except for the next
> bullet in the same document:
> "The above mechanism won’t work for private archives since the
> archives are gated with password and without a Mailman 2 list, there
> is no password. You can however import them to Mailman 3."
>
> Since keeping private lists private is also a requirement, that pretty
> much means rolling our own auth system on top of Mailman 3 or creating
> a bunch of HTTP redirect rules.
>
> On Sun, Aug 9, 2020 at 3:31 PM Amir Sarabadani 
> wrote:
> >
> > According to the upgrade guide (
> > https://docs.mailman3.org/en/latest/migration.html#other-considerations
> ):
> > "If you need your URLs for Mailman 2 archives to work, you can keep the
> > HTML files generated for the archives around and your web server
> > configuration for the archives intact (possibly with a notice to viewers
> > that it is now a read-only archive, see this list
> > <https://mail.python.org/pipermail/security-sig/> for example)."
> >
> > Here's an example: https://mail.python.org/pipermail/security-sig/
> (with a
> > notice).
> >
> > It means, the old archives will stay the same (and accessible/searchable
> > with the new interface as well) but new mails won't get added there to
> the
> > old archives. I think that's a good compromise.
> >
> >
> > On Sun, Aug 9, 2020 at 9:19 PM David Gerard  wrote:
> >
> > > yes - those links are thrown around as if they're archival. How will
> > > the change affect links to past messages? Will someone need to
> > > construct a redirect farm?
> > >
> > >
> > > - d.
> > >
> > > On Sun, 9 Aug 2020 at 19:54, AntiCompositeNumber
> > >  wrote:
> > > >
> > > > Glad to hear this is moving forward!
> > > >
> > > > Keeping archive links working, for both public and private lists,
> > > > should be a requirement. There's a lot of institutional knowledge
> > > > stored in the mailing list archives, and it's very important to keep
> > > > that around.
> > > >
> > > > ACN
> > > >
> > > > On Sun, Aug 9, 2020 at 1:05 PM Zoran Dori 
> > > wrote:
> > > > >
> > > > > Hello,
> > > > > this looks great. Especially because of "mobile friendly" function.
> > > > >
> > > > > Best regards,
> > > > > Zoran Dori
> > > > > Volunteer on Wikimedia Foundation's projects
> > > > > E: zorandori4...@gmail.com
> > > > > W: kizule.tk
> > > > > I: iamkizule <https://instagram.com/iamkizule>
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > >
> > > > ___
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> >
> >
> > --
> > Amir (he/him)
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upgrading mailman (the software behind mailing lists)

[Amir Sarabadani]

According to the upgrade guide (
https://docs.mailman3.org/en/latest/migration.html#other-considerations):
"If you need your URLs for Mailman 2 archives to work, you can keep the
HTML files generated for the archives around and your web server
configuration for the archives intact (possibly with a notice to viewers
that it is now a read-only archive, see this list
 for example)."

Here's an example: https://mail.python.org/pipermail/security-sig/ (with a
notice).

It means, the old archives will stay the same (and accessible/searchable
with the new interface as well) but new mails won't get added there to the
old archives. I think that's a good compromise.


On Sun, Aug 9, 2020 at 9:19 PM David Gerard  wrote:

> yes - those links are thrown around as if they're archival. How will
> the change affect links to past messages? Will someone need to
> construct a redirect farm?
>
>
> - d.
>
> On Sun, 9 Aug 2020 at 19:54, AntiCompositeNumber
>  wrote:
> >
> > Glad to hear this is moving forward!
> >
> > Keeping archive links working, for both public and private lists,
> > should be a requirement. There's a lot of institutional knowledge
> > stored in the mailing list archives, and it's very important to keep
> > that around.
> >
> > ACN
> >
> > On Sun, Aug 9, 2020 at 1:05 PM Zoran Dori 
> wrote:
> > >
> > > Hello,
> > > this looks great. Especially because of "mobile friendly" function.
> > >
> > > Best regards,
> > > Zoran Dori
> > > Volunteer on Wikimedia Foundation's projects
> > > E: zorandori4...@gmail.com
> > > W: kizule.tk
> > > I: iamkizule 
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Upgrading mailman (the software behind mailing lists)

[Amir Sarabadani]

Hey,
Mailman, the software that powers our mailing lists, is extremely old, by
looking at https://lists.wikimedia.org/ you can guess how old it is.

I would really like to upgrade it to mailman 3 which has these benefits:
* Much better security (including but not limited to
https://phabricator.wikimedia.org/T181803)
* Much better UI and UX
* Much easier moderation and maintaining mailing lists
* Ability to send mail from the web
* Ability to search in archives.
* Ability to like/dislike an email
* List admins will be able to delete emails, merge threads, and much more.
* Admins won't need to store passwords for each mailing list separately,
they just login as their account everywhere.
* The current mailman stores everything as files (even mailing list
settings), mailman3 actually uses a proper database for everything meaning
proper backup and recovery, high availability and much more.

I have already put up a test setup and humbly ask you (specially list
admins) to test it (and its admin interface), if you want to become a list
admin, drop me a message. Keep in mind that we don't maintain the software
so the most I can do is to change configuration and can't handle a feature
request or solve a bug (you are more than welcome to file it against
upstream though)

Here's the test setup:
* https://lists.wmcloud.org

Here's a mailing list:
* https://lists.wmcloud.org/postorius/lists/test.lists.wmcloud.org/

Here's an archive post:
*
https://lists.wmcloud.org/hyperkitty/list/t...@lists.wmcloud.org/thread/RMQPKSS4ID3WALFXAF636J2NGBVCN3UA/

Issues that I haven't figured out yet:
* This system has profile picture support but it's only gravatar which we
can't enable due to our privacy policy but when you disable it, it shows
empty squares and looks bad. Reported upstream [1] but also we can have a
gravatar proxy in production. And in the worst case scenario we can just
inject "$('.gravatar').remove();" and remove them. Feel free to chime in in
the phabricator ticket in this regard:
https://phabricator.wikimedia.org/T256541

* Upgrade will break archive links, making it work forever is not trivial
(you need write apache rewrite rule) (You can read about it in
https://docs.mailman3.org/en/latest/migration.html#other-considerations)

* Mailman allows us to upgrade mailing list by mailing list, that's good
but we haven't found a way to keep the old version and the new ones in sync
(archives, etc.). Maybe we migrate a mailing list and the archives for the
old version will stop getting updated. Would that work for you? Feel free
to chime in: https://phabricator.wikimedia.org/T256539

* We don't know what would be the size of the database after upgrade
because these two versions are so inherently different, one idea was to
check the size of a fully public mailing list, then move the files to the
test setup, upgrade it to the new version  and check how it changes, then
extrapolate the size of the final database. The discussion around the
database is happening in https://phabricator.wikimedia.org/T256538

If you want to help in the upgrade (like puppetzining its configuration,
etc.) just let me know and I add you to the project! It uses a stand-alone
puppetmaster so you don't need to get your puppet patches merged to see its
effects.

The main ticket about the upgrade: https://phabricator.wikimedia.org/T52864

[1] https://gitlab.com/mailman/hyperkitty/-/issues/303#note_365162201

Hope that'll be useful for you :)
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Ethical question regarding some code

[Amir Sarabadani]

Thank you all for the responses, I try to summarize my responses here.

* By closed source, I don't mean it will be only accessible to me, It's
already accessible by another CU and one WMF staff, and I would gladly
share the code with anyone who has signed NDA and they are of course more
than welcome to change it. Github has a really low limit for people who can
access a private repo but I would be fine with any means to fix this.

* I have read that people say that there are already public tools to
analyze text. I disagree, 1- The tools you mentioned are for English and
not other languages (maybe I missed something) and even if we imagine there
would be such tools for big languages like German and/or French, they don't
cover lots of languages unlike my tool that's basically language agnostic
and depends on the volume of discussions happened in the wiki.

* I also disagree that it's not hard to build. I have lots of experience
with NLP (with my favorite work being a tool that finds swear words in
every language based on history of vandalism in that Wikipedia [1]) and
still it took me more than a year (a couple of hours almost in every
weekend) to build this, analyzing a pure clean text is not hard, cleaning
up wikitext and templates and links to get only text people "spoke" is
doubly hard, analyzing user signatures brings only suffer and sorrow.

* While in general I agree if a government wants to build this, they can
but reality is more complicated and this situation is similar to security.
You can never be 100% secure but you can increase the cost of hacking you
so much that it would be pointless for a major actor to do it. Governments
have a limited budget and dictatorships are by design corrupt and filled
with incompotent people [2] and sanctions put another restrain on such
governments too so I would not give them such opportunity for oppersion in
a silver plate for free, if they really want to, then they must pay for it
(which means they can't use that money/resources on oppersing some other
groups).

* People have said this AI is easy to be gamed, while it's not that easy
and the tools you mentioned are limited to English, it's still a big win
for the integrity of our projects. It boils down again to increasing the
cost. If a major actor wants to spread disinformation, so far they only
need to fake their UA and IP which is a piece of cake and I already see
that (as a CU) but now they have to mess with UA/IP AND change their
methods of speaking (which is one order of magnitude harder than changing
IP). As I said, increasing this cost might not prevent it from happening
but at least it takes away the ability of oppressing other groups.

* This tool never will be the only reason to block a sock. It's more than
anything a helper, if CU brings a large range and they are similar but the
result is not conclusive, this tool can help. Or when we are 90% sure it's
a WP:DUCK, this tool can help too but blocking just because this tool said
so would imply a "Minority report" situation and to be honest and I would
really like to avoid that. It is supposed to empower CUs.

* Banning using this tool is not possible legally, the content of Wikipedia
is published under CC-BY-SA and this allows such analysis specially you
can't ban an offwiki action. Also, if a university professor can do it, I
don't see the point of banning using it by the most trusted group of users
(CUs). You can ban blocking based on this tool but I don't think we should
block solely based on this anyway.

* It has been pointed out by people in the checkuser mailing list that
there's no point in logging accessing this tool, since the code is
accessible to CUs (if they want to), so they can download and run it on
their computer without logging anyway.

* There is a huge difference between CU and this AI tool in matters of
privacy. While both are privacy sensitive but CU reveals much more, as a
CU, I know where lots of people are living or studying because they showed
up in my CUs and while I won't tell a soul about them but it makes me
uncomfortable (I'm also not implying CUs are not trusted, it's just we
should respect people's privacy and avoid "unreasonable search and
seizure"[3]) but this tool only reveals a connection between accounts if
one of them is linked to a public identity and the other is not which I
wholeheartedly agree is not great but it's not on the same level as seeing
people's IPs. So I even think in an ideal world where the AI model is more
accurate than CU, we should stop using CU and rely solely on the AI instead
(important: I'm not implying the current model is better, I'm saying if it
was better). This would help us understand why for example fishing for sock
puppets with CU is bad (and banned by the policy) but fishing for socks
using this AI is not bad and can be a good starting point. In other words,
this tool being used right, can reduce check user actions and protect
people's privacy instead.

* People have been saying 

[Wikitech-l] Ethical question regarding some code

[Amir Sarabadani]

Hey,
I have an ethical question that I couldn't answer yet and have been asking
around but no definite answer yet so I'm asking it in a larger audience in
hope of a solution.

For almost a year now, I have been developing an NLP-based AI system to be
able to catch sock puppets (two users pretending to be different but
actually the same person). It's based on the way they speak. The way we
speak is like a fingerprint and it's unique to us and it's really hard to
forge or change on demand (unlike IP/UA), as the result if you apply some
basic techniques in AI on Wikipedia discussions (which can be really
lengthy, trust me), the datasets and sock puppets shine.

Here's an example, I highly recommend looking at these graphs, I compared
two pairs of users, one pair that are not sock puppets and the other is a
pair of known socks (a user who got banned indefinitely but came back
hidden under another username). [1][2] These graphs are based one of
several aspects of this AI system.

I have talked about this with WMF and other CUs to build and help us
understand and catch socks. Especially the ones that have enough resources
to change their IP/UA regularly (like sock farms, and/or UPEs) and also
with the increase of mobile intern providers and the horrible way they
assign IP to their users, this can get really handy in some SPI ("Sock
puppet investigation") [3] cases.

The problem is that this tool, while being built only on public
information, actually has the power to expose legitimate sock puppets.
People who live under oppressive governments and edit on sensitive topics.
Disclosing such connections between two accounts can cost people their
lives.

So, this code is not going to be public, period. But we need to have this
code in Wikimedia Cloud Services so people like CUs in other wikis be able
to use it as a web-based tool instead of me running it for them upon
request. But WMCS terms of use explicitly say code should never be
closed-source and this is our principle. What should we do? I pay a
corporate cloud provider for this and put such important code and data
there? We amend the terms of use to have some exceptions like this one?

The most plausible solution suggested so far (thanks Huji) is to have a
shell of a code that would be useless without data, and keep the code that
produces the data (out of dumps) closed (which is fine, running that code
is not too hard even on enwiki) and update the data myself. This might be
doable (which I'm around 30% sure, it still might expose too much) but it
wouldn't cover future cases similar to mine and I think a more long-term
solution is needed here. Also, it would reduce the bus factor to 1, and
maintenance would be complicated.

What should we do?

Thanks
[1]
https://commons.wikimedia.org/wiki/File:Word_distributions_of_two_users_in_fawiki_1.png
[2]
https://commons.wikimedia.org/wiki/File:Word_distributions_of_two_users_in_fawiki_2.png
[3] https://en.wikipedia.org/wiki/Wikipedia:SPI
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Logo of mediawiki

[Amir Sarabadani]

Hello everyone,
Thank you for your enthusiasm around the logo. It made me so happy, and
also thank you for all of the feedback around the process, I'm certainly
learning.
Based on the feedback I want to say some notes:

With all other discussions happening at the same time this can be draining
and having a global pandemic going on is not going to help either. So the
deadlines are pretty relaxed.

It's already been a month (almost) and let's add three weeks for adding and
discussing proposals, meaning until 10th of August you can add and discuss
proposals. After that, we will have a month for the first round of voting,
In this round, you can vote on the logo designs in general, meaning until
10th of September you can vote on logos general look and design (not the
colors, wordmark, number of pedals, etc). The second round of voting will
be a month afterwards (meaning until 10th of October) which you can vote on
different variants of the top logos of the first round (like different
wordmarks, different colors, and so on) and after that we will move forward
with implementing the top logo and variant.

In short, there will be three phases:
 - Current phase (until 10th of August): Suggest proposal and discuss
proposals.
 - Second phase (until 10th of September): Vote on general design of all
proposals
 - Third phase (until 10th of October): Vote on different variants of the
top proposals.

If anyone wants to help with facilitating discussions, I'd be more than
happy to have some help. Send me a private message and we can form a group.

Also, I would like to humbly ask everyone to be more mindful in their
comments, each proposal is a product of someone's creative thinking and
work. Especially avoid criticizing people. In other words, please be
kinder, we all can use a little bit of kindness these days.

Thank you.

On Mon, Jun 22, 2020 at 2:39 AM Amir Sarabadani  wrote:

> Hello,
> The current logo of MediaWiki was adapted slightly more than fifteen years
> ago and hasn’t changed since. This logo despite having the nice concept of
> sunflower, is old. The sunflower represents the diversity, the constant
> growth and also the wildness.
>
> Among its biggest issues I can point out that it’s a bitmap picture so
> it’s unusable in large sizes (like large posters) and it’s too realistic
> making it unusable in small sizes.
>
> Most, virtually all, software products use a simpler and more abstract
> form. For example, docker, kubernetes, Ubuntu, VueJs, React, Apache Kafka,
> and many more. It’s a good time for MediaWiki to follow suit.
>
> My request is for changing the logo of MediaWiki and I have no plans or
> interest in changing logo of any other project.
>
> Please show your support, oppose or your comments in the discussion page.
> You can also add more suggestions.
>
> The discussion page:
> https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_mediawiki,_2020
>
> Best
> --
> Amir (he/him)
>
>

-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Codesearch host has moved

[Amir Sarabadani]

Hello,
Just a quick update: Since Cloud VPS tools are moving from *.wmflabs.org to
*.wmcloud.org. The codesearch has moved from https://codesearch.wmflabs.org
to https://codesearch.wmcloud.org

The old URL is now redirects to the new one.

Thank Bryan Davis for doing this! https://phabricator.wikimedia.org/T256206

Have a nice day.
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] New DB_REPLICA constant; DB_SLAVE deprecated

[Amir Sarabadani]

I'd be happy to rename it to "source". It would be also great if we clean
up puppet repo from bad terms

If people want to follow the discussion, I'd recommend checking this
ticket: https://phabricator.wikimedia.org/T254646

Best


On Thu, Jul 2, 2020, 11:30 Jaime Crespo  wrote:

> > I don't plan on doing anything to DB_MASTER, since it seems fine by
> itself,
> > like "master copy", "master tape" or "master key"
>
> MySQL has announced they chose to use "source":
> https://mysqlhighavailability.com/mysql-terminology-updates/ We can
> consider updating too, I think it is a more accurate representation of what
> those servers are (sending original data to, rather than controlling the
> replicas).
>
> On Sat, Sep 17, 2016 at 10:18 AM Amir Ladsgroup 
> wrote:
>
> > Sorry for late question. I guess we should deprecate wfWaitForSlaves()
> and
> > probably some other methods that still use this logic
> >
> > Best
> >
> > On Tue, Sep 6, 2016 at 11:22 AM Aaron Schulz 
> > wrote:
> >
> > > As of 950cf6016c, the mediawiki/core repo was updated to use DB_REPLICA
> > > instead of DB_SLAVE, with the old constant left as an alias. This is
> part
> > > of a string of commits that cleaned up the mixed use of "replica" and
> > > "slave" by sticking to the former. Extensions have not been mass
> > > converted. Please use the new constant in any new code.
> > >
> > > The word "replica" is a bit more indicative of a broader range of DB
> > > setups*, is used by a range of large companies**, and is more neutral
> in
> > > connotations.
> > >
> > > Drupal and Django made similar updates (even replacing the word
> > "master"):
> > > * https://www.drupal.org/node/2275877
> > > * https://github.com/django/django/pull/2692/files &
> > >
> > >
> >
> https://github.com/django/django/commit/beec05686ccc3bee8461f9a5a02c607a02352ae1
> > >
> > > I don't plan on doing anything to DB_MASTER, since it seems fine by
> > itself,
> > > like "master copy", "master tape" or "master key". This is analogous
> to a
> > > master RDBMs database. Even multi-master RDBMs systems tend to have a
> > > stronger consistency than classic RDBMs slave servers, and present
> > > themselves as one logical "master" or "authoritative" copy. Even in
> it's
> > > personified form, a "master" database can readily be thought of as
> > > analogous to "controller",  "governer", "ruler", lead "officer", or
> > such.**
> > >
> > > * clusters using two-phase commit, galera using certification-based
> > > replication, multi-master circular replication, ect...
> > > **
> > >
> > >
> >
> https://en.wikipedia.org/wiki/Master/slave_(technology)#Appropriateness_of_usage
> > > ***
> > >
> > >
> >
> http://www.merriam-webster.com/dictionary/master?utm_campaign=sd_medium=serp_source=jsonld
> > >
> > > --
> > > -Aaron
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Jaime Crespo
> 
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Logo of mediawiki

[Amir Sarabadani]

Hello,
The current logo of MediaWiki was adapted slightly more than fifteen years
ago and hasn’t changed since. This logo despite having the nice concept of
sunflower, is old. The sunflower represents the diversity, the constant
growth and also the wildness.

Among its biggest issues I can point out that it’s a bitmap picture so it’s
unusable in large sizes (like large posters) and it’s too realistic making
it unusable in small sizes.

Most, virtually all, software products use a simpler and more abstract
form. For example, docker, kubernetes, Ubuntu, VueJs, React, Apache Kafka,
and many more. It’s a good time for MediaWiki to follow suit.

My request is for changing the logo of MediaWiki and I have no plans or
interest in changing logo of any other project.

Please show your support, oppose or your comments in the discussion page.
You can also add more suggestions.

The discussion page:
https://www.mediawiki.org/wiki/Project:Proposal_for_changing_logo_of_mediawiki,_2020

Best
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] New Code of Conduct committee candidates

[Amir Sarabadani]

Hello again,
In the given timeframe no objection has been made, so as of today Martin
joins the main committee and Jay joins the aux members.

Join me in thanking the work of Lucie, Tpt and Rosalie and welcome Martin
and Jay to the team.

I had the pleasure of working with Lucie in the past couple of years and
her work was invaluable in solving complex issues while it hasn't been
public and barely anyone knows her contributions. Thank you Lucie.

Best

On Tue, Apr 28, 2020 at 9:27 PM Amir Sarabadani  wrote:

> Quick correction the final date is *09 June 2020* and not *09 June 2019"
> which I assume has past a while ago.
>
> Sorry
>
>
> On Tue, Apr 28, 2020 at 9:12 PM Amir Sarabadani 
> wrote:
>
>> Hello,
>> The committee has finished selecting new members and the new committee
>> candidates are (In alphabetical order):
>>
>>- Amir Sarabadani
>>- Martin Urbanec
>>- MusikAnimal
>>- Tonina Zhelyazkova
>>- Tony Thomas
>>
>> And auxiliary members will be (Also in alphabetical order):
>>
>>- Huji
>>- Jayprakash12345
>>- Matanya
>>- Nuria Ruiz
>>- Tpt
>>
>> You can read more about the members in [0]
>>
>> The changes compared to last term are:
>> * Lucie has resigned in September 2019. Tpt (as one of auxiliary members)
>> filled her seat in the meantime and now Tpt is moving to be an auxiliary
>> member again.
>> * Martin Urbanec is joining the main committee to fill Lucie/Tpt seat
>> * Jay Prakash is joining auxiliary members
>> * Rosalie is leaving the auxiliary members.
>>
>> This is not the final structure. According to the CoC [1], the current
>> committee publishes the new members and call for public feedback for *six
>> weeks* and after that, the current committee might apply changes to the
>> structure based on public feedback.
>>
>> Please let the committee know if you have any concern regarding the
>> members and its structure until *09 June 2019* and after that, the new
>> committee will be in effect and will serve for a year.
>>
>> [0]:
>> https://www.mediawiki.org/wiki/Code_of_Conduct/Committee/Members/Candidates
>> [1]:
>> https://www.mediawiki.org/wiki/Code_of_Conduct/Committee#Selection_of_new_members
>>
>> Amir, On behalf of the Code of Conduct committee
>> Best
>>
>
>
> --
> Amir (he/him)
>
>

-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Ops] Backport and Config changes window (name change)

[Amir Sarabadani]

hmm, What about B? Similar when we call "Trust and Safety" as "T"

On Tue, Jun 9, 2020 at 11:26 PM Martin Urbanec 
wrote:

> I think we need some shorter version to refer to this window. New name is
> cool, but swat is shorter.
>
> Martin
>
> On Tue, Jun 9, 2020, 5:17 PM Max Binder  wrote:
>
>> I was literally just thinking of this! Glad to hear it, for exactly the
>> reasons you said. Go team. :)
>>
>> On Mon, Jun 8, 2020 at 9:57 AM Greg Grossmeier 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> Until today if you wanted to get a backport or config change deployed to
>>> production you used what was termed the "SWAT" window.
>>>
>>> As of today that window of time is now called the "Backport and Config"
>>> window. This solves a few problems. First, the overly militarized and
>>> violence-implying name is no longer something we want in our community.
>>> Second, this now accurately and clearly describes what the purpose of the
>>> window is.
>>>
>>> This should now be changed everywhere including the Deployment
>>> Calendar[0], but if you see any errant mentions of it in documentation,
>>> please feel free to update it (WP:BOLD) or contact the Wikimedia Release
>>> Engineering Team.
>>>
>>> Thank you,
>>>
>>> Greg
>>>
>>> [0] https://wikitech.wikimedia.org/wiki/Deployments
>>>
>>>
>>> --
>>> | Greg Grossmeier  GPG: B2FA 27B1 F7EB D327 6B8E |
>>> | Dir. Engineering Productivity A18D 1138 8E47 FAC8 1C7D |
>>>
>> ___
>> Ops mailing list
>> o...@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/ops
>>
> ___
> Ops mailing list
> o...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/ops
>


-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Deprecation of a global function in mediawiki.

[Amir Sarabadani]

Hello,
The global function "wfWaitForSlaves()" that has been soft deprecated [1]
since 1.27 is now hard deprecated (from master and subsequently 1.35
release) and will be removed soon.

Please use LBFactory::waitForReplication instead.

I would like to thank Sam Reed for cleaning up all of the usages in
extensions. For more context, see https://phabricator.wikimedia.org/T181673

[1]: See Stable interface policy for the definitions:
https://www.mediawiki.org/wiki/Stable_interface_policy

Have a nice day
-- 
Amir (he/him)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l