[Wikitech-l] Re: [MediaWiki-l] Re: MediaWiki Insights - Fourth Monthly Email

[Dan Andreescu]

>
> Moriel and others have been working on mapping high level essential user
>>> workflows such as edit and patrol against platform components to explore
>>> workflow patterns and potential architectural opportunities in the
>>> platform. One outcome of this is going to be to describe the key challenges
>>> when trying to model our system. Many thanks to Moriel for leading on this
>>> work, and Daniel, Timo, Subbu, James, Cindy, Emanuele and Amir S for their
>>> support, great questions and ideas!
>>>
>>
Piling on the thanks for this work, it's super exciting and ripe with
opportunities for collaboration.  Do please make the rounds to all of us
with the outcomes and ideas.
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Comments like Google Docs

[Dan Andreescu]

+1 to Hypothesis and similar tools kind of already doing what you're
thinking of.  I would use the energy to contribute to it:
https://github.com/hypothesis/client, and it's possible that you might find
ways to optimize it for wikis via a gadget.

They've probably solved the "parent paragraph updating" problem you
mentioned here.  So at the very least you can find inspiration there.  But
I imagine the issue of storage and all that is tricky.  There was a W3C
working group that the Hypothesis people were part of, I remember we
half-heartedly wanted to join but never did.

On Tue, Dec 12, 2023 at 7:41 PM Gergő Tisza  wrote:

> If you just want to annotate a web page without assistance from that page,
> and let you share then annotations, that's not a Wikimedia-specific problem
> and there are generic tools for it. The popular one is Hypothesis
> , I think.  There even was an attempt to
> build something Wikimedia-specific on top of it (see the task Kosta
> linked), but it can be used directly as well.
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Collect telemetry (using WikimediaEvents) for a gadget

[Dan Andreescu]

+1 to statsd as a great way to get started with as little friction
as possible.  I've bubbled this up to the folks that make decisions about
instrumentation, so if you find yourself wanting to do things beyond what's
possible with statsv, please chime back in here and let us know.

On Thu, Nov 30, 2023 at 8:02 PM Krinkle  wrote:

> I would suggest using the Statsd counters that WikimediaEvents exposes to
> MediaWiki JavaScript (including Gadgets and user scripts!). This is a
> public API, with aggregate data publicly accessible via Grafana.
>
> These require no server-side configurations, schemas, or private data
> access. And (on the flipside) also do not record any personal information.
>
> To use it, call mw.track( counter.gadget_. ) in
> your gadget.
>
> For example:
>
> mw.track( 'counter.gadget_VariantAlly.storage_empty_dialog' );
>
> To make visualising easier, I've put together a generic dashboard to plot
> these:
> https://grafana.wikimedia.org/d/00037/gadget-stats
>
> --
> Timo Tijhof
> https://timotijhof.net/
>
>
>
>
> On Mon, 27 Nov 2023, at 09:33, psnbaotg via Wikitech-l wrote:
>
>
> Hi there,
>
> I'm User:Diskdance, and recently I'm developing a default gadget for
> Chinese Wikipedia enhancing MediaWiki's variant handling logic, and under
> certain circumstances a prompt is shown at page load asking for a user's
> preferred variant. Consider it as a conditional Cookie notice, and its
> English screenshot can be found at
> https://commons.wikimedia.org/wiki/File:VariantAlly-En.png.
>
> I *know *this can be very disruptive on UX, so I tend to be careful about
> its negative impact on page views. If the gadget can collect telemetry
> data about the prompt's display frequency and user interactions (using
> e.g. WikimediaEvents), I can know about its possible impact.
>
> Is this possible? It would be much appreciated if anybody could provide
> assistance.
>
> Best wishes,
> Diskdance
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Announcing Codex 1.0

[Dan Andreescu]

I am so happy.  Thank you.

On Wed, Oct 25, 2023 at 17:02 Amir Sarabadani  wrote:

> I want to echo what the DJ has said. I managed to write a pretty decent
> gadget in just a couple of hours thanks to Codex, something that used to
> take days at least. This is really exciting to see. Thank you to all who
> have worked on this tirelessly for years. Kudos and congratulations.
>
> Am Mi., 25. Okt. 2023 um 20:28 Uhr schrieb Derk-Jan Hartman <
> d.j.hartman+wmf...@gmail.com>:
>
>> Congratulations team (and the predecessor teams) !
>>
>> We started talking about needing a new way forward all the way back in
>> 2017, with an RFC started in 2019, we choose Vue in 2020 and the teams have
>> been building new foundations since, so far culminating in Codex !!
>>
>> I'm sure there is much work left to do as the web and MediaWiki is ever
>> evolving. Just recently the web finally added CSS nesting
>> 
>> for instance, and new features like :has()
>>  and :user-valid()
>> 
>> selectors. Unfortunately Wikipedia/MediaWiki and specifically user scripts
>> can have problems trying to keep up with so many changes. It is great to
>> see that es6 , less
>> , import,
>> require 
>> and recently even source maps 
>> are slowly finding their way into our ecosystem, all while supporting as
>> many older browsers as possible and building one of the fastest websites in
>> the world.
>>
>> These improvements are only possible because dedicated people work hard
>> to analyze the problems and these changes, devise solutions to slowly
>> introduce them without breaking everything, align peers, fight to get time
>> to work on it. All to shepherd them into existence, laborious and
>> challenging as it may be at times.
>>
>> Keep up the great work, but lets take a moment and celebrate the official
>> birth of an entire new design system !
>>
>> DJ
>>
>>
>>
>> On Wed, Oct 25, 2023 at 6:07 PM Roan Kattouw 
>> wrote:
>>
>>> Today the Design Systems Team
>>>  is announcing the
>>> release of Codex 1.0!
>>> What is Codex?
>>>
>>> Codex  is the new design
>>> system for Wikimedia. Over the past 2 years, the Design Systems Team and
>>> contributors from the Wikimedia Foundation, Wikimedia Deutschland, and the
>>> volunteer communities have collaborated to create a centralized design
>>> system to serve Wikimedia projects. Codex provides more equitable
>>> experiences for all Wikimedia movement participants, and makes it easier
>>> and faster to design and build consistent user interfaces. With Codex, we
>>> aim to enable more people to contribute to the mission.
>>>
>>> Codex provides a library of design tokens
>>> ,
>>> user interface components
>>> , and
>>> catalog of icons
>>>  to use
>>> with these components. Through the Codex Figma libraries, designers can
>>> reuse these shared components
>>> 
>>> , tokens
>>> ,
>>> and assets
>>> 
>>>  in
>>> their designs. For developers, Codex provides components built with Vue.js,
>>> as well as some CSS-only components that do not require JavaScript to use.
>>>
>>> Codex is already being used for Wikifunctions
>>> , Vector 2022
>>> ,
>>> the Growth Mentor Dashboard
>>>  and Impact
>>> Module ,
>>> the New Pages Feed
>>> 
>>> , MediaSearch ,
>>> NearbyPages ,
>>> QuickSurveys ,
>>> and ReadingLists .
>>> Projects currently under development using Codex include Accessibility
>>> for reading
>>>  and
>>> the 

[Wikitech-l] Re: Wiki content and other dumps new ownership, feedback requested on new version!

[Dan Andreescu]

Ariel, your dedication to dumps has always been an inspiration to me, both
in how you handled the technology and how you fused that with your care for
the ideas behind it.  Thank you, and hope we can do it justice with the
next phase of Dumps.

On Thu, Sep 28, 2023 at 12:39 AM Ariel Glenn WMF 
wrote:

> Hello folks!
>
> For some years now, I've been the main or only point of contact for the
> Wiki project sql/xml dumps semimonthly, as well as for a number of
> miscellaneous weekly datasets.
>
> This work is now passing to Data Platform Engineering (DPE), and your new
> points of contact, starting right away, will be Will Doran (email:wdoran)
> and Virginia Poundstone (email:vpoundstone). I'll still be lending a hand
> in the background for a little while but by the end of the month I'll have
> transitioned into a new role at the Wikimedia Foundation, working more
> directly on MediaWiki itself.
>
> The Data Products team, a subteam of DPE, will be managing the current
> dumps day-to-day, as well as working on a new dumps system intended to
> replace and greatly improve the current one. What formats will it produce,
> and what content, and in what bundles?  These are all great questions, and
> you have a chance to help decide on the answers. The team is gathering
> feedback right now; follow this link [
> https://docs.google.com/forms/d/e/1FAIpQLScp2KzkcTF7kE8gilCeSogzpeoVN-8yp_SY6Q47eEbuYfXzsw/viewform?usp=sf_link]
> to give your input!
>
> If you want to follow along on work being done on the new dumps system,
> you can check the phabricator workboard at
> https://phabricator.wikimedia.org/project/board/6630/ and look for items
> with the "Dumps 2.0" tag.
>
> Members of the Data Products team are already stepping up to manage the
> xmldatadumps-l mailing list, so you should not notice any changes as far as
> that goes.
>
> And as always, for dumps-related questions people on this list cannot
> answer, and which are not covered in the docs at
> https://meta.wikimedia.org/wiki/Data_dumps or
> https://wikitech.wikimedia.org/wiki/Dumps you can always email ops-dumps
> (at) wikimedia.org.
>
> See you on the wikis!
>
> Ariel Glenn
> ar...@wikimedia.org
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Word embeddings / vector search

[Dan Andreescu]

>
> On Meta there's a list of mailing lists that mentions "wikimedia-search",
> but that list seems to be dead and the archive is full of spam.
> Another list exists, called "discovery", but not listed on Meta.
> https://lists.wikimedia.org/hyperkitty/list/discov...@lists.wikimedia.org/


Indeed the Discovery

list is the one I've seen get Search-related traffic.  I wanted to update
the relevant Meta page
 but got very lost
when I saw the translate tag inside a row table template with an identifier
(397).  The list that's listed under Search and Discovery hasn't seen
actual traffic from anyone on the search team in over 7 years.  The
discovery list I linked here is the appropriate substitute, if anyone knows
how to edit that scary template thing.
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Word embeddings / vector search

[Dan Andreescu]

I encourage you to reach out to the search team, they're lovely folks and
even better engineers.

On Tue, May 9, 2023 at 1:53 PM Lars Aronsson  wrote:

> On 2023-05-09 09:27, Thiemo Kreuz wrote:
> > I'm curious what the actual question is. The basic concepts are
> > studied for about 60 years, and are in use for about 20 to 30 years.
>
> Sorry to hear that you're so negative. It's quite obvious that this is not
> currently used in Wikipedia, but is presented everywhere as a novelty
> that has not been around for 20 or 30 years.
>
> >
> https://www.elastic.co/de/blog/introducing-approximate-nearest-neighbor-search-in-elasticsearch-8-0
> > https://en.wikipedia.org/wiki/Special:Version
> >
> https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2023-2024/Draft/Product_%26_Technology#Objectives
> > https://wikitech.wikimedia.org/wiki/Search_Platform/Contact#Office_Hours
>
>
> Thanks! This answers my question. It's particularly interesting to read
> the talk page to the plan. Part of the problem is that "word embedding"
> and "vector search" are not mentioned there, but a vector search could
> have found the "ML-enabled natural language search" that is mentioned.
> If and when this is tried, we will need to evaluate how well it works for
> various languages.
>
>
> --
>Lars Aronsson (l...@aronsson.se, user:LA2)
>Linköping, Sweden
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Python requests broken by urllib3 version 2.x

[Dan Andreescu]

For Java, we run an instance of Archiva: https://archiva.wikimedia.org/

It's not a perfect approach but I think we can and should move in that
direction with all our other ecosystems (python, Javascript, PHP).  Our
reduction in security-relevant surface area alone would be worth it.

On Fri, May 5, 2023 at 11:18 AM Kosta Harlan  wrote:

> Tangent: is it worthwhile to establish a consensus for best practices with
> package pinning and package management for Python projects in the Wikimedia
> ecosystem? When I last worked on a python project (
> https://wikitech.wikimedia.org/wiki/Add_Link) I found it confusing that
> we have so many different tools and approaches for doing these things, and
> seems like we'd benefit from having a standard, supported way. (Or maybe
> that already exists and I haven't found it?)
>
> Kosta
>
> On 5. May 2023, at 13:51, Slavina Stefanova 
> wrote:
>
> Poetry is a modern lockfile-based packaging and dependency management tool
> worth looking into. It also supports exporting dependencies into a
> requirements.txt file, should you need that (nice if you want to
> containerize an app without bloating the image with Poetry, for instance).
>
> https://python-poetry.org/  
>
> --
> Slavina Stefanova (she/her)
> Software Engineer - Technical Engagement
>
> Wikimedia Foundation
>
>
> On Fri, May 5, 2023 at 1:38 PM Sebastian Berlin <
> sebastian.ber...@wikimedia.se> wrote:
>
>> A word of warning: using `pip freeze` to populate requirements.txt can
>> result in a hard to read (very long) file and other issues:
>> https://medium.com/@tomagee/pip-freeze-requirements-txt-considered-harmful-f0bce66cf895
>> .
>>
>> *Sebastian Berlin*
>> Utvecklare/*Developer*
>> Wikimedia Sverige (WMSE)
>>
>> E-post/*E-Mail*: sebastian.ber...@wikimedia.se
>> Telefon/*Phone*: (+46) 0707 - 92 03 84
>>
>>
>> On Fri, 5 May 2023 at 13:17, Amir Sarabadani  wrote:
>>
>>> You can also create an empty virtual env, install all requirements and
>>> then do
>>> pip freeze > requirements.txt
>>>
>>> That should take care of pinning
>>>
>>> Am Fr., 5. Mai 2023 um 13:11 Uhr schrieb Lucas Werkmeister <
>>> lucas.werkmeis...@wikimedia.de>:
>>>
 For the general case of Python projects, I’d argue that a better
 solution is to adopt the lockfile pattern (package-lock.json,
 composer.lock, Cargo.lock, etc.) and pin *all* dependencies, and only
 update them when the new versions have been tested and are known to work.
 pip-tools  can help with that,
 for example (requirements.in specifies “loose” dependencies;
 pip-compile creates a pinned requirements.txt; pip-sync installs it; 
 pip-compile
 -U upgrades requirements.txt later; you check both requirements.in and
 requirements.txt into version control.) But I don’t know if that
 applies in your integration/config case.

 Am Do., 4. Mai 2023 um 18:08 Uhr schrieb Antoine Musso >>> >:

> Hello,
>
> This is for python projects.
>
> Today, May 4th, urllib3 
> has released a new major version 2.0.2 which breaks the extremely popular
> requests  library.
>
> The fix is to pin urllib3<2 to prevent the new major version from
> being installed (example
> 
> ).
>
> https://phabricator.wikimedia.org/T335977
>
> Upstream issue: https://github.com/psf/requests/issues/6432
>
>
> Antoine "hashar" Musso
> Wikimedia Release Engineering
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



 --
 Lucas Werkmeister (he/er)
 Software Engineer

 Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
 Phone: +49 (0)30-577 11 62-0
 https://wikimedia.de

 Imagine a world in which every single human being can freely share in
 the sum of all knowledge. Help us to achieve our vision!
 https://spenden.wikimedia.de

 Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
 Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
 der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
 Körperschaften I Berlin, Steuernummer 27/029/42207.
 ___
 Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
 To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org

 https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>>
>>> --
>>> Amir (he/him)
>>>
>>> ___
>>> 

[Wikitech-l] Re: New flamegraphs selector (now with two year range!)

[Dan Andreescu]

This is beautiful, I love the visualization and the information
architecture.  I hope we in other teams can make some time to study this
and understand it better.

On Mon, Apr 24, 2023 at 11:47 AM Timo Tijhof  wrote:

> == *What* ==
>
> Flamegraphs provide insight to where and how your MediaWiki component or
> extension spends its time. We instrument real web requests on the MediaWiki
> app servers at WMF (using php-excimer, a low-overhead sampling profiler)
> and continually publish interactive SVG visuals by entrypoint (e.g.
> index.php, JobQueue, ResourceLoader load.php, or rest.php API). [1] [2]
>
> Read more about the Arc Lamp pipeline, introduced by Ori during the 2014
> HHVM migration [3], through our blog post:
>
> https://techblog.wikimedia.org/2021/03/03/profiling-php-in-production-at-scale/
>
> == *What's New* ==
>
> The user interface at https://performance.wikimedia.org/php-profiling/
> has undergone major changes.
>
>
>- Date: You can now choose any date in the last *2+ years*.
>Previously, this was limited to 2-3 weeks. In the dropdown menu, choose
>"Other", and select any date using the native datepicker. Existing URLs
>remain compatible.
>- Source: Real-time performance data from the *MediaWiki-on-Kubernetes*
>deployment is now available through here (excimer-k8s and
>excimer-k8s-wall). While web traffic remains largely on bare metal (except
>for test2wiki), internal MediaWiki API traffic is already partly on
>Kubernetes. Refer to T333120
> for work by SRE ServiceOps
>(e.g. CXServer uses MediaWiki-on-Kubernetes).
>- Entrypoint: *JobQueue* (RunSingleJob) is now included in the
>selector, plus the new detailed breakdowns
> for *EditAction*, *PreSend*,
>and *PostSend* are now promoted here. These were previously available
>via the file browser only.
>
>
> == *How* ==
>
> The new interface is an HTML form with vanilla JavaScript to navigate you
> to the right SVG.
> Details at https://gerrit.wikimedia.org/r/c/performance/docroot/+/908374.
>
> As before, we promote "good" defaults. You don't have to make any choices
> to start exploring the data. Press the big blue button to instantly view
> the most recent index.php flamegraph with the default settings.
>
> --
> Timo Tijhof,
> Performance Team,
> Wikimedia Foundation.
>
> [1]
> https://wikitech.wikimedia.org/wiki/Performance/Runbook/Arc_Lamp_service
> [2] https://www.mediawiki.org/wiki/Excimer
> [3]
> https://techblog.wikimedia.org/2014/12/29/how-we-made-editing-wikipedia-twice-as-fast/
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Use case for sha1 in revisions

[Dan Andreescu]

As for external use, it's very useful.  Before MW instrumented reverts, we
have ~20 years of revisions and we don't know which ones revert previous
revisions.  With the sha, we join and determine this information.  You can
see that code here
,
and the use of sha1.

On Thu, Mar 23, 2023 at 2:16 PM Gergő Tisza  wrote:

> On Wed, Mar 22, 2023 at 11:06 PM Brian Wolff  wrote:
>
>> This kind of sounds like a non-answer, but its mostly useful if you want
>> a hash of the revision. I dont think mw core really uses it, but it can be
>> useful for quickly detecting duplicate revisions. I think primarily it is
>> for external users.
>>
>
> Core does duplicate detection (when so configured) for adding the revert /
> reverted change tags.
>
> On Thu, Mar 23, 2023 at 4:03 AM Vi to  wrote:
>
>> Afair this avoids a new rev to be createad if you click "save" without
>> changing contents.
>>
>
> That actually uses to slot sha1 now (I guess as a micro-optimization
> because that way you don't need to re-hash slots which weren't edited).
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Language team celebrates 10 years of MediaWiki Language Extension Bundle

[Dan Andreescu]

Joyeux Anniversaire, MLEB!

On Wed, Feb 8, 2023 at 4:46 AM Niklas Laxström 
wrote:

> Slightly over ten years ago the Language team released the first MediaWiki
> Language Extension Bundle, or MLEB for short. To date, we have made 64
> releases supporting MediaWiki versions from 1.19 to 1.40alpha, and PHP
> versions from 5.2 to 8.
>
> Some things are still the same:
> * MediaWiki version compatibility policy
> * tarball releases
> * the goal of providing top notch language support in a convenient package
>
> Some things have changed:
> * we went from monthly releases to quarterly releases
> * LocalizationUpdate is no longer included in the bundle because it was
> made obsolete by automatic translation backports
>
> We don't know much about the users of the MLEB, but according to our
> statistics, each tarball release is downloaded on average approximately 300
> times. Some people use MLEB releases via git.
>
> As the initiator of MLEB, I am happy to see how far we have come. Thanks
> belong to Abijeet Patro and Kartik Mistry who together have done the actual
> work to make the releases in the past years.
>
> Fun fact: the tool used to make MLEB releases is called melange. I don't
> remember for sure, but I think the name was chosen because it's short for
> "MEdiawiki LANGuagE'' and it had a quite appropriate meaning: "A mixture of
> different things; a disordered mixture" (English Wiktionary).
>
> The first release announcement is quoted below.
>
>   -Niklas
>
> ke 28. marrask. 2012 klo 12.17 Niklas Laxström (niklas.laxst...@gmail.com)
> kirjoitti:
>
>> The Wikimedia Language Engineering team is pleased to announce the
>> first release of the MediaWiki Language Extension Bundle. The bundle
>> is a collection of selected MediaWiki extensions needed by any wiki
>> which desires to be multilingual.
>>
>> This first bundle release (2012.11) is compatible with MediaWiki 1.19,
>> 1.20 and 1.21alpha.
>> Get it from https://www.mediawiki.org/wiki/MLEB
>>
>> The Universal Language Selector is a must have, because it provides an
>> essential functionality for any user regardless on the number of
>> languages he/she speaks: language selection, font support for
>> displaying scripts badly supported by operating systems and input
>> methods for typing languages that don't use Latin (a-z) alphabet.
>>
>> Maintaining multilingual content in a wiki is a mess without the
>> Translate extension, which is used by Wikimedia, KDE and
>> translatewiki.net, where hundreds of pieces of documentation and
>> interface translations are updated every day; with Localisation Update
>> your users will always have the latest translations freshly out of the
>> oven. The Clean Changes extension keeps your recent changes page
>> uncluttered from translation activity and other distractions.
>>
>> Don't miss the chance to practice your rusty language skills and use
>> the Babel extension to mark the languages you speak and to find other
>> speakers of the same language in your wiki. And finally the cldr
>> extension is a database of language and country translations.
>>
>> We are aiming to make new releases every month, so that you can easily
>> stay on the cutting edge with the constantly improving language
>> support. The bundle comes with clear installation and upgrade
>> installations. The bundle is tested against MediaWiki release
>> versions, so you can avoid most of the temporary breaks that would
>> happen if you were using the latest development versions instead.
>>
>> Because this is our first release, there can be some rough edges.
>> Please provide us a lot of feedback so that we can improve for the
>> next release.
>>
>>   -Niklas
>>
>> --
>> Niklas Laxström
>>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Reducing size of pageviews dump (shared link on the article)

[Dan Andreescu]

@Dušan Kreheľ: I think there's a misunderstanding.  I read your re-written
article.  In it, you say that the current format is:

domain_code page_title count_views total_response_size

For an example, you give this:

sk Kreheľ 2 0

But, actually, that format is deprecated and the new format is pageviews
complete, which looks like this:

sk.wikipedia Kreheľ null desktop 13 B2D2G2J2O2T1V1X1

The B2D2G2J2O2T1V1X1 is exactly the kind of encoding you're talking about,
and no 0-values are present.

You made the point that we are missing a yearly rollup in this new format.
This would be quite a large file, but if there's a good use case for such a
dump, a request in phabricator is a good way to proceed.

On Sat, Oct 1, 2022 at 9:58 AM Dušan Kreheľ  wrote:

> The big update of the article is done. Please, You look.
>
> Gergő Tisza: The current fresh hour format can remain. Later it can be
> converted to another format. And thus be more suitable for others.
>
> 2022-09-18 22:35 GMT+02:00, Dušan Kreheľ :
> > I have updated the document. I added the export of human pageviews for
> > year 2021. The statistics are in the article. A download link has been
> > added.
> >
> > Dan Andreescu: None problem was to understand You.
> >
> > 2022-09-05 21:48 GMT+02:00, Dan Andreescu :
> >> Hi Dušan,
> >>
> >> I added the details on pageviews_complete to the talk page on your
> >> proposal
> >> <
> https://en.wikipedia.org/w/index.php?title=User_talk:Du%C5%A1an_Krehe%C4%BE/Signpost_draft:New_pageview_dump_export_format_(concept)=1108690384
> >.
> >> Please let me know if it's still confusing.
> >>
> >
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Reducing size of pageviews dump (shared link on the article)

[Dan Andreescu]

Hi Dušan,

I added the details on pageviews_complete to the talk page on your proposal
.
Please let me know if it's still confusing.
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Reducing size of pageviews dump (shared link on the article)

[Dan Andreescu]

Our pageview dumps were in the middle of a refactor when our team changed a
lot.  We haven't been able to finish it, but we do actually have a
well-compressed version that we just haven't properly launched as a new
dataset.  I'm working on prioritizing that.

On Sun, Sep 4, 2022 at 02:58 Gergő Tisza  wrote:

> I'd imagine the current format is optimized for being able to output
> hourly dumps (and thus reducing data latency and data processing costs),
> not so much for storage space
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: A random thank you to the Wikimedia tech community

[Dan Andreescu]

I feel like... I have to answer this... because otherwise our statistics
would suffer :P

I'm doubling down on the love, *thank you everyone, and <3*.  When I was
new this list was incredibly valuable to me.  Ten years passed and I am
even more new, since every day I find another area I never knew existed, so
this list is even more valuable.

On Tue, Aug 30, 2022 at 4:17 PM Strainu  wrote:

> Hi all,
>
> With the risk of being off-topic, I want to express my gratitude to
> all the members of the Wikimedia tech community for being such a
> supportive and helpful group! Not only here, but on all communication
> channels.
>
> It's been years since one of my questions (most of which could be
> classified as obscure) has gone unanswered. Also, I recently had to go
> through all my emails since June and I noticed that except for a few
> announcements and obvious spam, all other other threads had at least
> one answer. For me, this is the sign of a great community to be in.
>
> Thanks again and keep up the good work!
>Strainu
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Commons app - v4.0 and my resignation

[Dan Andreescu]

Thanks so much for your leadership and work Josephine, it's been awesome to
watch over the years.  Good luck on your next adventures!

On Thu, Aug 25, 2022 at 2:17 AM Josephine Lim 
wrote:

> Hi all,
>
> Hope you have been well! We have just released v4.0 of the Commons Android
> app to production on Google Play - this version includes tons of new
> features (a map displaying nearby Commons pictures, custom SPARQL queries,
> user profiles, and a custom picture selector), as well as fixes for major
> bugs such as the "date uploaded" bug. The app can be downloaded from the
> Play Store[1] or directly from our GitHub repository[2].
>
> I should probably also add that this will likely be my last release - I am
> currently transitioning to a new career, and will be stepping down from the
> role of project lead, which I have held for the past 6 years. I intend to
> stick around on a volunteer basis, but the amount of time that I can put
> into the app from now on will be much more limited. I have discussed this
> with the app's community, and I believe that a new grant team might be
> emerging, and one of the volunteers might step up to take on the project
> lead role.
>
> Thank you very much for the support that you have all shown to the Commons
> app over the years. It was an honor and a pleasure to work with all of you
> - I have learned so much and met so many amazing people in the Wikimedia
> community, and I certainly hope to continue being part of it. I also hope
> that the app and the community surrounding it may have benefited from my
> work here, and that the new team will be able to bring it to places that I
> might not have ever foreseen.
>
> Best regards,
> Josephine (User: Misaochan)
>
> [1] https://play.google.com/store/apps/details?id=fr.free.nrw.commons
> [2] https://github.com/commons-app/apps-android-commons/releases
>
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Abstract Schema: Milestone reached for WMF Deployed Extensions

[Dan Andreescu]

Thank you so much for this work.  There are additional benefits as we all
have a clearer understanding of how data moves around our various systems.
My team (Data Engineering) is standing up a data catalog and we'll see
about ingesting these schemas directly to centralize the information.

On Thu, Jul 14, 2022 at 17:00 Der Umherirrende 
wrote:

> Hello,
>
> read previous update news about the abstract schema on the mediawiki-l
> list under <
> https://lists.wikimedia.org/hyperkitty/list/mediawik...@lists.wikimedia.org/thread/LLP6GJEGXJH2S62MU2UXXL4XVVPTYRMJ/
> >
>
> The migration of mediawiki-core from sql schema file to the abstract
> schema finished in May 2021, while the migration of WMF Deployed Extensions
> is still ongoing on the goal task T261912[1] where 38 extensions with own
> database tables are listed.
>
> But I am happy to write:
> The migration of all WMF Deployed Extensions is now finished!
>
> At the end there are 35 extensions affected and all now shipes a json file
> with the schema defintion and the sql file is generated from the json file.
> The most patch sets were merged this week and will be released with
> REL1_39.
>
> To be able to provide the abstract schema file some extensions needs
> updates beforehand, like standardise of timestamp columns or remove of
> foreign keys.
> The new json files makes it easier to analyse the schemas now and find
> more optimisation with automation, like duplicate indexes or missing
> primary keys.
>
> A benefit of the abstract schema is also to provide a postgres and sqlite
> schema for free for all these extensions.
> It also easier for schema changes in the future to keep the schema file in
> sync and hopefully gives a better support for third party users with
> postgres or sqlite.
>
> [Disclaimer: Do not expect that the extension will work directly with
> postgres or sqlite. There are some RDBMS differences needs to handle in the
> code, like different timestamp formats on postgres, but feel free to patch
> the extension or fill bugs if you find issues, where the differences are
> not addressed in the code.]
>
> There are still tasks to fix for better integration of the abstract
> schema, like testing in CI if the schema file and the generated sql files
> are in sync to avoid issues on partial updates[2]. This is only done for
> core at the moment, please take a deeper look, when reviewing such patches
> on extensions, thanks.
> Knowing that all extensions have one of the three supported rdbms schema
> makes it easier to enable voting CI jobs on merge for postgres and sqlite
> in the future, but that needs fixing of tests first. Help is welcome here.
>
> Looking ahead:
> In the future the abstract schema of the extension could also be used to
> detect schema drifts on extension tables on wmf wikis, this happens at the
> moment only for core[3].
>
> Hopefully some of the long standing requests to bundle popular extensions
> with the tarball can be addressed[4], because the extensions are now
> fulfill the requirement to support all rdbms. Testing is welcome for this
> as well!
>
> It is possible that there are some areas needs updates, like documentation
> or guidelines, feel free to improve if you find something outdated.
>
>
> I would like to thanks Sam Reed to start with the convert of extensions,
> as well as various other developer converting extensions in the last
> years/months! Great work I could benefit from while creating my own patch
> sets.
> Also thanks to James Forrester for the review of many of my patch sets.
>
> Greetings,
> Der Umherirrende
>
> [1] https://phabricator.wikimedia.org/T261912
> [2] https://phabricator.wikimedia.org/T261057
> [3] https://drift-tracker.toolforge.org/
> [4] https://phabricator.wikimedia.org/T305072
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: How to get Top 1000 contributors list?

[Dan Andreescu]

The top 100 editors
 is
an official Wikistats metric.  It's available in the web interface
 or
via API call
.
You can break it down by the type of editor, the type of page being edited,
or both.  There are some nuances here that you wouldn't be able to get from
the other interfaces.  For example, we track whether the editor was a bot
at the time they were making the edit, as opposed to at present time.  That
sometimes makes a big difference as some accounts toggle the bot flag.

If the top 1000 is more useful, please file a feature request (the link for
this  can be
found in the Wikistats web interface footer).

On Sat, Jun 25, 2022 at 4:09 PM Ivo Kruusamägi 
wrote:

> There is also https://ta.wikiscan.org/users
>
> Ivo Kruusamägi
>
> Kontakt John () kirjutas kuupäeval L, 25.
> juuni 2022 kell 19:24:
>
>> Depends on what you call the top editors. Is that edit count, articles
>> created, total bytes of content added etc.
>>
>> On Sat, Jun 25, 2022 at 12:20 PM Roy Smith  wrote:
>>
>>> This should do it:
>>>
>>> https://quarry.wmcloud.org/query/65641
>>>
>>> In general, this is an inefficient query because user_editcount isn't
>>> indexed.  Fortunately, tawiki has a relatively small number of users, so it
>>> works.  Running the same query on a much larger wiki, say en, would
>>> probably time out.
>>>
>>>
>>> On Jun 25, 2022, at 12:07 PM, Shrinivasan T 
>>> wrote:
>>>
>>> Hello all,
>>>
>>> I  like to get top 1000 contributors for ta.wikipedia.org based on
>>> their usercontribution metric.
>>>
>>> is there any code this or api?
>>>
>>> Please share.
>>>
>>> Thanks.
>>>
>>> --
>>> Regards,
>>> T.Shrinivasan
>>>
>>>
>>> My Life with GNU/Linux : http://goinggnu.wordpress.com
>>> Free E-Magazine on Free Open Source Software in Tamil :
>>> http://kaniyam.com
>>>
>>> Get Free Tamil Ebooks for Android, iOS, Kindle, Computer :
>>> http://FreeTamilEbooks.com 
>>> ___
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>> ___
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> ___
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Stagnant quality control process for [[mw:API:Client_code]]

[Dan Andreescu]

+1

On Thu, May 26, 2022 at 5:52 AM Thiemo Kreuz 
wrote:

> I'm curious. How was this evaluation process originally meant to work?
> Which group did it, how often, and based on which criteria? I probably
> missed it, but couldn't find this information on the page.
>
> From all I see at the moment I would say: Go for it and merge the pages.
>
> Best
> Thiemo
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re:  Amir Sarabadani receives Web Perf Hero award!

[Dan Andreescu]

Congratulations!  So ninja, love it.

On Thu, May 26, 2022 at 10:22 AM Krinkle  wrote:

> I'm happy to share that the first Web Perf Hero award of 2022 goes to Amir
> Sarabadani!
>
> This award is in recognition of Amir's work (@Ladsgroup) over the past six
> months, in which he demonstrated deep expertise of the MediaWiki platform
> and significantly sped up the process of saving edits in MediaWiki. This
> improved both the potential of MediaWiki core, and as experienced
> concretely on WMF wikis, especially on Wikidata.org.
>
> Refer to the below medawiki.org page to read about what it took to *cut
> latencies by half*:
>
> https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Web_Perf_Hero_award#Amir_Sarabadani
>
> This award is given on a quarterly basis, and manifests as a Phabricator
> badge:
> https://phabricator.wikimedia.org/badges/view/17/
>
> -- Timo Tijhof, on behalf of WMF Performance Team.
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Usecases for Low-Code Platforms within the Wikimedia Projects

[Dan Andreescu]

I think there are many possible applications, here are two that sound
interesting to me (but my opinion really doesn't and shouldn't count):

* Abstract Wikipedia wiki functions
: Snap!
seems like an easy way for more people to get involved
* Lua templates : generating
the lua code could open this up to more people

On Mon, May 23, 2022 at 2:43 PM  wrote:

> At the Wikimedia Hackathon 2022 that ended yesterday I have showed a
> program in the Showcase that can convert blocks from
> visual-programming-language Snap! to source code. This is the link to the
> folder where the program is located in.
> https://public.paws.wmcloud.org/User:Hog%C3%BC-456/BlocktoCode/ The
> program reads an XML-File with the definition of a program and gives the
> source code as an output. The platform for creating the blocks I used is
> called Snap!. This is an further development of Scratch. Scratch is an
> visual programming language based on blocks, that can be combined to create
> a program. A block is a small sentence with gaps for the variables. In
> Snap! it is possible to create own blocks and it includes an feature to
> directly convert blocks to code.  I dont know how far this is developed and
> after I havent understand how to export the result with the code I have
> written a own program to do that.
>
> What do you think are potential use cases for low code platforms like
> Snap! within the Wikimedia Projects. From my point of view such platforms
> offer a chance to make programming accessible to more people. It is from my
> point of view easier with such a platform to write small programs as
> without such an support. I am interested in use cases where the built-in
> codification feature or my program can be used to generate code that will
> be then useful within the Wikimedia projects.
>
> Have a nice day and I am interested in your thoughts.
> Hogü-456
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: [Wikitech-ambassadors] Wikimedia Hackathon: Call for Sessions

[Dan Andreescu]

>
> On Thursday, April 28, 2022, Roy Smith  wrote:
>
>> The last time I tried to set a phab ticket's priority, I was basically
>> told that volunteers shouldn't be doing that, i.e. changing priorities of
>> phab tickets was reserved to WMF staff.  Some clarity around this would be
>> appreciated :-)
>>
>
Brian explains this well, I'd just like to add that it's still very useful
to know if you think something is a high priority.  Instead of setting the
priority of the task, you can comment and explain a bit, this kind of
context is always welcome, especially for newly hired WMF staff.
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Removal of the mw.eventLog.Schema class without deprecation

[Dan Andreescu]

On Thu, Apr 14, 2022 at 12:07 PM David Lynch  wrote:

> This is still used in some MobileFrontend and DiscussionTools code. The
> codesearch would be better done as
> https://codesearch.wmcloud.org/search/?q=mw.eventLog.Schema=nope=js==
>

ah! I missed the open paren ( in the original search, good eye!
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Different cache invalidation rules for similar users?

[Dan Andreescu]

>
> No, the issue is that the users would disable the gadget and it would
> still be enabled after 12h.
>

Were they enabling it by putting directly into User:.../common.js or
through Special:Preferences?  The former might be subject to weird caching
but I always assumed it was properly invalidated on changes.  If using
Special:Preferences, that sounds like a more serious bug and it would be
good to know.
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

[Wikitech-l] Re: Production Excellence #38: November 2021

[Dan Andreescu]

I just want to say thank you so much for these emails, they're great on
their own, but together they paint a clear picture at a level usually
inaccessible for those of us outside everyday mw development.  Thank you!

On Sat, Dec 11, 2021 at 20:39 Krinkle  wrote:

> How’d we do in our strive for operational excellence last month? Read on
> to find out!
> Incidents
>
> 6 documented incidents last month. That's above the two-year and five-year
> median of 4 per month (per Incident graphs
> ).
>
> 2021-11-04 large file upload timeouts
> ;
> Impact: For 9 months, editors were unable to upload large files (e.g. to
> Commons). Editors would receive generic error messages, typically after a
> timeout. In retrospect, a dozen different distinct production errors had
> been reported and regularly observed that were related and provided
> different clues, however most of these remained untriaged and
> uninvestigated for months. This may be related to the affected components
> having no active code steward.
>
> 2021-11-05 TOC language converter
> ;
> Impact: For 6 hours, wikis experienced a blank or missing table of contents
> on many pages. For up to 3 days prior, wikis that have multiple language
> variants (such as Chinese Wikipedia) displayed the table of contents in an
> incorrect or inconsistent language variant (which are not understandable to
> some readers).
>
> 2021-11-10 cirrussearch commonsfile outage
> ;
> Impact: For ~2.5 hours, the Search results page was unavailable on many
> wikis (except English Wikipedia). On Wikimedia Commons the search
> suggestions feature was unresponsive as well.
>
> 2021-11-18 codfw ipv6 network
> ;
> Impact: For 8 minutes, the Codfw cluster experienced partial loss of IPv6
> connectivity for upload.wikimedia.org. This did not affect availability
> of the service because the "Happy Eyeballs
> " algorithm ensures
> browsers (and other clients) automatically fallback to IPv4. The Codfw
> cluster generally serves Mexico and parts of the US and Canada. The
> upload.wikimedia.org service serves photos and other media/document
> files, such as displayed in Wikipedia articles.
>
> 2021-11-23 core network routing
> ;
> Impact: For about 12 minutes, Eqiad was unable to reach hosts in other data
> centers via public IP addresses. This was due to a BGP routing error. There
> was no impact on end-user traffic, and impact on internal traffic was
> limited (only Icinga alerts themselves) because internal traffic generally
> uses local IP subnets which we currently route with OSPF instead of BGP.
>
> 2021-11-25 eventgate-main outage
> ;
> Impact: For about 3 minutes, eventgate-main was down. This resulted in
> 25,000 MediaWiki backend errors due to inability to queue new jobs. About
> 1000 user-facing web requests failed (HTTP 500 Error). Event production
> briefly dropped from ~3000 per second to 0 per second.
> Incident follow-up
>
> Remember to review and schedule Incident Follow-up work
>  in Phabricator,
> which are preventive measures and tech debt mitigations written down after
> an incident is concluded. Read more about past incidents at Incident
> status  on Wikitech.
>
> Recently resolved incident follow-up:
>
> Disable DPL on wikis that aren't using it
> 
> Filed after a July 2021 incident, done by Amir (Ladsgroup) and Kunal
> (Legoktm).
>
> Create easy access to MySQL ports for faster incident response and
> maintenance 
> Filed in Sep 2021, and carried out by Stevie (Kormat).
>
> Create paging alert for primary DB hosts
> 
> Filed after a Sept 2019 incident, done by Stevie (Kormat).
>
> Trends
>
> November saw 27 new production error reports of which 14 were resolved,
> and 13 remain open and carry over to the next month.
>
> Of the 301 errors still open from previous months, 16 were resolved.
> Together with the 13 carried over from November that brings the workboard
> to 298 unresolved tasks.
> Figure 1: Unresolved error reports by month
> 
> .
>
>
> Outstanding errors
>
> Take a look at the workboard 

[Wikitech-l] Re: Upstream shutdowns

[Dan Andreescu]

I suspect there's much more than just this list, even if we just restrict
it to "major" components.  On Analytics, we've had to adjust as the
following systems EOL-ed or changed licensing:

* Pivot (moved to Turnilo, a fork)
* Camus (moving to Gobblin)
* Kafka Connect

On Wed, Jun 2, 2021 at 5:16 AM Antoine Musso  wrote:

> Le 02/06/2021 à 03:35, K. Peachey a écrit :
>
> There is also one of the search systems migrating to non O/S from
> memory... elastic?
>
> Hello,
>
> Yes that is ElasticSearch. They have recently announced they are now
> licensing code under Server Side Public License (SSPL)
>  it is a viral
> license which in short requires that any service based on such code get
> licensed with the same license.  I am not a lawyer, but I guess that would
> imply that our whole stack switch to it as well.  As such it is not
> considered free by Debian or the Open Source Initiative (OSI).
>
> As I understand it, the move has been made possible since all
> contributions were subject to a Contributor License Agreement (CLA)
> . So that
> although the code was placed under a free license, the CLA effectively
> grants unrestricted publishing right to an organization, they can thus
> relicense the code however they want.   If I get it right, the old code is
> still under a free license but it can also be used under the new non free
> license.As I get it the move has been done due to frictions with Amazon
> which is providing a search service.   Amazon announced they would be
> behind the free community fork: https://www.opensearch.org/
>
>
> Antoine "hashar" Musso
>
>
> ___
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
___
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

Re: [Wikitech-l] Performance regression expected with Debian Buster upgrade

[Dan Andreescu]

These are amazing, thanks for sharing.  /me bookmarks patches for bedtime
reading

On Fri, Feb 12, 2021 at 04:25 Kunal Mehta  wrote:

> Hi,
>
> On 2/3/21 5:35 PM, Kunal Mehta wrote:
> > What now? We're going to continue with the upgrade as planned, but we
> > also need help to try and make some performance improvements to reduce
> > the impact of the regression.
>
> A week later I'd like to highlight and recognize some of the performance
> improvements that have been made:
>
> * Upgrading utfnormal to use native mbstring functions instead of PHP
> implementations  (MaxSem,
> James F, Reedy and myself)
> * Optimizations to ApiResult
> 
> (Daimona, Thiemo, Krinkle and James F)
> * Using PCRE for faster UTF-8 validation in Parsoid
>  (Skizzerz and cscott)
> * Reducing the size of the ExtensionRegistry cache in APCU
> <
> https://gerrit.wikimedia.org/r/q/hashtag:%2522smaller-extension-cache%2522>
>
> (Krinkle and myself)
> * Reduce impact of HookContainer loading 500+ interfaces
>  (Skizzerz, myself, Tim
> Starling and Ori)
>
> If I missed any other improvements people have been working on, my
> apologies, please share them! I've been using the Gerrit hashtag
> "faster-mw-plz" 
> to try and track these.
>
> -- Kunal
>
> P.S. reimaging to Buster is 70% complete now.
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] The future of UploadWizard

[Dan Andreescu]

It's also worth mentioning that WMF is early in the process of adopting
vue.  Hopefully we do this in a sane way, porting the good things from
OOJS-UI into a centralized design system with a comprehensive component
library.  I mention it because you may want to keep an eye on that progress
and see how your plans here might line up.  I think this is the place to
look: https://www.mediawiki.org/wiki/Vue.js/Migration_Project_Charter

I'll also add that you're awesome and the list of fixes you mention looks
amazing :)

On Thu, Feb 4, 2021 at 5:00 PM Strainu  wrote:

> În joi, 4 feb. 2021 la 21:31, Ostrzyciel Nożyczek
>  a scris:
> > The things that I have on mind are:
> >
> > Rework config handling to make it more consistent (now only campaign
> configs are parsed, the main config is not) and robust (unit testing
> included!).
> > Simplify the task of including more licenses in UW (message loading
> based on config), add more built-in icons to make that even simpler for
> site admins.
> > Change the tutorial from an image to wikitext, which should be much
> easier to edit.
> > Restructure documentation to be third-party centric, maybe make a brief
> configuration guide (configuring UW now requires one to carefully study a
> not-so-friendly PHP file).
> > Add a few quick hacks to make the UI responsive, at least to some degree
> (that is very much possible with just CSS). The solution can be polished
> later.
> > Remove Wikibase-related code and other Wikimedia-specific stuff that
> will just make testing harder.
> > Improve configurability for all fields in the wizard, ensure sensible
> default settings.
> > Add an option to use single-language fields. Multi-language fields are
> unnecessary on most wikis.
> > Look into how different stages of UW could be streamlined / improved to
> make the upload process faster, especially on wikis requiring less detailed
> information.
> > Make all kinds of file description syntax configurable.
> > (Maybe) prepare and package a few ready-to-use configuration sets,
> together with the templates necessary to make it work. That would really
> simplify the process of bringing UW to a wiki.
>
> Just a quick note to say that out of the 11 items you list above, 8
> would also improve the Wikimedia experience :)
>
> Strainu
>
> >
> > ...and more! This may be a bit ambitious, but I think it's doable with
> just a few people interested in the project and some spare time. I am
> certainly on board. :P
> >
> >
> > --
> > Ostrzyciel (he/him)
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Is preprocessDump.php still useful?

[Dan Andreescu]

As far as Analytics / Statistics are concerned, this is just an interesting
artifact.  The problems we have to solve while reading and processing XML
dumps are quite different (custom file formats to handle files bigger than
a single HDFS block, etc).  Safe to delete in my opinion, the less code the
better.

On Wed, Jan 20, 2021 at 8:40 PM Krinkle  wrote:

> This script was created in 2011 and takes an offline XML dump file,
> containing page content wikitext, and feeds its entries through the
> Preprocessor without actually importing any content into the wiki.
>
> The documented purpose of the script is to "get statistics" or "fill the
> cache". I was unable to find any stats being emitted. I did find that the
> method called indeed fills "preprocess-hash" cache keys, which have a TTL
> of 24 hours (e.g. Memcached).
>
> I could not think of a use case for this and am wondering if anyone
> remembers its original purpose and/or knows of a current need for it.
>
> -- Timo
>
> [1] First commit: http://mediawiki.org/wiki/Special:Code/MediaWiki/80466
> [2] Commit history:
> https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+log/1.35.0/maintenance/preprocessDump.php
>
> PS: This came up because there's a minor refactor proposed to the script,
> and I was wondering how to test it and whether we it makes sense to
> continue maintenance and support for it.
> https://gerrit.wikimedia.org/r/641323
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] TechCom meeting 2020-12-02

[Dan Andreescu]

>
>
>- Create WikiTeq group on Gerrit
> is now more clear (see note
>from Daniel  to
>discuss, which we should do here)
>
> Committee board activity:
>
> The process for adding a "trusted organization" isn't clear, but since the
> group is not maintaining any extensions deployed by wmf, we can probably
> just go ahead and add them.
>
+1

>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] TechCom meeting 2020-12-02

[Dan Andreescu]

This is the weekly TechCom board review. If there are additional topics for
TechCom to review, please let us know by replying to this email. However,
please keep discussion about individual RFCs to the Phabricator tickets.
This week, the meeting is async, so we'll be replying to the email as
well, *feel
free to join in*!

Activity since Wednesday 2020-11-25 on the following boards:

https://phabricator.wikimedia.org/tag/techcom/
https://phabricator.wikimedia.org/tag/techcom-rfc/

Committee inbox:

   - Automatically index extensions in Codesearch
    is still in the inbox, no
   activity since last week's discussion
   - Create WikiTeq group on Gerrit
    is now more clear (see note
   from Daniel  to
   discuss, which we should do here)

Committee board activity:

   - General ParserCache service class for large "current" page-derived data
    was declined by Daniel, see
   his reasoning there

New RFCs: (none)

Phase progression: (none)

IRC meeting request: (none)

Other RFC activity:

   - RFC: Re-evaluate librsvg as SVG renderer for WMF wikis
    saw more conversation
   - RFC: Discourage use of MySQL's ENUM type
    Amir points out the need to
   overhaul all db documentation (I agree, could we have a doc sprint about
   this?) including policies
   - RFC: Amendment to the Stable interface policy, November 2020
    has a suggestion/question
   about clarifying default status and deprecation procedure
   - RFC: Store WikibaseQualityConstraint check data in persistent storage
    WMDE calls for help from
   TechCom and the Platform team to collaborate here.  See Adam Shorland's
   comment about putting it through the new Technical Decision Making Process
   - RFC: Expand API title generator to support other generated data
    Carly Bogen is asking for
   clarity on the need for a steward.  Timo's comment
    on the task indeed
   seem to contradict his note from last week's grooming email.  Timo, would
   you please clarify.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] TechCom meeting 2020-11-25

[Dan Andreescu]

Quick reminder that the IRC meeting is starting in 10 minutes.  We will
collaborate on:

RFC: Provide mechanism for configuration sets for development and tests
https://phabricator.wikimedia.org/T267928

On Wed, Nov 25, 2020 at 4:00 PM Krinkle  wrote:

> This is the weekly TechCom board review in preparation of our meeting on
> Wednesday. If there are additional topics for TechCom to review, please let
> us know by replying to this email. However, please keep discussion about
> individual RFCs to the Phabricator tickets.
>
> Activity since Monday 2020-11-02 on the following boards:
>
> https://phabricator.wikimedia.org/tag/techcom/
> https://phabricator.wikimedia.org/tag/techcom-rfc/
>
> Committee inbox:
>
>- T268328: Automatically index extensions in Codesearch
>
>   - Daniel is raising that people effectively use Codesearch to guide
>   deprecation efforts under theStable Interface policy. As such, we should
>   define what inclusion criteria it has (or should have), and simply or
>   document how to implement that in practice through adding and removing
>   repositories from its index (esp those not hosted by Wikimedia).
>- T267085 : Clarify
>deprecation of method overrides
>   - A question about the stable interface policy.
>
> Committee board activity:
>
>- T175745 : Do not
>overwrite edits when conflicting with self
>   - Some renewed interest on this question about how MW should handle
>   when e.g. someone starts editing the same page from multiple tabs and 
> then
>   submits those edits.
>- T227776 : General
>ParserCache service class
>   - Addshore asking for an update.
>
> New RFCs:
>
>- T268326: RFC: Amendment to the Stable interface policy (Nov 2020)
>
>   - Proposal by Daniel, to:
>  - … fill some gaps (e.g. traits, and member fields).
>  - … allow for removal without (released) deprecation if it is
>  unused in code we know about and is considered "maintained". Input 
> welcome.
>
> Phase progression:
>
>- T266866 RFC : Bump Basic
>browser support to require TLS 1.2 for MediaWiki core
>   - Ed lists which Web APIs and other browser capabilities would
>   become safe to use in the base layer (HTML/CSS), as well as some JS
>   features that will automatically become available to Grade A.
>   - Ed confirmed TLS 1.2 mapping to browser names/versions.
>   - Moved to Phase 3: Explore.
>- T260330 RFC: PHP microservice for containerized shell
>
>   - Moved to Last Call last week, until 2 December (next week).
>   - Tim answered and added a section to clarify the backwards
>   compatible nature of the PHP interface in core, for third-parties that
>   would not or have not installed Shellbox.
>- T259771: RFC: Drop support for database upgrade older than two LTS
>
>   - Last week's concerns about detection and failure prevention have
>   been answered by Amir.
>   - The Platform Engineering Team has filled the ownership gap for
>   this policy.
>   - Moved to Phase 4: Tune.
>
> IRC meeting request:
>
>
>- Later today (Wed 25 Nov), this RFC will be discussed in
>#wikimedia-office on Freenode IRC:
>RFC: Provide mechanism for configuration sets for development and tests
>https://phabricator.wikimedia.org/T267928
>
>
> Other RFC activity:
>
>- T263841 RFC : Expand API
>title generator to support other generated data
>   - Rescoped from potential software change to policy update.
>   - Awaiting resourcing from core API steward to confirm support,
>   risk, compatibility as proposed.
>- T250406 RFC: Hybrid extension management
>
>   - Conversation about what we would need to commit to for WMF
>   software, and seeking placing and approval of said resourcing.
>- T119173: RFC: Discourage use of MySQL ENUM type
>
>   - Next step is for the consensus to be turned into concrete wording
>   for the policy.
>- T40010: RFC: Re-evaluate librsvg as SVG renderer for WMF wikis
>
>   - Some general clarifications, and statistics from production.
>
>
> -- Timo
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org

Re: [Wikitech-l] TechCom topics 2020-11-04 (fixed)

[Dan Andreescu]

>
> Maybe something exists already in Hadoop
>

The page properties table is already loaded into Hadoop on a monthly basis
(wmf_raw.mediawiki_page_props).  I haven't played with it much, but Hive
also has JSON-parsing goodies, so give it a shot and let me know if you get
stuck.  In general, data from the databases can be sqooped into Hadoop.  We
do this for large pipelines like edit history

and
it's very easy

to add a table.  We're looking at just replicating the whole db on a more
frequent basis, but we have to do some groundwork first to allow
incremental updates (see Apache Iceberg if you're interested).
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Image search

[Dan Andreescu]

I was using the new search interface that Eric mentioned above and I was
able to find a few exterior corner lamps indeed.  So I'm wondering if you
tried it and if you have any feedback:

https://commons.wikimedia.org/wiki/Special:MediaSearch?type=bitmap=exterior+lantern+Corner

On Tue, Nov 17, 2020 at 12:47 PM Lars Aronsson  wrote:

> On 2020-11-16 17:08, Roy Smith wrote:
> > Perhaps not exactly what was asked, but Google Images has the ability.
> >  Go to images.google.com , click the camera
> > icon, and then you can paste a URL for the image.
>
>
> This is perhaps the best suggestion so far. But it is limited.
>
> Here I can input images from e-retail websites,
> screenshots from Google Streetview, or just any
> copyrighted photos and find similar photos.
> If I add site:wikimedia.org to that search, I will find
> free images on Wikimedia Commons. Good!
>
> But the search is not so very useful. If I submit a
> photo where much of the building is visible, I get
> images of buildings in general, with or without lamps.
> If I submit a photo where much blue sky background
> is visible, I get light-blue images in general, with
> or without buildings or lamps. There is a rough
> visual similarity to the images, but not on a detailed
> level and not by semantic content.
>
> The search does not allow me, as I would wish, to refine
> the search by indicating which images are better or
> worse for my search.
>
> It would be interesting to know what kind of data model
> is behind the similarity function. My guess is that it
> needs to be 100 to 1000 times larger.
>
> Both in OCR (optical character recognition) and in
> image similarity search, we now have so much data
> that we should be able to organize a huge databank
> of recognition data based on crowdsourcing. We'd
> need some game, where millions of users answer
> 1. What does this image depict?
> 2. How well does this image depict XYZ?
>
>
>
> --
>Lars Aronsson (l...@aronsson.se)
>Linköping, Sweden
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Hourly Pageviews Data

[Dan Andreescu]

Hi Michael,

The new data is available, but we found a small formatting bug that we have
to fix.  Because of that, we haven't announced it widely yet, and we
haven't rolled up the data to the monthly level.

The data: https://dumps.wikimedia.org/other/pageview_complete/
The bug: some rows have 6 columns and some rows have 5 columns, where
page_id is missing.  We are inserting "null" and re-writing the files, but
it's almost 3 Terabytes so it'll take a while.  If you want to download and
use the data in the meantime, you're welcome to, just make your parsing
robust to the inconsistency.

Thanks for your patience.  Once we have this sorted out we will make a wide
announcement and explain the history of this data and how going forward
there will be a single unified dataset with all the history we have.

Good suggestion to post updates on the -ez page.  We will do that.

On Mon, Nov 16, 2020 at 9:46 AM Michael Tartre  wrote:

> This was brought up in a previous thread (link here
> ),
> but the aggregated hourly view dumps haven't been published since
> 2020-09-24 (see here
> , also
> mirrored here
> ).
> The response to the previous thread by Dan suggested that the new data
> would be available in a week, but it's already a month past that expected
> deadline. Are there any updates on the status of that new dump, any new
> estimates of when it would become available? I would also suggest posting
> information about the pending change and new system to the information page
> (at https://dumps.wikimedia.org/other/pagecounts-ez/) -- from reading
> that page, there is no indication that data delivery has stopped or that a
> new pipeline will be available shortly.
>
> Thanks for any information,
>
> Michael
>
> --
> *Michael Tartre*
> Senior Machine Learning Engineer
>
> mich...@predata.com
> t: +1 415 857 0967
> 1 Liberty Plaza
> New York, NY 10006
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] TechCom topics 2020-11-04 (fixed)

[Dan Andreescu]

I don't know enough about the parser cache to give Daniel good advice on
his question:

> That's another issue I wanted to raise: Platform Engineeing is working on
> switching ParserCache to JSON. For that, we have to make sure extensions
> only put JSON-Serializable data into ParserOutput objects, via
> setProperty() and setExtensionData(). We are currently trying to figure out
> how to best do that for TemplateData.
>
> TemplateData already uses JSON serialization, but then compresses the JSON
> output, to make the data fit into the page_props table. This results in
> binary data in ParserOutput, which we can't directly put into JSON. There
> are several solutions under discussion, e.g.: [...(see Daniel's original
> message for the list of ideas or propose your own)...]
>
But I see some people hiding in the back who might have some good ideas :)
This is just a bump to invite them to respond.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] TechCom topics 2020-11-04 (fixed)

[Dan Andreescu]

On Wed, Nov 4, 2020 at 12:23 AM Krinkle  wrote:

> *RFC: Expiring watch list entries*
> https://phabricator.wikimedia.org/T124752
>
> This just missed the triage window, but it looks like this was implemented
> and deployed meanwhile (it was in Phase 3). I'm proposing we put this on
> Last Call for wider awareness and so that the team can answer any questions
> people might have, and to address any concerns that people might have based
> on reviewing the proposal we now know the team wanted/has chosen.
>

+1 to this as well
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] TechCom topics 2020-11-04 (fixed)

[Dan Andreescu]

On Tue, Nov 3, 2020 at 4:38 AM Daniel Kinzler 
wrote:

> Am 02.11.20 um 19:24 schrieb Daniel Kinzler:
>
> T262946  *"Bump Firefox
> version in basic support to 3.6 or newer"*: last call ending on
> Wednesday, November 4. Some comments, no objections.
>
>
> Since we are not having a meeting on Wednesday, I guess we should try and
> get quorum to approve by mail.
>
> I'm in favor.
>
+1
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What is JSON (in JavaScript code)?

[Dan Andreescu]

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON

(supported in most browsers I think, it's just available in the global
scope)

On Fri, Oct 30, 2020 at 11:06 AM Strainu  wrote:

> Hi,
>
> I'm looking at solving the following console warning on ro.wp:
> "JQMIGRATE: jQuery.parseJSON is deprecated; use JSON.parse" which
> appears due to outdated Twinkle code. Just making the replacement does
> not work, since JSON is not defined. As a matter of fact, I cannot
> find it anywhere else in the code loading on a normal Romanian
> Wikipedia page.
>
> Alas, the generic name of that object makes searching on mw.org or
> Google rather useless. I can see some similar changes in Phabricator,
> but they seem to work.
>
> So, what is JSON and how can I use it in my code?
>
> Thanks,
>Strainu
>
> P.S. Please don't suggest updating Twinkle...
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] TechCom meeting 2020-10-14

[Dan Andreescu]

As for the minutes of our meeting today, Grant joined and we discussed
the Proposal
for a Technical Decision Making Process
<https://docs.google.com/document/d/1LOAi3RjVeWY-QRdOjnfHCchnM8bSh_D3ug9GhnQ6Nf0/edit#>.
Specifically, we went over Scope, Transition, Resourcing, and a bit about
the Forum and Timeboxing.  The next steps are to establish a more concrete
transition plan, from the current Tech Com process to the new process.  And
Kate will be reaching out to team managers to begin establishing the Forum.

On Wed, Oct 14, 2020 at 9:58 PM Dan Andreescu 
wrote:

> This is the weekly TechCom board review, usually in preparation for our
> meeting, a few days late this week because I forgot.  Apologies.
>
> Activity since Monday 2020-10-06 on the following boards:
>
> https://phabricator.wikimedia.org/tag/techcom/
> https://phabricator.wikimedia.org/tag/techcom-rfc/
>
> Committee inbox:
>
>- T263904 <https://phabricator.wikimedia.org/T263904>: Are traits part
>of the stable interface?
>- T239742 <https://phabricator.wikimedia.org/T239742>: "Should npm
>packages maintained by Wikimedia be scoped or unscoped?"
>   - Both have been sitting in the inbox for a month or so, Tim first
>   pointed this out a few weeks ago but we haven't triaged them yet.
>
> Committee board activity (none)
>
> New RFCs (none)
>
> Phase progression (none)
>
> IRC meeting request (none)
>
> Other RFC activity:
>
>
>- T262946: <https://phabricator.wikimedia.org/T262946> Bump Firefox
>version in basic support to 3.6 or newer
>- Some agreement between Volker and Timo to keep the scope at just the
>   Firefox version bump and move this RFC forward.  I think Volker is 
> welcome
>   to move this back to P4 unless I missed some other ongoing discussion?
>- T259771: <https://phabricator.wikimedia.org/T259771> RFC: Drop
>support for database upgrade older than two LTS releases
>- Discussion about upgrading from 1.16 (this humble reader says
>   upgrading from a version that's 10 years old should not be a priority)
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] TechCom meeting 2020-10-14

[Dan Andreescu]

This is the weekly TechCom board review, usually in preparation for our
meeting, a few days late this week because I forgot.  Apologies.

Activity since Monday 2020-10-06 on the following boards:

https://phabricator.wikimedia.org/tag/techcom/
https://phabricator.wikimedia.org/tag/techcom-rfc/

Committee inbox:

   - T263904 : Are traits part
   of the stable interface?
   - T239742 : "Should npm
   packages maintained by Wikimedia be scoped or unscoped?"
  - Both have been sitting in the inbox for a month or so, Tim first
  pointed this out a few weeks ago but we haven't triaged them yet.

Committee board activity (none)

New RFCs (none)

Phase progression (none)

IRC meeting request (none)

Other RFC activity:


   - T262946:  Bump Firefox
   version in basic support to 3.6 or newer
   - Some agreement between Volker and Timo to keep the scope at just the
  Firefox version bump and move this RFC forward.  I think Volker
is welcome
  to move this back to P4 unless I missed some other ongoing discussion?
   - T259771:  RFC: Drop support
   for database upgrade older than two LTS releases
   - Discussion about upgrading from 1.16 (this humble reader says
  upgrading from a version that's 10 years old should not be a priority)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Update on abstract schema and schema changes

[Dan Andreescu]

I hadn’t followed along with this work and this summary is amazing, thank
you so much!

On Sat, Oct 10, 2020 at 21:14 Amir Sarabadani  wrote:

> Hello,
> It has been a while since I gave an update on the state of abstracting
> schema and schema changes in mediawiki
> . So here's a really long one.
>
> So far around half of the mediawiki core tables have been migrated to
> abstract schema (plus lots of extensions lika Wikibase, Babel, Linter,
> BetaFeatures, etc.). Special thanks to Tgr for reviewing most of the
> patches and Sam Reed and James Forrester for doing the extensions.
>
> With the growing number of schemas being abstracted, this is going to
> affect your development if you work on schema and schema changes in core or
> any of the extensions. So If you do, please read Manual:Schema changes
>  in mediawiki.org
>
> You might think that abstraction is just migrating SQL to JSON but it's
> much more, we are making the database schema of mediawiki much more
> consistent, We are basically addressing several long standing issues like
> T164898  and T42626
>  as well.
>
> *Improvement aspects*
>
> First aspect is drifts between different DBMSes. Sqlite schema is being
> produced by regex replacement (this code
> )
> which is less than great but at least it comes from one place. For
> Postgres, its schema and MySQL/Sqlite has drifted so drastically, that
> fixing it so far required 76 schema changes fixing issues ranging from
> missing indexes to missing PKs, extra AUTO_INCREMENT where it shouldn't be,
> missing DEFAULT values, drifting data types and much more.  You can follow
> the fixes of Postgres in here .
>
> The second aspect is the inconsistency in the schema itself. How do we
> model strings? VARCHAR? VARBINARY()? VARCHAR() BINARY? (all three are
> different things). You'd be surprised how inconsistent our MySQL is. So
> far, we are migrating all VARCHAR() BINARY fields to VARBINARY() (so far
> ten schema changes).
>
> Another inconsistency is timestamps. In MySQL, around half of them are
> BINARY(14) and the other half VARBINARY(14) (but in Postgres all are
> TIMESTAMPTZ), there is even a ticket
>  about it. It makes sense to
> migrate all of them to BINARY(14) but not all timestamps are 14 characters,
> e.g. expiry fields accept "infinity" as value and it's a valid timestamp in
> Postgres ¯\_(ツ)_/¯ When you turn an expiry field to BINARY(14), "infinity"
> becomes "  infinity" and as the result mediawiki doesn't recognize it
> as infinity ("infinity" != "  infinity"). There are several ways to
> move forward handling expiry fields, you can follow the discussion in this
> gerrit patch .
>
> Another fun aspect: Booleans. MySQL doesn't have boolean, it translates
> them to TINYINT(1) but other DBMSes don't have TINYINT, they have SMALLINT
> and BOOL though (and we mostly use SMALLINT for them), we decided to go
> with SMALLINT for these cases (which is different than what Doctrine DBAL
> does, it uses BOOL, so we introduced our own custom type for booleans).
>
> Last but not least: ENUMs. MySQL and Postgres support that but Sqlite
> doesn't. Doctrine DBAL doesn't support ENUM at all (as it's an
> anti-pattern) while core has eight fields that are ENUM. There's an RFC to
> discourage using it in general. Feel free to comment on it.
> 
>
> A miscellaneous note: The directories that hold the archive of sql patches
> of schema change are exploding (some of the sql patches are even orphan but
> we can't find them because there are so many of them). So I started a RFC
> to clean that mess up: Drop support for database upgrade older than two
> LTS releases 
>
> *What's next?*
>
>-  We continue to migrate more tables, hopefully we will get two third
>of them by the end of the year (fingers crossed). You can follow the
>progress in its ticket .
>-  We will support abstract schema changes, really soon, like in a
>couple of weeks. Basically you start a json file containing snapshots of
>before and after of a table and then a maintenance script will produce the
>needed sql patches for you for different schemas. This will increase the
>developer productivity drastically, since 1- Schema change sql files become
>more reliable and consistent and less prone to errors like adding the
>column to the wrong table in some DBMSes
>
> 

Re: [Wikitech-l] pageview dumps for 2020-09?

[Dan Andreescu]

Hi Gabriel,

My team maintains this dataset.  We have been working on a new version
which will deprecate all other pagecount and pageview datasets.  Release of
this new dataset is imminent.  When we saw the old job failed, we couldn't
fix the problem quickly and have been thinking maybe it isn't worth the
effort since the new data is coming out soon.

However, if you have an immediate need, do let us know.  So, in other
words, if you can wait a week or two and tweak your scripts to work with
the new data, that would probably be best.  Otherwise, let us know and we
can look at the failure more closely.

On Tue, Oct 6, 2020 at 3:37 PM Gabriel Altay 
wrote:

> Apologies if this is the wrong list, but it appears that the page views
> dumps for september were stalled on the 24th
>
> https://dumps.wikimedia.org/other/pagecounts-ez/merged/2020/2020-09/
> 
>
> and the full month dump (pagecounts-2020-09-views-ge-5-totals.bz2) was not
> generated,
> https://dumps.wikimedia.org/other/pagecounts-ez/merged/
>
> does anyone know what the situation is?
>
> best,
> -Gabriel
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l]  Wikimedia production errors help

[Dan Andreescu]

>
> For example, of the 30 odd backend errors reported in June, 14 were still
> open a month later in July [1], and 12 were still open – three months later
> – in September. The majority of these haven't even yet been triaged,
> assigned assigned or otherwise acknowledged. And meanwhile we've got more
> (non-JavaScript) stuff from July, August and September adding pressure. We
> have to do better.
>
> -- Timo
>

This feels like it needs some higher level coordination.  Like perhaps
managers getting together and deciding production issues are a priority and
diverting resources dynamically to address them.  Building an awesome new
feature will have a lot less impact if the users are hurting from growing
disrepair.  It seems to me like if individual contributors and maintainers
could have solved this problem, they would have by now.  I'm a little
worried that the only viable solution right now seems like heroes stepping
up to fix these bugs.

Concretely, I think expanding something like the Core Platform Team's
clinic duty might work.  Does anyone have a very rough idea of the time it
would take to tackle 293 (wow we went up by a dozen since this thread
started) tasks?
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Tech Com Board Grooming 2020-08-26

[Dan Andreescu]

Activity since Monday 2020-08-26.

https://phabricator.wikimedia.org/tag/techcom/
https://phabricator.wikimedia.org/tag/techcom-rfc/

Committee inbox: none

Committee board activity:

   - Timo updated T253461 Liberate the @ for AtEase
   

New RFCs:

   - Created / closed / opened / closed all since last week: T261133 Ban IP
   edits on pt.wiki .  Basically
   this was a policy discussion, not a technical RFC

Phase progression: none

IRC meeting request: none

Other RFC activity:

   - Discussion spawned from T240884 (evaluate user-provided regex)
    to T260330 (the PHP service
   Tim's building)  around
   encoding parameters, security, and more
   - Product Data Engineering team shows interest in resourcing RFC: Better
   interface for generating metrics in MediaWiki
    (that would be an
   interesting centralization of all things "metrics", whether system level or
   high level)
   - Development starts on RFC: Parsoid Extension API
   



NOTE: The technical committee is hoping for wider and less formal
participation, starting with this message.  Feel free to chime in, we'll
start a meta-thread about this soon.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki API pageview issue

[Dan Andreescu]

There are two hard problems here.  One is historical page titles.  You can
get those from our new dataset (docs here:
https://dumps.wikimedia.org/other/mediawiki_history/readme.html) by
downloading the months you're interested in from
https://dumps.wikimedia.org/other/mediawiki_history/2020-01/enwiki/, and
looking at the history of the pages you're interested in [1].  As others
have mentioned, page histories can sometimes be very complicated, do let us
know if we didn't get it right for the pages you're interested in, we
worked really hard at vetting the data but there may be exceptions left
unaccounted for.

The second problem is historical redirects.  Sadly, there is no historical
information about redirect status in the databases, only whether or not the
page is a redirect right now.  To find historical information, we have to
parse the wikitext itself, that's why the answers above are complicated.
We are starting to do this but don't yet have the compute power.

To clarify something from above, the flow of data is like this:

0. Historical aggregate data from 2007-2015, kept for reference but uses a
slightly different counter so not directly comparable
1. Webrequest log flowing in through Kafka
--> pageviews found in the log
--> aggregate data simplified and pushed to the dumps referenced by
bawolff
--> aggregate data loaded into the Pageview API (a part of AQS
referenced by Gergo)
--> mediawiki API queries this to respond to action API queries
about pageviews
--> wmflabs pageviews tool does some crazy sophisticated stuff
on top of the API
2. Wikitext dumps
--> processed and loaded into Hadoop
--> [FUTURE] parsed for content like historical redirects and
published as an API or set of dumps files

[1] As a quick intro, each line is an "event" in this wiki, that is
performed on a particular "entity" in {page, user, revision}.  The first
three fields are wiki, entity, and event type, so in your case you'd be
interested in looking for lines starting with enwiki--->page--->move ...
.  Each line has the page id, title of the page as
of today, and title of the page as of the timestamp on that line.  So this
way you can collect all titles for a particular page id or page title.

(if this is useful maybe I should put it on the Phab task about historical
redirects)

On Mon, Feb 24, 2020 at 9:50 PM bawolff  wrote:

> On Tue, Feb 25, 2020 at 1:27 AM MusikAnimal  wrote:
>
> > Unfortunately there's no proper log of redirect changes (I recently
> filed <
> > https://phabricator.wikimedia.org/T240065> for this). There are change
> > tags
> >  that identify redirect
> changes
> > -- "mw-new-redirect" and "mw-changed-redirect-target", specifically --
> but
> > I am not sure if this is easily searchable via the action API. Someone on
> > this list might know.
> >
>
> You can do
>
> https://en.wikipedia.org/w/api.php?action=query=2019%E2%80%9320%20Wuhan%20coronavirus%20outbreak=revisions=timestamp|tags|ids|content=max=mw-new-redirect=2=main
> 
> or
>
> https://en.wikipedia.org/w/api.php?action=query=2019%E2%80%9320%20Wuhan%20coronavirus%20outbreak=revisions=timestamp|tags|ids|content=max=mw-changed-redirect-target=2=main
> 
> (You cannot do both in one query, you can only specify one tag at a time).
> Furthermore, it looks like given a revision id, you would have to determine
> where it redirects yourself, which is unfortunate. I suppose you could look
> at
>
> https://en.wikipedia.org/w/api.php?action=parse=941491141=2=text|links
> (taking the oldid as the revid from the other query) and either try and
> parse the html, or just assume if there is only one main namespace link,
> that that is the right one.
>
> Also keep in mind, really old revisions won't have those tags.
>
> --
> Brian
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] [Data Release] Active Editors by country

[Dan Andreescu]

Today we are releasing a new dataset meant to help us understand the impact
of grants and programs on editing.  This data was requested several years
ago, and we took a long time to bring in the privacy and security experts
whose help we needed to release it.  With that work done, you can download
the data here: https://dumps.wikimedia.org/other/geoeditors/ and read about
it here:
https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors/Public

You can send questions or comments on this thread or on the discussion page.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikimetrics] Sunsetting Wikimetrics

[Dan Andreescu]

Federico, we're talking about *Wikimetrics*, which is a very different
project intended to run stats on groups of users (cohorts) for a specific
set of standard metrics.  This tool was mostly developed for and used by
grantmaking and program teams at WMF and affiliates around the world, but
its use is too low to justify the effort it would take to maintain it.
Better tools are taking its place, and we're working on some of the
infrastructure will support those better tools.

I think you may have been talking about *Wikistats*, which I agree would've
been very nice to keep maintaining.  I think ultimately we don't have too
many of the skills that made Erik successful at what he did.  Perl was just
one of them, and not a major part in my opinion.
If I'm wrong and you're indeed talking about Wikimetrics, do please
elaborate as we're just starting to make this decision.

On Mon, Feb 25, 2019 at 5:21 PM Federico Leva (Nemo) 
wrote:

> Marcel Ruiz Forns, 22/02/19 21:01:
> > Wikimetrics  development has been frozen
> > since 2017, and we Analytics consider it's time to sunset this tool.
>
> This is sad, for sure. As someone who managed to patch and run the
> original WikiStats scripts despite a limited knowledge of Perl, I think
> it would have been more reasonable than often assumed to continue their
> maintenance. However it's clear that we're well past the time when such
> a decision might have been possible, and it's good to see a growing
> interest in feature parity for the new WikiStats.
>
> I think the purposes of this list can be served by the analytics list
> without too much sweat.
>
> Federico
>
> ___
> Wikimetrics mailing list
> wikimetr...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimetrics
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikistats 2.0 - Now with Maps!

[Dan Andreescu]

יגאל חיטרון‎, the -- means "Unknown", perhaps we should name it that.

That's a good idea, Yuri, we'll do that.

On Wed, Feb 14, 2018 at 9:24 PM, Yuri Astrakhan 
wrote:

> Nuria, well done, looks awesome!  I think you want to set zoom limits -
> right now you can zoom in and zoom out almost infinitely. Congrats!
>
> On Wed, Feb 14, 2018 at 8:47 PM, Michael Schönitzer <
> michael.schoenit...@wikimedia.de> wrote:
>
> > here a screenshot:
> >
> > There's a second anomaly: "SX" is also listed…
> >
> > 2018-02-15 1:35 GMT+01:00 יגאל חיטרון :
> >
> > > Sorry, I can't. I opened the link you gave on hewiki. Changed from map
> > view
> > > to table view. I get a list of countries - US, France, Spain, --,
> Japan,
> > > and so on. It's a link, and clicking it opens unexisting wiki article
> > with
> > > the same name.
> > > Igal
> > >
> > >
> > > On Feb 15, 2018 02:23, "Nuria Ruiz"  wrote:
> > >
> > > > Hello,
> > > >
> > > > >Hello and thank you.
> > > > >What do you mean in country named "--"?
> > > >
> > > > Not sure what you are asking, maybe a screenshot would help?
> > > >
> > > > On Wed, Feb 14, 2018 at 3:04 PM, יגאל חיטרון 
> > wrote:
> > > >
> > > > > Hello and thank you.
> > > > > What do you mean in country named "--"?
> > > > > Igal (User:IKhitron)
> > > > >
> > > > >
> > > > > On Feb 15, 2018 00:15, "Nuria Ruiz"  wrote:
> > > > >
> > > > > Hello from Analytics team:
> > > > >
> > > > > Just a brief note to announce that Wikistats 2.0 includes data
> about
> > > > > pageviews per project per country for the current month.
> > > > >
> > > > > Take a look, pageviews for Spanish Wikipedia this current month:
> > > > > https://stats.wikimedia.org/v2/#/es.wikipedia.org/reading/
> > > > > pageviews-by-country
> > > > >
> > > > > Data is also available programatically vi APIs:
> > > > >
> > > > > https://wikitech.wikimedia.org/wiki/Analytics/AQS/
> > > > > Pageviews#Pageviews_split_by_country
> > > > >
> > > > > We will be deploying small UI tweaks during this week but please
> > > explore
> > > > > and let us know what you think.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Nuria
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > > ___
> > > > > Wikitech-l mailing list
> > > > > Wikitech-l@lists.wikimedia.org
> > > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > > ___
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > >
> >
> >
> >
> > --
> > Michael F. Schönitzer
> >
> >
> >
> > Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> > Tel. (030) 219 158 26-0
> > http://wikimedia.de
> >
> > Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
> > Wissens frei teilhaben kann. Helfen Sie uns dabei!
> > http://spenden.wikimedia.de/
> >
> > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
> > Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
> unter
> > der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> > Körperschaften I Berlin, Steuernummer 27/681/51985.
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit login oddities

[Dan Andreescu]

If you're like me, you go to clear your cookies and you have two
called GerritAccount,
that's probably what's causing the problem.  I looked at the date, and
deleted the earlier-dated one.  That seems to work and show you as
logged-in without having to log in again.

On Tue, Feb 13, 2018 at 1:27 PM, Chad  wrote:

> On Tue, Feb 13, 2018 at 7:39 AM Derk-Jan Hartman <
> d.j.hartman+wmf...@gmail.com> wrote:
>
> > Ah, this explains why i wasn't able to login... Really confusing..
> > Can't we set a varnish/apache/nginx/whatever rule or something to
> > overwrite older now broken cookie settings ?
> >
> > DJ
>
>
> >
>
> Got a fix in the works for this. It's affecting more people than I
> initially thought!
>
> -Chad
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Can we drop revision hashes (rev_sha1)?

[Dan Andreescu]

So, as things stand, rev_sha1 in the database is used for:

1. the XML dumps process and all the researchers depending on the XML dumps
(probably just for revert detection)
2. revert detection for libraries like python-mwreverts [1]
3. revert detection in mediawiki history reconstruction processes in Hadoop
(Wikistats 2.0)
4. revert detection in Wikistats 1.0
5. revert detection for tools that run on labs, like Wikimetrics
?. I think Aaron also uses rev_sha1 in ORES, but I can't seem to find the
latest code for that service

If you think about this list above as a flow of data, you'll see that
rev_sha1 is replicated to xml, labs databases, hadoop, ML models, etc.  So
removing it and adding it back downstream from the main mediawiki database
somewhere, like in XML, cuts off the other places that need it.  That means
it must be available either in the mediawiki database or in some other
central database which all those other consumers can pull from.

I defer to your expertise when you say it's expensive to keep in the db,
and I can see how that would get much worse with MCR.  I'm sure we can
figure something out, though.  Right now it seems like our options are, as
others have pointed out:

* compute async and store in DB or somewhere else that's central and easy
to access from all the branches I mentioned
* update how we detect reverts and keep a revert database with good
references to wiki_db, rev_id so it can be brought back in context.

Personally, I would love to get better revert detection, using sha1 exact
matches doesn't really get to the heart of the issue.  Important phenomena
like revert wars, bullying, and stalking are hiding behind bad revert
detection.  I'm happy to brainstorm ways we can use Analytics
infrastructure to do this.  We definitely have the tools necessary, but not
so much the man-power.  That said, please don't strip out rev_sha1 until
we've accounted for all its "data customers".

So, put another way, I think it's totally fine if we say ok everyone, from
date XYZ, you will no longer have rev_sha1 in the database, but if you want
to know whether an edit reverts a previous edit or a series of edits, go
*HERE*.  That's fine.  And just for context, here's how we do our revert
detection in Hadoop (it's pretty fancy) [2].


[1] https://github.com/mediawiki-utilities/python-mwreverts
[2]
https://github.com/wikimedia/analytics-refinery-source/blob/1d38b8e4acfd10dc811279826ffdff236e8b0f2d/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/denormalized/DenormalizedRevisionsBuilder.scala#L174-L317

On Mon, Sep 18, 2017 at 9:19 AM, Daniel Kinzler  wrote:

> Am 16.09.2017 um 01:22 schrieb Matthew Flaschen:
> > On 09/15/2017 06:51 AM, Daniel Kinzler wrote:
> >> Also, I believe Roan is currently looking for a better mechanism for
> tracking
> >> all kinds of reverts directly.
> >
> > Let's see if we want to use rev_sha1 for that better solution (a way to
> track
> > reverts within MW itself) before we drop it.
>
>
> The problem is that if we don't drop is, we have to *introduce* it for the
> new
> content table for MCR. I'd like to avoid that.
>
> I guess we can define the field and just null it, but... well. I'd like to
> avoid
> that.
>
>
> --
> Daniel Kinzler
> Principal Platform Engineer
>
> Wikimedia Deutschland
> Gesellschaft zur Förderung Freien Wissens e.V.
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Wikistats 2.0 Prototype

[Dan Andreescu]

Hello!  We've built an interactive prototype of the next version of
Wikistats  [1] based on community
priorities and feedback.  We'd love your input on the visual design and
look & feel.  We have some follow-up questions but mostly an open
discussion here

[2].  Please comment by *Monday, May 1st* so we can include your feedback
into the first release of Wikistats 2.0.  Thank you very much!


Your Wikistats 2.0 Team


[1] https://analytics-prototype.wmflabs.org/

[2]
https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project/RequestforFeedback/Round2


p.s. apologies for cross-posting
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Wikistats 2.0 Design Preview

[Dan Andreescu]

Hello, and apologies for cross-posting. We have designed a set of
wireframes that reflects the Wikistats community's priorities [1].  We’re
now looking for feedback on the design, and we’d love your input.  We have
key questions that touch on different sections, and links for feedback at
mediawiki.org [2] (if you prefer email, just reply to this message).
Please comment by Monday, February 13th so we can include your thoughts and
iterate. Thank you very much!

Your Wikistats 2.0 Team

[1]
https://www.mediawiki.org/wiki/Analytics/Wikistats/DumpReports/Future_per_report
[2]
https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project/RequestforFeedback/Round1
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes, 2016-04-13

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2016-04-13

note: it looks like some of the wiki markup on etherpad wasn't proper, so
weird things happened like Collaboration is nested under Discovery.  It's
not a sign of a re-org, just that people should go clean that up.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Dumps.wm.o access will be https only

[Dan Andreescu]

:) I would like to think that April Fools would be a little more extreme
than this

Petr, the data people are downloading from dumps is not sensitive itself,
but a user's private information (such as User Agent, IP, etc. is
vulnerable over plain http).  So the move to https protects that from
snoopers like the NSA.  I don't have much to do with this change, and I
consider it beneficial, but does it cause any problems?

On Fri, Apr 1, 2016 at 10:24 AM, Huib Laurens  wrote:

> Aprils fool?
>
> On Friday, 1 April 2016, Petr Bena  wrote:
>
> > Can you give us some justification for this change? It's not like when
> > downloading dumps you would actually leak some sensitive data...
> >
> > On Fri, Apr 1, 2016 at 1:03 PM, Ariel Glenn WMF  > > wrote:
> > > We plan to make this change on April 4 (this coming Monday),
> redirecting
> > > plain http access to https.
> > >
> > > A reminder that our dumps can also be found on our mirror sites, for
> > those
> > > who may have restricted https access.
> > >
> > > Ariel Glenn
> > > ___
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org 
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org 
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Met vriendelijke groet,
>
> Huib Laurens
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Dumps.wm.o access will be https only

[Dan Andreescu]

Cross-posting this announcement since people on Analytics-l tend to use
dumps a lot

basically http access is being redirected to https starting April 4th

On Fri, Apr 1, 2016 at 7:03 AM, Ariel Glenn WMF  wrote:

> We plan to make this change on April 4 (this coming Monday), redirecting
> plain http access to https.
>
> A reminder that our dumps can also be found on our mirror sites, for those
> who may have restricted https access.
>
> Ariel Glenn
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Analytics] [Data Release] [Data Deprecation] [Analytics Dumps]

[Dan Andreescu]

On Wed, Mar 23, 2016 at 1:06 PM, Federico Leva (Nemo) <nemow...@gmail.com>
wrote:

> Dan Andreescu, 23/03/2016 15:58:
>
>>
>> *Clean-up:* Analytics data on dumps was crammed into /other with
>> unrelated datasets.  We made a new page to receive current and future
>> datasets [3] and linked to it from /other and /.  Please let us know if
>> anything there looks confusing or opaque and I'll be happy to clarify.
>>
>
> I assume the old URLs will redirect to the new ones, right?
>

Good question, we didn't change any old URLs actually, so if you're trying
to get to other/pagecounts-ez, other/pagecounts-raw and all that, they're
all still there, just linked-to from /analytics.  We did it this way
because we figured people had scripts that depended on those URLs.  We
thought about moving and symlinking but it's probably unlikely that we'll
ever be able to delete the other/** location.

So mainly we just have a new page where we can do a better job of focusing
on the analytics datasets.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Data Release] [Data Deprecation] [Analytics Dumps]

[Dan Andreescu]

cc-ing our friends in research and wikitech (sorry I forgot initially)

We're happy to announce a few improvements to Analytics data releases on
> dumps.wikimedia.org:
>
> * We are releasing a new dataset, an estimate of Unique Devices accessing
> our projects [1]
> * We are officially making available a better Pageviews dataset [2]
> * We are deprecating two older pageview statistics datasets
> * We moved Analytics data from /other to /analytics [3]
>
> Details follow:
>
>
> *Unique Devices:* Since 2009, the Wikimedia Foundation used comScore to
> report data about unique web visitors.  In January 2016, however, we
> decided to stop reporting comScore numbers [4] because of certain
> limitations in the methodology, these limitations translated into
> misreported mobile usage. We are now ready to replace comscore numbers with
> the Unique Devices Dataset [5][1]. While unique devices does not equal
> unique visitors, it is a good proxy for that metric, meaning that a major
> increase in the number of unique devices is likely to come from an increase
> in distinct users. We understand that counting uniques raises fairly big
> privacy concerns and we use a very private conscious way to count unique
> devices, it does not include any cookie by which your browser history can
> be tracked [6].
>
> We invite you to explore this new dataset and hope it’s helpful for the
> Wikimedia community in better understanding our projects. This data can
> help measurethe reach of wikimedia projects on the web.
>
> *Pageviews:* This [2] is the best quality data available for counting the
> number of pageviews our projects receive at the article and project level.
> We've upgraded from pagecounts-raw to pagecounts-all-sites, and now to
> pageviews, in order to filter out more spider traffic and measure something
> closer to what we think is a real user viewing content.  A short history
> might be useful:
>
> * pagecounts-raw: was maintained by Domas Mituzas originally and taken
> over by the analytics team.  It was and still is the most used dataset,
> though it has some majore problems.  It does not count access to the mobile
> site, it does not filter out spider or bot traffic, and it suffers from
> unknown loss due to logging infrastructure limitations.
> * pagecounts-all-sites: uses the same pageview definition as
> pagecounts-raw, and so also does not filter out spider or bot traffic.  But
> it does include access to mobile and zero sites, and is built on a more
> reliable logging infrastructure.
> * pagecounts-ez: is derived from the best data available at the time.
> So until December 2015, it was based on pagecounts-raw and
> pagecounts-all-sites, and now it's based on pageviews.  This dataset is
> great because it compresses very large files without losing any
> information, still providing hourly page and project level statistics.
>
> So the new dataset, pageviews, is what's behind our pageview API and is
> now available in static files for bulk download back to May 2015.  But the
> multiple ways to download pageview data is confusing for consumers, so
> we're keeping only pageviews and pagecounts-ez and deprecating the other
> two.  If you'd like to read more about the current pageview definition,
> details are on the research page [7].
>
> *Deprecating:* We are deprecating the pagecounts-raw and
> pagecounts-all-sites datasets in May 2016 (discussion here:
> https://phabricator.wikimedia.org/T130656 ).  This data suffers from many
> artifacts, lack of mobile data, and/or infrastructure problems, and so is
> not comparable to the new way we track pageviews.  It will remain here
> because we have historical data that may be useful, but it will not be
> maintained or updated beyond May 2016.
>
> *Clean-up:* Analytics data on dumps was crammed into /other with
> unrelated datasets.  We made a new page to receive current and future
> datasets [3] and linked to it from /other and /.  Please let us know if
> anything there looks confusing or opaque and I'll be happy to clarify.
>
>
> [1] http://dumps.wikimedia.org/other/unique_devices
> [2] http://dumps.wikimedia.org/other/pageviews
> [3] http://dumps.wikimedia.org/analytics/
> [4] https://meta.wikimedia.org/wiki/ComScore/Announcement
> [5] https://meta.wikimedia.org/wiki/Research:Unique_Devices
> [6]
> https://meta.wikimedia.org/wiki/Research:Unique_Devices#How_do_we_count_unique_devices.3F
> [7] https://meta.wikimedia.org/wiki/Research:Page_view
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] [Analytics] Pageview API

[Dan Andreescu]

>
> It does! The API framework as a whole is at
> https://github.com/wikimedia/restbase and you want
>
> https://github.com/wikimedia/restbase/blob/master/specs/analytics/v1/pageviews.yaml
> and https://github.com/wikimedia/restbase/blob/master/mods/pageviews.js
> for the pageviews-specific modules.


And Chris, you're welcome to #wikimedia-analytics to ask any questions you
might have.  I'm known as milimetric around those parts :)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Pageview API

[Dan Andreescu]

Quick general reminder.  Please tag tasks with "Analytics-Backlog" instead
of "Analytics" for now.  We need to clean that up, but we just haven't
gotten around to it.

On Wed, Nov 18, 2015 at 9:05 AM, Dan Andreescu <dandree...@wikimedia.org>
wrote:

> Nice work on the API!
>>
>> I wrote a basic consumer of this API at
>> http://codepen.io/Krinkle/full/wKOMMN#wikimdia-pageviews
>
>
> Cool!  Check out dv.wikipedia.org though, some of the RTL is messing with
> your (N views) parens.
>
> The only hurdle I found is that the 'articles' property is itself
>> nested/double encoded JSON, instead of a plain object. This was somewhat
>> unexpected and makes the API harder to use.
>>
>
> Right, for sure.  The data had to be stuffed that way to save space in
> Cassandra.  So we could parse it and reshape the response in RESTBase, and
> that seems like a good idea and probably wouldn't hurt performance too
> much.  Do you think it's worth the breaking change to the format?  I'll
> post on the bug that MZ filed.
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Pageview API

[Dan Andreescu]

>
> Nice work on the API!
>
> I wrote a basic consumer of this API at
> http://codepen.io/Krinkle/full/wKOMMN#wikimdia-pageviews


Cool!  Check out dv.wikipedia.org though, some of the RTL is messing with
your (N views) parens.

The only hurdle I found is that the 'articles' property is itself
> nested/double encoded JSON, instead of a plain object. This was somewhat
> unexpected and makes the API harder to use.
>

Right, for sure.  The data had to be stuffed that way to save space in
Cassandra.  So we could parse it and reshape the response in RESTBase, and
that seems like a good idea and probably wouldn't hurt performance too
much.  Do you think it's worth the breaking change to the format?  I'll
post on the bug that MZ filed.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Analytics] Pageview API

[Dan Andreescu]

>
> Finally! I waited so many years for a formal tool like that. Thank you!
>

Itzik, I remember your requests for this type of data from a long long time
ago, when I was just starting at the foundation!!  You and the many others
with similar requests are the people we were thanking in the announcement :)

And demo on wmflabs is great, but it will be great to add option to export
> the data to CSV file. Also, the data are only from the begging of October,
> any chance we can load past data as well?
>

So, I agree with what Madhu said that you could file a Phabricator ticket
for this.  But for now, we're not looking to build a UI for it that is
production ready.  The demo was meant just to show that it's very simple to
do so.  It took one of our engineers only a few days and under 300 lines of
code to get this done.  We'd like to be patient and see if the community at
large has an interest in running something like stats.grok.se now that
heavy lifting of data is no longer required, and performance is guaranteed
by us within reasonable expectations.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Pageview API

[Dan Andreescu]

Dear Data Enthusiasts,


In collaboration with the Services team, the analytics team wishes to
announce a public Pageview API
.
For an example of what kind of UIs someone could build with it, check out
this excellent demo  (code)

.


The API can tell you how many times a wiki article or project is viewed
over a certain period.  You can break that down by views from web crawlers
or humans, and by desktop, mobile site, or mobile app.  And you can find
the 1000 most viewed articles

on any project, on any given day or month that we have data for.  We
currently have data back through October and we will be able to go back to
May 2015 when the loading jobs are all done.  For more information, take a
look at the user docs
.


After many requests from the community, we were really happy to finally
make this our top priority and get it done.  Huge thanks to Gabriel, Marko,
Petr, and Eric from Services, Alexandros and all of Ops really, Henrik for
maintaining stats.grok, and, of course, the many community members who have
been so patient with us all this time.


The Research team’s Article Recommender tool 
already uses the API to rank pages and determine relative importance.  Wiki
Education Foundation’s dashboard  is going
to be using it to count how many times an article has been viewed since a
student edited it.  And there are other grand plans for this data like
“article finder”, which will find low-rated articles with a lot of
pageviews; this can be used by editors looking for high-impact work.  Join
the fun, we’re happy to help get you started and listen to your ideas.
Also, if you find bugs or want to suggest improvements, please create a
task in Phabricator and tag it with #Analytics-Backlog
.


So what’s next?  We can think of too many directions to go into, for
pageview data and Wikimedia project data, in general.  We need to work with
you to make a great plan for the next few quarters.  Please chime in here
 with your needs.


Team Analytics
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wmfall] Transitioning wikistats pageview reports to use new pageview definition

[Dan Andreescu]

The old definition was never put down clearly in words, but it can be
understood easiest I think from this SQL query that generates
pagecounts-all-sites (which was what wikistats used until this switch):
https://github.com/wikimedia/analytics-refinery/blob/master/oozie/pagecounts-all-sites/load/insert_hourly_pagecounts.hql#L46

Strainu I know you can read that, but if others are curious I'll try to put
it in words.

On Wednesday, November 11, 2015, Strainu  wrote:

> Hi Nuria,
>
> What is the "old" definition? For some wikis the difference is from
> simple to double.
>
> Thanks,
>   Strainu
>
> 2015-11-11 2:00 GMT+02:00 Nuria Ruiz >:
> > Hello!
> >
> > The analytics team wishes to announce that we have finally transitioned
> > several of the pageview reports in stats.wikimedia.org  to the new
> pageview
> > definition [1]. This means that we should no longer have two conflicting
> > sources of pageview numbers.
> >
> > While we are not not fully done transitioning pageview reports we feel
> this
> > is an important enough milestone that warrants some communication. BIG
> > Thanks to Erik Z. for his work on this project.
> >
> > Please take a look at a report using the new definition (a banner is
> > present when report has been updated)
> > http://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm
> >
> > Thanks,
> >
> > Nuria
> >
> >
> > [1] https://meta.wikimedia.org/wiki/Research:Page_view
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org 
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes, 2015-04-15

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-04-15
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes, 2015-02-18

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-02-18
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes 2015-02-11

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-02-11

If anyone with better wiki kung fu than me could take a look and figure out
why the table of contents is not rendering on that page like it does on all
the other [1] ones, I would appreciate it.

[1] https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-02-04
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes for 2015-02-04

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-02-04
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-01-14

note: no meeting next week as All Staff is happening.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums notes since December 24th (2014-12-24, 2014-12-31, 2015-01-07)

[Dan Andreescu]

Here are notes from the last three scrum of scrums meetings:

https://www.mediawiki.org/wiki/Scrum_of_scrums/2014-12-24

https://www.mediawiki.org/wiki/Scrum_of_scrums/2014-12-31

https://www.mediawiki.org/wiki/Scrum_of_scrums/2015-01-07


Special thanks! to the Wikidata team for putting up their update for
today's meeting.

Happy New Year everyone :)
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Scrum of Scrums Notes for 2014-12-17

[Dan Andreescu]

https://www.mediawiki.org/wiki/Scrum_of_scrums/2014-12-17
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Labs-l] Announcement: Yuvi Panda joins Ops

[Dan Andreescu]

 At the same time I find it sad that he moves to the USA because he was
 invaluable because he was available when the other Labs people were not
 around. As such he provided a much needed service exactly because he lives
 in India.


Oh that's a common misunderstanding which arises from the assumptions that
Yuvi sleeps and Yuvi is human.  In reality, Yuvi is always up on all time
zones so everyone thinks he helps their time zone.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] HTML templating progress; Knockout Components curly brace syntax

[Dan Andreescu]

On Tue, Jul 8, 2014 at 10:23 AM, Chad innocentkil...@gmail.com wrote:

 On Tue, Jul 8, 2014 at 7:16 AM, Tyler Romeo tylerro...@gmail.com wrote:

  I just want to be clear that any sort of template syntax that resembles
 or
  can be confused with wikitext is not something that we can or should
 allow
  in core. If MobileFrontend and whatnot want to use this syntax, so be it.
 

 We can absolutely allow it, so that's not strictly true. Whether we should
 is another matter. I just want to be clear that this is your opinion.


I'm not sure I follow, it seems to me knockout syntax and wikitext are
different, and have different purposes.  Can you elaborate a bit more,
Tyler?
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] new terms of use/paid contributions - do they apply to mediawiki

[Dan Andreescu]

cc-ing Luis as I think this consequence, whether intended or not, would
interest him


On Mon, Jun 16, 2014 at 4:25 PM, Brian Wolff bawo...@gmail.com wrote:

 Hi,

 So from what I understand, there's now been an amendment to WMF's
 terms of use to require disclosure of paid contributions [1]. Its a
 little unclear how this applies to MediaWiki as a project, but a
 literal reading of the policy makes it seem like MediaWiki is
 included.

 * MediaWiki is arguably a project of the Wikimedia foundation. The
 foundation's website says as much [2]
 *A commit/patchset certainly seems like a contribution.

 Thus the new policy would require anyone submitting code to use to
 declare who they work for. Personally this seems both unnecessary to
 me, as well as unlikely to be followed. For example, I see no reason
 why some person who uses our software should have to declare who they
 work for when they upstream a bug fix, etc.

 I would suggest we follow commons' lead [3], and declare that we do
 not have disclosure requirements for people giving us code.

 --bawolff

 [1]
 https://wikimediafoundation.org/w/index.php?title=Terms_of_Usediff=0oldid=90463
 [2] https://wikimediafoundation.org/wiki/Our_projects
 [3]
 https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Alternative_paid_contribution_disclosure_policy

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] new terms of use/paid contributions - do they apply to mediawiki

[Dan Andreescu]

 The terms of use is basically a clickwrap
 https://en.wikipedia.org/wiki/Clickwrap agreement, right? When you make
 a
 contribution, you see by clicking you accept blah blah blah somewhere,
 once you click knowing that, you made a sort of contract from a legal point
 of view. So the terms of use only applies to you if you do actually see
 that piece of text, which does not seem to be present on mediawiki.org,
 nor
 on gerrit.wikimedia.org, and obviously not in the git interface when you
 are pushing a patch.


I agree with this reading, but I also think people like Bartosz will be
legitimately confused.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Maps-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

 FYI, Limn is also sort of taken in the MediaWiki/WMF namespace:
 https://www.mediawiki.org/wiki/Analytics/Limn


Yep, but that Limn is dying slowly.  If we do this new extension properly,
it will take its place.  As Erik said, this is not staffed for success
right now but it would address many shared needs.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

 Overall this is very exciting work with lots of potential future
  applications. I don't think it's resourced for success yet, but let's
  figure out where it should sit in our roadmap since it would address
  many shared needs if done right.
 

 True! It would be nice to talk about it during Wikimania, from my side I
 started a community consultation here:

 https://meta.wikimedia.org/wiki/Requests_for_comment/How_to_deal_with_open_datasets


Thanks for the interest Micru, and especially that RFC.  I am not scheduled
to attend Wikimania but I could fly myself there if you think this topic
will receive attention.  (Also, I don't know how Wikimania works :))
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

 Sorry, but Limn sounds pretty pretentious, and it is hard to pronounce.


no problem at all, you're not hurting my feelings :)  Sounds like people
don't like the name, so let's drop it.  The top contender right now is
Visual: and I'll ping the Visual Editor folks right now to see if they mind.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

 2014-05-14 19:43 GMT+03:00 Dan Andreescu dandree...@wikimedia.org:
  For the short term, I think further exploration of the Map namespace is
  great, but I think generic visualization work could go into the
  Visualization namespace.  My suggestion for a name for this namespace may
  seem a bit obscure.  It's a word that means to illuminate: Limn [5].

 3) It's a word that is difficult to translate.


I'm open to alternatives, but my original choice (WikiViz) was taken
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

I like Visual:, any +1s? +2s?

The only downside might be a slight conflict with VisualEditor


On Thu, May 15, 2014 at 11:20 AM, dan-nl dan.entous.wikime...@gmail.comwrote:

 PictureIt:
 Envision:
 Imagine:

 On May 15, 2014, at 17:06 , Jon Robson jdlrob...@gmail.com wrote:

  Visual: ?
  On 14 May 2014 10:44, Derk-Jan Hartman d.j.hartman+wmf...@gmail.com
  wrote:
 
  PS, i'm building an instance that is running this extension.
 
  On Wed, May 14, 2014 at 12:34 AM, Jon Robson jrob...@wikimedia.org
  wrote:
  During the Zurich hackathon, DJ Hartman, Aude and I knocked up a
  generic maps prototype extension [1]. We have noticed that many maps
  like extensions keep popping up and believed it was time we
  standardised on one that all these extensions could use so we share
  data better.
 
  We took a look at all the existing use cases and tried to imagine what
  such an extension would look like that wouldn't be too tied into a
  specific use case.
 
  The extension we came up with was a map extension that introduces a
  Map namespace where data for the map is stored in raw GeoJSON and can
  be edited via a JavaScript map editor interface. It also allows the
  inclusion of maps in wiki articles via a map template.
 
  Dan Andreescu also created a similar visualisation namespace which may
  want to be folded into this as a map could be seen as a visualisation.
  I invite Dan to comment on this with further details :-)!
 
  I'd be interested in people's thoughts around this extension. In
  particular I'd be interested in the answer to the question For my
  usecase A what would the WikiMaps extension have to support for me to
  use it.
 
  Thanks for your involvement in this discussion. Let's finally get a
  maps extension up on a wikimedia box!
  Jon
 
  [1] https://github.com/jdlrobson/WikiMaps
 
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l


 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

By the way, the Vega work is deployed to this wiki:

http://analytics-wiki.wmflabs.org/mediawiki/Main_Page#Visualization

Jon/DJ/Aude, if you want I can add a more generic proxy name for that wiki
and put your stuff on there too?  It's in the analytics project in labs but
I'm happy to give anyone rights to it.


On Thu, May 15, 2014 at 1:35 PM, Dan Andreescu dandree...@wikimedia.orgwrote:

 I like Visual:, any +1s? +2s?

 The only downside might be a slight conflict with VisualEditor


 On Thu, May 15, 2014 at 11:20 AM, dan-nl 
 dan.entous.wikime...@gmail.comwrote:

 PictureIt:
 Envision:
 Imagine:

 On May 15, 2014, at 17:06 , Jon Robson jdlrob...@gmail.com wrote:

  Visual: ?
  On 14 May 2014 10:44, Derk-Jan Hartman d.j.hartman+wmf...@gmail.com
  wrote:
 
  PS, i'm building an instance that is running this extension.
 
  On Wed, May 14, 2014 at 12:34 AM, Jon Robson jrob...@wikimedia.org
  wrote:
  During the Zurich hackathon, DJ Hartman, Aude and I knocked up a
  generic maps prototype extension [1]. We have noticed that many maps
  like extensions keep popping up and believed it was time we
  standardised on one that all these extensions could use so we share
  data better.
 
  We took a look at all the existing use cases and tried to imagine what
  such an extension would look like that wouldn't be too tied into a
  specific use case.
 
  The extension we came up with was a map extension that introduces a
  Map namespace where data for the map is stored in raw GeoJSON and can
  be edited via a JavaScript map editor interface. It also allows the
  inclusion of maps in wiki articles via a map template.
 
  Dan Andreescu also created a similar visualisation namespace which may
  want to be folded into this as a map could be seen as a visualisation.
  I invite Dan to comment on this with further details :-)!
 
  I'd be interested in people's thoughts around this extension. In
  particular I'd be interested in the answer to the question For my
  usecase A what would the WikiMaps extension have to support for me to
  use it.
 
  Thanks for your involvement in this discussion. Let's finally get a
  maps extension up on a wikimedia box!
  Jon
 
  [1] https://github.com/jdlrobson/WikiMaps
 
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l


 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Transcluding non-text content as HTML on wikitext pages

[Dan Andreescu]

  Can you outline how RL modules would be handled in the transclusion
  scenario?

 The current patch does not really address that problem, I'm afraid. I can
 think
 of two solutions:

 * Create an SyntheticHtmlContent class that would hold meta info about
 modules
 etc, just like ParserOutput - perhaps it would just contain a ParserOutput
 object.  And an equvalent SyntheticWikitextContent class, perhaps. That
 would
 allow us to pass such meta-info around as needed.

 * Move the entire logic for HTML based transclusion into the wikitext
 parser,
 where it can just call getParserOutput() on the respective Content object.
 We
 would then no longer need the generic infrastructure for HTML transclusion.
 Maybe that would be a better solution in the end.

 Hm... yes, I should make an alternative patch using that approach, so we
 can
 compare.


Thanks a lot Daniel, I'm happy to help test / try out any solutions you
want to experiment with.  I've moved my work to gerrit:
https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/Limnand
the last commit (with a lot of help from Matt F.) may be ready for you
to use as a use case.  Let me know if it'd be helpful to install this
somewhere in labs.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Maps-l] Output from Zurich Hackathon - yet another maps extension!

[Dan Andreescu]

Thanks for starting this Jon, the end result is going to be awesome.  So
here's how I see things, it's roughly along the lines of what you've been
saying:

Server-side rendering and scaling is important.  This is one of the main
reasons I picked Vega [2] for my hack.  The same visualization grammar can
be used to generate png or svg [2].  I see the approach to visualization as
being similar to Parsoid:

* user creates a visualization (with a visual editor) and saves
* Vega parses that server side and generates an image, then refreshes the
caches accordingly
* a transcluded visualization renders the image from cache as a link to the
interactive version
* when the link is clicked, something like MediaViewer shows the
interactive visualization
* alternatively, we can allow editors to show the interactive version in
the transclusion itself, but that has performance implications until
browser caches are filled.

Now a little bit about where I see the Map namespace.  A visualization in
the world of Vega has three parts:

* Data (one or more sets of data, can be geojson, topojson, tsv, csv,
layers from OSM [3], OHM [4], etc.)
* Transformations on that data (scales, normalization, etc.)
* Marks (visual representations)

Transformations and Marks can be written by hand or by a visual editor that
introspects the specification to show what's possible.  Data to me is the
tricky part.  We may need to restrict Vega to only consume open data that's
hosted and curated by us, and that could be done in a few different ways:

* Namespaces like the Maps namespace that enables awesome collaborative
editing of geojson
* Datasets in WikiData using an alternative data model
* File namespace serving raw data from Commons (where people are familiar
with take down notices and have the infrastructure to deal with that)

But yes, I do see the Maps namespace as one of the sources of data that we
could visualize with Vega.  And recent developments in Vega make me feel
that it's really a solid choice for generic visualization.  We have
interactivity, headless mode, a seemingly clear path to a visual editor via
introspection of the grammar specification, and pretty much everything I
can think of needing from such a tool.

For the short term, I think further exploration of the Map namespace is
great, but I think generic visualization work could go into the
Visualization namespace.  My suggestion for a name for this namespace may
seem a bit obscure.  It's a word that means to illuminate: Limn [5].
 There's an old project by that name of which I'm not very fond (despite
writing some of it myself), but I've always thought the word was beautiful
and fit.  To what Antoine was saying earlier, we should illuminate the
world's knowledge with beautiful visualizations.


[1] https://github.com/trifacta/vega
[2] https://github.com/trifacta/vega/wiki/Headless-Mode
[3] OSM - Open Street Maps http://wiki.openstreetmap.org/wiki/Main_Page
[4] OHM - Open Historical Maps
http://wiki.openstreetmap.org/wiki/Open_Historical_Map
[5] Limn - depict or describe in painting or words:
https://github.com/wikimedia/mediawiki-extensions-Limn


On Wed, May 14, 2014 at 9:43 AM, Jon Robson jrob...@wikimedia.org wrote:

 Tim I completely agree. This is something we need to setup.
 Patches very much welcomed! :-)



 On Wed, May 14, 2014 at 7:51 AM, Tim Alder t...@alder-digital.de wrote:
  I think the most important feature is to create on serverside a
  thumbnail for each map by using something like http://phantomjs.org/
  This thumbnails should than be in the big WMF caches. The map would
  become interactively only in the case a user click on it.
  This would reduce the numbers of request for loading a page and JS
  overhead and it would increase the stability of the system.
  Without this feature I afraid to never see the extension live in
 Wikipedia.
 
  Other nice features you can see at umap.openstreetmap.fr:
  *Choosing different backgrounds
  *POIs with interactive descriptions
  *Geometry import from OSM (WIWOSM)
  *different layers
  *...
 
  Greeting Tim alias Kolossos
 
 
  Am 14.05.2014 00:34, schrieb Jon Robson:
  During the Zurich hackathon, DJ Hartman, Aude and I knocked up a
  generic maps prototype extension [1]. We have noticed that many maps
  like extensions keep popping up and believed it was time we
  standardised on one that all these extensions could use so we share
  data better.
 
  We took a look at all the existing use cases and tried to imagine what
  such an extension would look like that wouldn't be too tied into a
  specific use case.
 
  The extension we came up with was a map extension that introduces a
  Map namespace where data for the map is stored in raw GeoJSON and can
  be edited via a JavaScript map editor interface. It also allows the
  inclusion of maps in wiki articles via a map template.
 
  Dan Andreescu also created a similar visualisation namespace which may
  want to be folded into this as a map could be seen

Re: [Wikitech-l] [Engineering] Help! Phabricator and our code review process

[Dan Andreescu]

On Mon, May 5, 2014 at 10:24 PM, Matthew Flaschen
mflasc...@wikimedia.orgwrote:

 On 05/02/2014 03:56 PM, C. Scott Ananian wrote:

 [greg-g] cscott: James_F crazy idea here: can some teams use it for
 real (I think growth is, kinda?) and export/import to a future real
 instance?
 frontend...


 No, we're not using it for real currently.  We (Growth) have talked about
 potentially being an early adopter, but have not committed to this yet.

 Matt Flaschen


Likewise for Analytics
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] RfC on Product Management Tools and Development Toolchain

[Dan Andreescu]

Andre,

It's my understanding that the current Phabricator instance is temporary.
 Indeed, it includes mostly jokes and throw-away testing tasks, comments,
boards, etc.  Could we stand up an instance to which we would potentially
migrate to?  We would have to reserve task ids 1-10 to allow us to port
from Bugzilla, but do we have any other blockers?

I had volunteered my team to try it out for one of our projects, but I've
been hesitating until we have a blessed version.

Dan


On Mon, Apr 14, 2014 at 8:02 PM, Andre Klapper aklap...@wikimedia.orgwrote:

 Hi,

 as previously announced [1], we've been facilitating a collective review
 of Wikimedia's current product management tools and development
 toolchain.

 The most popular idea at the moment is to consolidate Wikimedia's
 product management and infrastructure tools (such as Bugzilla, Gerrit,
 RT, Mingle, Trello) into all-in-one Phabricator. We have therefore put
 together a Request for comment to bring this up for wider discussion.

 This discussion affects anyone who deals with bug reports, feature
 requests and code changes in Wikimedia, so it's critical that you test
 Phabricator for your own use and make your voice heard in the RFC:

   https://www.mediawiki.org/wiki/Requests_for_comment/Phabricator

 We're compiling a list of Frequently asked questions at
 https://www.mediawiki.org/wiki/Requests_for_comment/Phabricator/FAQ ;
 You're welcome to add more and help answer them :)

 We'll host a few IRC discussions while the RFC is running to help answer
 questions, etc. Our tentative times and dates are at

 https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Phabricator#IRC_discussions

 Thank you for your input!

 Guillaume and Andre

 [1] http://lists.wikimedia.org/pipermail/wikitech-l/2014-March/074896.html

 --
 Andre Klapper | Wikimedia Bugwrangler
 http://blogs.gnome.org/aklapper/


 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] MediaWiki on Google App Engine

[Dan Andreescu]

I've done things like port mako templates to Google App Engine.  As
mentioned by Jeremy Baron, the common problem to porting anything of
sufficient complexity is that you're not allowed to write files to the
disk.  To get mako to work, since it caches compiled templates to disk, I
patched it to write the templates to memcache instead.  This worked well
enough but I didn't need persistent files.  So you'd probably have to
redirect mediawiki to write files somewhere else more permanent.  Google
Drive seems like a decent place but last time I tried, integrating Drive
with App Engine was silly hard.  This might have changed since I haven't
tried for a few years.


On Sun, Apr 6, 2014 at 1:44 AM, Daniel Friesen
dan...@nadir-seen-fire.comwrote:

 Never tried, but the topic did interest me enough awhile back to write a
 detailed tracking bug for it:
 https://bugzilla.wikimedia.org/show_bug.cgi?id=55475

 ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

 On 2014-04-05, 1:25 PM, Denny Vrandečić wrote:
  Did anyone manage to get MediaWiki running on Google App Engine? I am a
 bit
  dense, it seems, and would appreciate a few pointers.
 
  Cheers,
  Denny
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l


 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] CentralAuth questions

[Dan Andreescu]

Any links to documentation on consuming data from the CentralAuth databases
is welcome.  We searched a bit and found mostly installation instructions.


On Thu, Mar 27, 2014 at 4:20 PM, Teresa Cho tcho...@gmail.com wrote:

 Hi all,

 I'm trying to add a feature to Wikimetrics that will allow users to create
 a cohort with a username and find all accounts across wikis. I want to use
 the CentralAuth database, because as far as I can tell, it stores the
 global username and all the local usernames. However, I don't see where it
 connects the globalusers to the localusers. Is it just the username?

 Does the username have to be the same across local wikis and you query the
 localuser table with what you think is the global username? If that's the
 case, I suppose I don't need to look at the global table.

 Thanks,
 Teresa
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] CentralAuth questions

[Dan Andreescu]

Thank you very much for the reply Max, +1 beer for next time we meet.


On Thu, Mar 27, 2014 at 5:10 PM, MZMcBride z...@mzmcbride.com wrote:

 Teresa Cho wrote:
 I'm trying to add a feature to Wikimetrics that will allow users to
 create a cohort with a username and find all accounts across wikis. I
 want to use the CentralAuth database, because as far as I can tell, it
 stores the global username and all the local usernames. However, I don't
 see where it connects the globalusers to the localusers. Is it just the
 username?
 
 Does the username have to be the same across local wikis and you query
 the localuser table with what you think is the global username? If that's
 the case, I suppose I don't need to look at the global table.

 Hi.

 Broadly, I think the answer for you're looking for is no: CentralAuth
 accounts (global user accounts that match to local user accounts) are not
 fully unified on Wikimedia wikis. It's a long-term goal, but it's a
 disruptive change to make, so it's taken a while. :-)

 It sounds like you want programmatically retrieve the info from:
 https://www.mediawiki.org/wiki/Special:CentralAuth/Jimbo_Wales.

 If so, I'd recommend the MediaWiki Web API
 (https://www.mediawiki.org/w/api.php) for this. Perhaps the
 globaluserinfo API module?
 https://www.mediawiki.org/w/api.php?action=querymeta=globaluserinfoguiuse
 r=Jimbo+Walesguiprop=groups|merged|unattached

 If you must directly query the MediaWiki database using SQL, you'll likely
 need to read through the source code of the CentralAuth MediaWiki
 extension to figure out exactly what the PHP and SQL is doing with the
 underlying data. The source code of the CentralAuth MediaWiki extension
 can be found here:
 https://git.wikimedia.org/tree/mediawiki%2Fextensions%2FCentralAuth.git.
 You'll likely want to read through central-auth.sql in particular.

 Dan Andreescu wrote:
 Any links to documentation on consuming data from the CentralAuth
 databases is welcome.  We searched a bit and found mostly installation
 instructions.

 Well, very generally you (or your program) probably shouldn't be querying
 the databases directly, but if you can provide more specific information
 about where you looked, we can probably add some redirects for future
 w[ao]nderers.

 For general clarity, while I used www.mediawiki.org in the examples in
 this e-mail, because the Web API is retrieving global (wiki farm-wide)
 data, the equivalent URL paths should work on other Wikimedia wikis such
 as en.wikipedia.org or meta.wikimedia.org.

 MZMcBride



 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] IRC Office Hour on Project Management Tools Review: Friday 28th, 17:00UTC

[Dan Andreescu]

Ha!  I had added wikimetrics but it wasn't working at first.  It's working
now so that's great, I'll take a deeper look.


On Tue, Mar 25, 2014 at 12:35 PM, Gilles Dubuc gil...@wikimedia.org wrote:

 You can add external repos, people have already added a bunch:
 http://fab.wmflabs.org/diffusion/ This allows you to comment/raise issue
 on
 commits that have already been pushed. Example:
 http://fab.wmflabs.org/rMMV49bc5edd9384ecc22a05a22a88bc70cd2439c5b3

 And I'm pretty sure the phabricator command line tool (arcanist) should
 just work to upload diffs for review:
 https://secure.phabricator.com/book/phabricator/article/arcanist_diff/


 On Tue, Mar 25, 2014 at 5:39 PM, Dan Andreescu dandree...@wikimedia.org
 wrote:

  looking forward to attending this, thanks for organizing
 
 
   If you'd like to learn more about Phabricator's project management and
   roadmap functionality, I invite you to test it live on the test
   instance at http://fab.wmflabs.org/ . Alternatively, you can also
   peruse http://phabricator.org/ and http://phabricator.org/tour/ . I
   strongly encourage everyone to do so before the IRC discussion, so we
   can all have a more informed chat about Phabricator and the other
   options currently under consideration.
  
 
  I've taken a look but I think it would be awesome if we could set up
  repositories and do a mock code review.  I'd love to see how that
  integrates with tasks (so we can compare with gerrit - bugzilla for
  example).
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] IRC Office Hour on Project Management Tools Review: Friday 28th, 17:00UTC

[Dan Andreescu]

looking forward to attending this, thanks for organizing


 If you'd like to learn more about Phabricator's project management and
 roadmap functionality, I invite you to test it live on the test
 instance at http://fab.wmflabs.org/ . Alternatively, you can also
 peruse http://phabricator.org/ and http://phabricator.org/tour/ . I
 strongly encourage everyone to do so before the IRC discussion, so we
 can all have a more informed chat about Phabricator and the other
 options currently under consideration.


I've taken a look but I think it would be awesome if we could set up
repositories and do a mock code review.  I'd love to see how that
integrates with tasks (so we can compare with gerrit - bugzilla for
example).
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] About the bug 46453 - Uzbek: Change date and decimal separators

[Dan Andreescu]

 Hi,
 I'm having some trouble when trying to upload the fix to gerrit. I followed
 the instructions in [1]. But I get the bellow error when trying to run
 git review -s

 Error.

 Problems encountered installing commit-msg hook
 The following command failed with exit code 1
 scp  gerrit.wikimedia.org:hooks/commit-msg .git/hooks/commit-msg
 ---
 .git/hooks/commit-msg: No such file or directory
 ---


 [1]
 http://www.mediawiki.org/wiki/Gerrit/Tutorial#Make_and_commit_your_change


I'm certainly no fan of gerrit, and I hope gerrit-patch-uploader that
Sumana mentioned helps you.  But if not, the error you're having above
sounds like you are doing git review -s outside your repository's
directory.  So you might try:

git clone repository-address folder-to-clone-into
cd folder-to-clone-into
git review -s
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Zurich Hackathon: Creating a map namespace

[Dan Andreescu]

This is awesome.  I have a decent amount of experience with different
technologies that we could use to build maps and present them both
statically through our cache layers and dynamically for editors.  Let's
kick this project's butt in Zurich.


On Wed, Mar 5, 2014 at 3:54 PM, Jon Robson jdlrob...@gmail.com wrote:

 This may be extremely ambitious, but I'm keen to kick off development
 around the creation of a map namespace during the Zurich hackathon.

 The goal would be to setup an editable map namespace that could be
 used for a variety of things, one of which would be adding a map view
 to the Special:Nearby page provided via the mobile site. The goal is a
 proof of concept not necessarily anything production ready (but that
 would be great if we could get to that point!)

 Please let me know if you would also be interested on hacking such a
 thing -
 https://www.mediawiki.org/wiki/Z%C3%BCrich_Hackathon_2014/Geo_Namespace
 - or if doing so would be a terrible idea (but if you have to go down
 that route please provide constructive reasoning on what would be a
 less terrible idea)

 Excited to hack on cool things in Zurich!

 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Minor Bingle improvements - users please update your code

[Dan Andreescu]

thanks Arthur!


On Thu, Feb 20, 2014 at 8:19 PM, Arthur Richards aricha...@wikimedia.orgwrote:

 I had some free time today and made some minor improvements to Bingle to
 take care of two longstanding, really annoying issues:
 https://github.com/awjrichards/bingle/issues/10
 https://github.com/awjrichards/bingle/issues/11
 (Same issues also reported via BZ:
 https://bugzilla.wikimedia.org/show_bug.cgi?id=57830)

 The changes should improve the formatting of bug descriptions/comments in
 Mingle. Bingle users, please consider git pull'ing for the latest :)

 --
 Arthur Richards
 Software Engineer, Mobile
 [[User:Awjrichards]]
 IRC: awjr
 +1-415-839-6885 x6687
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Non-Violent Communication

[Dan Andreescu]

 I assume your good faith, and I foresee its consequences. You couldn't
 employ your NVC skills because you were, quote, in a hurry, end quote. That
 means, NVC just doesn't work when it's needed. I don't think everyone here
 has a lot of spare time to mix original thoughts with a dump of meaningless
 requests and pardons. You want to share how you feel? I don't think it's
 the right place to do this. Don't ask to ask, just ask, and so on.


I think this and other responses to non-violent communication make a lot of
sense.  They're in line with the old quote First they ignore you, then
they laugh at you, then they fight you, then you win.  But this process
takes years and we seem to be at the laugh and fight stage.

I think violence is a particularly efficient way of getting what you want.
 Assume good faith is just a way to apologize in advance for employing
violence.  And honestly, I come from a culture where violence is a totally
acceptable form of communication, and I'm a violent communicator.  I creep
myself out when I try to not be violent, but I recognize that much harmony
would result from adopting the principles of NVC.  Anyway I don't have any
opinion on either side of this discussion, just wanted to point out that
the responses are to be expected.  And to say to Derric thank you, your
post was not in vain and it did not turn me off to the subject.  On the
contrary, it made me admire that more people are willing to try it.




 On Tue, Feb 18, 2014 at 5:38 PM, Derric Atzrott 
 datzr...@alizeepathology.com wrote:

  Question for Derric: why didn't you formulate your suggestion using
  NVC?
 
  I was excited and in a hurry.  In retrospect I really think that I should
  have.
 
  After reading some of the replies I felt rather disappointed and
  frustrated, and even a little sad as I didn't feel my need for
  understanding was met.
 
  In the future I will try to take a little more time writing emails to the
  list.  I'm sorry to anyone who felt offended by it or felt that my email
  was, well, violent.  That was not my intention at all.  I just began
 myself
  looking into and trying to practice NVC in the past six months or so,
 and I
  am, as of now, still not terribly great at it.
 
  Again, I want to express my apologies, and I really hope that I didn't
  turn anyone off to the subject.  I guess all I was really trying to say
 in
  that email is that when conversation on this list gets heated, I feel
  frustrated because my needs for calm and community are not met.  I end up
  not wanting to participate because I don’t think that I will be heard or
  understood.  I would like to request that people onlist look into
  strategies to help everyone get along, whether that is AGF, or NVC, or
  something else, does not matter as much to me.  I suggested NVC because
 it
  has been a very useful tool for me in the past.
 
  Thank you,
  Derric Atzrott
 
 
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 



 --
 З павагай,
 Павел Селіцкас/Pavel Selitskas
 Wizardist @ Wikimedia projects
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Bingle/Bugello broken post-Bugzilla upgrade

[Dan Andreescu]

 python-requests bundles it's own cacert list (although the ubuntu .deb
 version might use the central certificate store - not sure about that),
 which might be outdated. Some older cacert lists have issues with RapidSSL
 certificates (this is an issue with the list bundled with httplib2, for
 example).


We use pywikibot's httplib2 which solves the issue Merlijn mentions here.
 For requests though, the last commit message on their cacert.pem file
makes me like your original thought that requests should be updated:

https://github.com/kennethreitz/requests/blob/master/requests/cacert.pem
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Bingle/Bugello broken post-Bugzilla upgrade

[Dan Andreescu]

Bingle is actually a python tool: https://github.com/awjrichards/bingle

Arthur, sorry I spend a couple minutes brainstorming and came up empty.
 Keep us updated and I'll take a more serious look if the problem persists.


On Thu, Feb 13, 2014 at 3:58 PM, Daniel Zahn dz...@wikimedia.org wrote:

 Arthur,

 i think i know the issue here. zirconium is indeed a shared host, so it
 runs several misc. web services
 using https on a single IP, so we rely on clients speaking SNI to get the
 correct virtual host.

 java 6 and IE on XP are among the few clients who don't.

 I think your applications are java and don't speak SNI, so they are getting
 the first virtual host, which is planet.

 this can be fixed by either:

 use Java 7  which should support SNI .. see f.e.

 http://stackoverflow.com/questions/12361090/server-name-indication-sni-on-java

 quote  on Java 7 use

 new URL(https://cmbntr.sni.velox.ch/;).openStream()

 until HTTPCLIENT-1119 is fixed


 or i can cheat by changing the order Apache loads the site configs, f.e. i
 could make it


 sites-enabled/001-Bugzilla , ./002-Planet etc. Then those clients who don't
 speak SNI get Bugzilla (but the Planet users don't get their planet, but
 Bugzilla seems more important.


 or we would have to get an extra IP address just for Bugzilla
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Proposal for Zürich Hackathon - getting close to a production-like Vagrant instance

[Dan Andreescu]

   A few of us have been discussing how awesome it would be to use
  MediaWiki-Vagrant[1] to create a portable production-like environment.
  This
  would give Mediawiki engineers a common ground from which to develop
 core
  code and/or extensions, and by coming close to mimicking Wikimedia's
  production environment, would hopefully reduce the amount of friction
  around getting features out to production. Also, this is something that
 we
  would be able to pre-package this for new engineers - imagine handing a
  new
  MediaWiki engineer a USB stick with an up-to-date MediaWiki-Vagrant
  instance that closely mimics production at say, a future hackathon.
 
  We started chatting about what it would take to get us there, and
  identified some initial steps that we'd like to tackle at the Zürich
  Hackathon - namely, turning a few puppetized production services into
  roles
  that we could use in MediaWiki-Vagrant.
 
  We've created a corresponding 'topic' on the Hackathon's topic page[2]
 to
  describe what we'd like to achieve at the Hackathon. Please review,
  comment, and certainly add yourself as an 'interested person' if this
  catches your fancy and you plan to attend the hackathon.


I am keeping my options open for the hackathon but wanted to support this
effort enthusiastically.  And to mention that the analytics team loves
mediawiki-vagrant and tries to get most of its projects integrated with it
as roles.

One thing I'd be interested in is to hear how Linux folks are installing
vagrant / virtualbox.  I've not been able to vagrant up my
mediawiki-vagrant since I upgraded to Ubuntu 13.10 and I've heard of at
least two other people with the same experience.   Maybe 14.04 will just
solve all that, but it's worth looking into so we can post any gotchas in
the install instructions.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Proposal for Zürich Hackathon - getting close to a production-like Vagrant instance

[Dan Andreescu]

On Tue, Feb 11, 2014 at 2:51 PM, Brad Jorsch (Anomie) bjor...@wikimedia.org
 wrote:

 On Tue, Feb 11, 2014 at 2:33 PM, Dan Andreescu dandree...@wikimedia.org
 wrote:

  One thing I'd be interested in is to hear how Linux folks are installing
  vagrant / virtualbox.  I've not been able to vagrant up my
  mediawiki-vagrant since I upgraded to Ubuntu 13.10 and I've heard of at
  least two other people with the same experience.   Maybe 14.04 will just
  solve all that, but it's worth looking into so we can post any gotchas in
  the install instructions.
 

 I can't comment about Ubuntu, but using Debian Sid I was finally able to
 get it to work this morning.

 All I had to do was apt-get install virtualbox-dkms and vagrant. Versions
 here are 4.3.2-dfsg-1 for virtualbox, 1.4.3-1 for vagrant, and 3.12.9-1 for
 the kernel.


Thanks Brad.  Those versions didn't work for me so I finally stopped being
lazy and looked into it.  It was user error (well, sort of).  VT-x was
disabled in my BIOS.  Added a note in the README:
https://gerrit.wikimedia.org/r/#/c/112800/
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Action Script API client library?

[Dan Andreescu]

On Tue, Feb 11, 2014 at 4:18 PM, Brandon Harris bhar...@wikimedia.orgwrote:


 Both ActionScript and JavaScript are ECMAScript languages and are
 thus pretty similar.  I last did AS coding about 4 years ago but I don't
 think the language has changed significantly since then.

 I'd be surprised if there was an ActionScript library for
 MediaWiki, so your best bet is JavaScript.


When I used ActionScript it was fairly easy to communicate back and forth
with JavaScript.  I'd use the JavaScript API and talk to it from
ActionScript.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] InstantClick

[Dan Andreescu]

 Hi,

 Today, I heard about a JavaScript library called InstantClick
 (http://instantclick.io/). Basically, it's based on the principle that
 latency is responsible for a lot of the Web's slowness. It also
 considers that there are about 250ms between hovering over and
 clicking on a link. Therefore, it starts pre-loading the page on
 hover, and then switches to it via AJAX when the user clicks the link.
 It can also do this on mousedown only, which causes no additional
 server load and still provides a performance boost, according to its
 website, similarly to Rails' turbolinks functionality.

 Is there any chance this could work on MediaWiki?

 Regards,
 -Kudu.


This is pretty neat.  Some a/b tests on mobile would help us understand
just how much extra data this would cause people to consume vs. how much
time it saves them.

On desktop, I bet a combination of this with something like cursor
proximity based events [1] would be interesting to look into.


[1] https://github.com/padolsey/jquery.fn/tree/master/proximity-event
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l