Re: [Wikitech-l] Google Code-in 2016 just started and we need your help!

[John Mark Vandenberg]

What happens if a mentor doesn't respond with 36 hrs?
The weekend is going to be the most problematic stage, especially this
first weekend.

Does this "36 hrs" also apply to reviewing in Gerrit and comments in
Phabricator?

I am seeing students not submit their GCI task for approval once they
have uploaded their patch into Gerrit.  This is actually "good" in one
sense, as they haven't finished the task until it is merged, but if we
are not tracking Gerrit review times of GCI patches, it means the
mentors are not obligated to do code review with 36 hrs and the
participants are surprised at how long the code review phase is
taking. (48 hrs in one case).

Is there any warning when the 36 hour limit is approaching?

Are Wikimedia org admins watching this limit somehow?
Is there some process in place?
e.g. 24 hr "this is getting worrying" status, where we find another
mentor / code reviewer?


On Tue, Nov 29, 2016 at 12:00 AM, Andre Klapper  wrote:
> Google Code-in 2016 just started:
> https://www.mediawiki.org/wiki/Google_Code-in_2016
>
> In the next seven weeks, many young people are going to make their
> first contributions to Wikimedia. Expect many questions on IRC and
> mailing lists by onboarding newcomers who have never used IRC or lists
> before. Your help and patience is welcome to provide a helping hand!
>
> Thanks to all mentors who have already registered & provided tasks!
>
> You have not become a mentor yet? Please do consider it.
> It is fun and we do need more tasks! :)
>
> * Think of easy tasks in your area that you could mentor.
>   Areas are: Code, docs/training, outreach/research, quality
>   assurance, and user interface. "Easy" means 2-3h to complete for
>   you, or less technical ~30min "beginner tasks" for onboarding).
> * OR: provide an easy 'clonable' task (a task that is generic and
>   could be repeated many times by different students).
> * Note that you commit to answer to students' questions and to
>   evaluate their work within 36 hours (but the better your task
>   description the less questions. No worries, we're here to help!)
>
> For the full info, please check out
> https://www.mediawiki.org/wiki/Google_Code-in_2016/Mentors
> and ask if something is unclear!
>
> Thank you again for giving young contributors the opportunity to learn
> about and work on all aspects of Free & Open Source Software projects!
>
> Cheers,
> andre
> --
> Andre Klapper | Wikimedia Bugwrangler
> http://blogs.gnome.org/aklapper/
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Update on WMF account compromises

[John Mark Vandenberg]

On Mon, Nov 21, 2016 at 10:42 PM, Bartosz Dziewoński
 wrote:
> Just for the record, I'm not having such problems, so it might be in some
> way specific to you. I've heard someone else recently complaining about
> getting logged in often, I don't think this is related to 2FA.
>
> If you need to disable it, you can do it yourself (visit Preferences, click
> "Disable two-factor authentication" and follow the steps).

I switch devices regularly, and switch browsers also.
Desktop session continues without a hitch, mostly.
Mobile devices are always being logged out.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Update on WMF account compromises

[John Mark Vandenberg]

Ya, this is why I haven't done it.

Also, I should be able to set it up such that TFA is not necessary
until my account attempts to do an admin action.

On Mon, Nov 21, 2016 at 4:37 PM, Florence Devouard  wrote:
> Hello
>
> I had the super bad idea of implementing the two-factor authentication and
> now I need help :)
>
> The system is not "recording" me as registered. Which means that I am
> disconnected every once in a while. Roughly every 15 minutes... and every
> time I change project (from Wikipedia to Commons etc.)
>
> Which means that every 15 minutes, I need to relogin... retype login and
> password... grab my phone... wake it up... launch the app... get the
> number... enter it... validate... OK, good to go for 15 minutes...
>
> So... how do I fix that ?
>
> Thanks
>
> Florence
>
>
> Le 16/11/2016 à 10:57, Tim Starling a écrit :
>>
>> Since Friday, we've had a slow but steady stream of admin account
>> compromises on WMF projects. The hacker group OurMine has taken credit
>> for these compromises.
>>
>> We're fairly sure now that their mode of operation involves searching
>> for target admins in previous user/password dumps published by other
>> hackers, such as the 2013 Adobe hack. They're not doing an online
>> brute force attack against WMF. For each target, they try one or two
>> passwords, and if those don't work, they go on to the next target.
>> Their success rate is maybe 10%.
>>
>> When they compromise an account, they usually do a main page
>> defacement or similar, get blocked, and then move on to the next target.
>>
>> Today, they compromised the account of a www.mediawiki.org admin, did
>> a main page defacement there, and then (presumably) used the same
>> password to log in to Gerrit. They took a screenshot, sent it to us,
>> but took no other action.
>>
>> So, I don't think they are truly malicious -- I think they are doing
>> it for fun, fame, perhaps also for their stated goal of bringing
>> attention to poor password security.
>>
>> Indications are that they are familiarising themselves with MediaWiki
>> and with our community. They probably plan on continuing to do this
>> for some time.
>>
>> We're doing what we can to slow them down, but admins and other users
>> with privileged access also need to take some responsibility for the
>> security of their accounts. Specifically:
>>
>> * If you're an admin, please enable two-factor authentication.
>> 
>> * Please change your password, if you haven't already changed it in
>> the last week. Use a new password that is not used on any other site.
>> * Please do not share passwords across different WMF services, for
>> example, between the wikis and Gerrit.
>>
>> (Cross-posted to wikitech-l and wikimedia-l, please copy/link
>> elsewhere as appropriate.)
>>
>> -- Tim Starling
>>
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Engaging student devs with another outreach event like GSoC, GCI

[John Mark Vandenberg]

This is probably a good thread to introduce a program we have been
running in Indonesia called Besut Kode, funded by Ford Foundation.

We noticed there were not many GCI/GSOC participants from Indonesia,
and also not many Wikimedia devs from Indonesia, and are trying to fix
that.

We are using a competitive training program format, with eliminations
and prizes, to ensure that we spend more time mentoring the
participants with the most potential to succeed in the OSS world.

The project has two halves, the first targeting high school (called
SMA) students preparing them for GCI, and then the second targeting
University students preparing them for GSOC.  Between them we have had
almost 1000 registrations.

The participant list for both is at:
https://github.com/BesutKode/BesutKode.github.io

The high school program is entirely in private repositories, allowing
the students to make all kinds of mistakes, and learn from them, with
the goal being to submit a real significant patch an to OSS project.
Six have finished the program, with merged contributions to real OSS
projects, and a few more are likely to finish it before GCI starts,
but have struggled to fit it in with their other activities.

The program for university students is more public.  The English
version of the program is at

http://wikimedia-id.github.io/besutkode/university-modules-en.html

One of the features is that we eliminate participants if they are not
active on GitHub every three days, requiring that they complete a
small patch to a pre-selected set of repositories that have increasing
difficulty and slowly moving them more towards GSOC relevant
repositories.

http://wikimedia-id.github.io/besutkode/university-activity-repositories-en.html

You can see their ongoing activity at http://tinyurl.com/bku-other-repos .

In addition, they have to work on some quite difficult tasks, which
they can work on together but must have distinct solutions.

The first of these large tasks is public at
https://github.com/BesutKode/uni-task-1

So far, nine participants have completed that task and are now working
on the second task.

https://github.com/orgs/BesutKode/teams/peserta-universitas-task-2

All of the program materials will be public and CC-BY after the
competition is over, as is required by all Ford grants.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] CI is having issues, hang with us

[John Mark Vandenberg]

On Thu, Aug 11, 2016 at 10:05 AM, John Mark Vandenberg <jay...@gmail.com> wrote:
> On Thu, Aug 11, 2016 at 7:30 AM, Greg Grossmeier <g...@wikimedia.org> wrote:
>> Things are back.
>>
>> Apologies.
>
> I'm noticing long queues for pywikibot; ~19 mins before the jobs started.
>
> The Zuul graphs seem to imply the delays are being felt elsewhere also.
>
> https://integration.wikimedia.org/zuul/

Ah, looks like the workers have caught up, and now the builds are
starting normally.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Engineering] CI is having issues, hang with us

[John Mark Vandenberg]

On Thu, Aug 11, 2016 at 7:30 AM, Greg Grossmeier  wrote:
> Things are back.
>
> Apologies.

I'm noticing long queues for pywikibot; ~19 mins before the jobs started.

The Zuul graphs seem to imply the delays are being felt elsewhere also.

https://integration.wikimedia.org/zuul/

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Number of differences

[John Mark Vandenberg]

On Wed, Aug 3, 2016 at 4:44 AM, Strainu  wrote:
> Hi,
>
> Short answer: The API does not seem to directly provide such info, but
> you could use action=compare, then search for class=\"diff-lineno\"
> and divide by 2 (it appears once on each side of the diff).
>
> Do note that the question you ask is not very well defined. The method
> above will give you the number of chunks the diff engine displays,
> which can be different from what you consider as a modification. For
> instance 
> https://ro.wikipedia.org/w/api.php?action=compare=9648318=10670757
> shows 1 chunk, but there is a modified line and an added line (the
> category), which you might consider as 2 different modifications. You
> need to clearly set the expectations before searching for a solution.

fwiw, Pywikibot has an APISite method "compare" to fetch the MediaWiki diff

https://doc.wikimedia.org/pywikibot/api_ref/pywikibot.html#pywikibot.site.APISite.compare

and another function "html_comparator" to convert the MediaWiki diff
into something a bit more usable for simple uses

https://doc.wikimedia.org/pywikibot/api_ref/pywikibot.html#pywikibot.diff.html_comparator

We have one unit test that puts these two functions together in a
hopefully understandable fashion

https://github.com/wikimedia/pywikibot-core/blob/master/tests/diff_tests.py#L87

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Loosing the history of our projects to bitrot. Was: Acquiring list of templates including external links

[John Mark Vandenberg]

On Tue, Aug 2, 2016 at 8:34 AM, Gergo Tisza  wrote:
> On Mon, Aug 1, 2016 at 5:27 PM, Rob Lanphier  wrote:
>
>> Do you believe that declaring "the implementation is the spec" is a
>> sustainable way of encouraging contribution to our projects?
>
>
> Reimplementing Wikipedia's parser (complete with template inclusions,
> Wikidata fetches, Lua scripts, LaTeX snippets and whatever else) is
> practically impossible. What we do or do not declare won't change that.

Correct, re-implementing the MediaWiki parser is a mission from hell.
And yet, WMF is doing that with parsoid ... ;-)
And, WMF will no doubt do it again in the future.
Changing infrastructure is normal for systems that last many generations.

But the real problem of not using a versioned spec is that nobody can
reliably do anything, at all, with the content.

Even basic tokenizing of wikitext has many undocumented gotchas, and
even with the correct voodoo today there is no guarantee that WMF
engineers wont break it tomorrow, and not inform everyone that the
spec has changed.

> There are many other, more realistic ways to encourage contribution by
> users who are interested in wikis, but not in Wikimedia projects.
> (Supporting Markdown would certainly be one of them.) But historically the
> WMF has shown zero interest in the wiki-but-not-Wikimedia userbase, and no
> other actor has been both willing and able to step up in its place.

The main reason for a spec should be the sanity of the Wikimedia
technical user base, including WMF engineers paid by donors, who build
parsers in other languages for various reasons, including supporting
tools that account for a very large percent of the total edits to
Wikimedia and are critical in preventing abuse and assisting admins
performing critical tasks to keep the sites from falling apart.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Loosing the history of our projects to bitrot. Was: Acquiring list of templates including external links

[John Mark Vandenberg]

There is a slow moving discussion about this at
https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Markdown

The bigger risk is that the rest of the world settles on using
CommonMark Markdown once it is properly specified.  That will mean in
the short term that MediaWiki will need to support Markdown, and
eventually it would need to adopt Markdown as the primary text format,
and ultimately we would loose our own ability to render old revisions,
because the parser would bit rot.

One practical way to add more discipline around this problem is to
introduce a "mediawiki-wikitext-announce" list, similar to the
mediawiki-api-announce list, and require that *every* breaking change
to the wikitext parser is announced there.

wikitext is file format, and there are alternative parsers, which need
to be updated any time the Php parser changes.

https://www.mediawiki.org/wiki/Alternative_parsers

It should be managed just like the MediaWiki API, with appropriate
notices sent out, so that other tools can be kept up to date, and so
there is an accurate record of when breaking changes occurred.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Gerrit 2.12.2 test instance - PLEASE TEST

[John Mark Vandenberg]

On Wed, Jul 13, 2016 at 10:03 AM, Chad  wrote:
> On Tue, Jul 12, 2016 at 7:39 PM Danny B.  wrote:
>> I assume that the slowness is just because it is testing environment and
>> production will be faster, but just in case, I'm mentioning that too (ask
>> for details should you need any).
>>
>>
> I'm not seeing slowness myself, but I have heard this complaint from a
> few others. For what it's worth, this *is* running on production hardware
> with higher specs than the old machine, so if anything it should be faster!
> There's some new features and other things we're not quite making use
> of yet, so we might need to do some further fine-tuning.

My laptop (Intel i3-2310M CPU @ 2.10GHz × 4) with Firefox on Ubuntu
isn't liking the extra client side workload being asked of it.

I'll survive.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Phabricator monthly statistics - 2016-05

[John Mark Vandenberg]

Hi Alex,

That increase could be a result of GSOC projects starting, which must
create boat loads of tasks in the first month to plot the path of
their project.

On Wed, Jun 1, 2016 at 7:24 AM, Alex Monk  wrote:
> If you look back through all the previous monthly statistics emails,
> account creations regularly go above 300 per month. Phabricator account
> creations are done through first MediaWiki logins via OAuth or LDAP, yes.
>
> Here are the numbers that I'd like to draw people's attentions to:
> Tasks created in (2016-05): 2572
> Tasks closed in (2016-05): 2275
> That's a difference of 297 tasks...
> Difference (2016-05): 297
> Difference (2016-04): 243
> Difference (2016-03): 454
> Difference (2016-02): 238
> Difference (2016-01): 418
> These difference should really be negative, but right now it looks like
> we're slowly accumulating more and more bugs...
>
> On 1 June 2016 at 01:05, Pine W  wrote:
>
>> I'm impressed by how many new accounts were created; 256 in a single month.
>> I'd be interested to know what typically prompts people to create accounts.
>> Also, does the 256 number include people who use their MediaWiki
>> credentials to log into Phabricator for the first time?
>>
>> Just curious,
>> Pine
>>
>> On Tue, May 31, 2016 at 5:00 PM,  wrote:
>>
>> >
>> > Hi Community Metrics team,
>> >
>> > This is your automatic monthly Phabricator statistics mail.
>> >
>> > Accounts created in (2016-05): 256
>> > Active users (any activity) in (2016-05): 861
>> > Task authors in (2016-05): 484
>> > Users who have closed tasks in (2016-05): 260
>> >
>> > Projects which had at least one task moved from one column to another on
>> > their workboard in (2016-05): 0
>> >
>> > Tasks created in (2016-05): 2572
>> > Tasks closed in (2016-05): 2275
>> > Open and stalled tasks in total: 30026
>> >
>> > Median age in days of open tasks by priority:
>> >
>> > Unbreak now: 19
>> > Needs Triage: 170
>> > High: 280
>> > Normal: 433
>> > Low: 743
>> > Lowest: 555
>> >
>> > (How long tasks have been open, not how long they have had that priority)
>> >
>> > TODO: Numbers which refer to closed tasks might not be correct, as
>> > described in https://phabricator.wikimedia.org/T1003 .
>> >
>> > Yours sincerely,
>> > Fab Rick Aytor
>> >
>> > (via community_metrics.sh on iridium at Wed Jun  1 00:00:10 UTC 2016)
>> >
>> > ___
>> > Wikitech-l mailing list
>> > Wikitech-l@lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Should we switch the default category collation to uca-default?

[John Mark Vandenberg]

Yes, of course, & a meta discussion will likely unearth many reasons to
opt-out ;)

Does uca (or extension) do the right thing for West Frisian (fy) wrt y & i ?

Or, ... it would be helpful to put the list of 94 wiki somewhere easy to
consume.
On 28 May 2016 06:37, "Ryan Kaldari"  wrote:

> There are currently 94 WMF wikis using UCA category collation rather than
> the default "uppercase" collation. The Unicode Collation Algorithm (UCA) is
> the official standard for how to sort Unicode characters, and generally
> follows how a human would typically alphabetize strings. For example,
> uppercase collation sorts Aztec, Ärsenik, Zoo, Aardvark as "Aardvark,
> Aztec, Zoo, Ärsenik", but uca-default collation sorts them as "Aardvark,
> Ärsenik, Aztec, Zoo". UCA collation also (optionally) supports natural
> numeric sorting so that 100, 1, 99 sorts as "1, 99, 100" rather than "1,
> 100, 99". The WMF Community Tech team has recently posted proposals on
> English Wikipedia and several Wiktionaries asking if these communities
> would support switching to UCA collation. The proposal on English Wikipedia
> has received unanimous support so far.[1] We thought that Wiktionaries
> would be more skeptical of the change, but so far we have received only
> positive responses.[2]
>
> Since it seems that most wikis are receptive to switching to UCA, maybe we
> should just make it the default rather than waiting on all the wikis to
> request it separately. Of the large Wikipedias, French, Dutch, Polish,
> Portuguese, and Russian are already using UCA, and German is in the process
> of switching.[3] For non-Latin scripts, my understanding is that UCA will
> be a big improvement, especially if we switch them to language-specific
> implementations, like uca-ja, uca-zh, uca-ar, etc.
>
> Three questions:
> 1. Does switching the default collation from "uppercase" to "uca-default"
> sound like a good idea?
> 2. Should this be proposed on meta or is it too technical?
> 3. Are there any wikis that would need to opt out of this for some reason?
> (I know there are issues with Kurdish,[4] but that's the only one I know
> about.)
>
> 1.
>
> https://en.wikipedia.org/wiki/Wikipedia_talk:Categorization#OK_to_switch_English_Wikipedia.27s_category_collation_to_uca-default.3F
> 2. https://phabricator.wikimedia.org/T128502
> 3. https://phabricator.wikimedia.org/T128806
> 4. https://phabricator.wikimedia.org/T48235
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Best practice for WIP patches to help code review office hours

[John Mark Vandenberg]

On Wed, May 18, 2016 at 10:41 PM, Chad  wrote:
> On Wed, May 18, 2016 at 8:35 AM Daniel Kinzler 
> wrote:
>
>> Am 13.05.2016 um 18:23 schrieb Greg Grossmeier:
>> > 
>> >> Gerrit also has drafts...
>> >
>> > Drafts are only visible to the author, unfortunately.
>>
>> And anyone the author adds as a reviewer. Works nicely in my experience.
>>
>>
> Which I guess is nice if you really only want a specific group of people to
> see it, but it keeps the casual reviewer from chiming in.
>
> This is why they seemed useful for security patches at first, except that
> they're disclosable over `git fetch` if you know where to look.
>
> So yeah, I'm convinced they're defective by design as they're both too
> secret and not secret enough at the same time.

For non-security patches, discovery isnt a problem.
They are cookie-licked bugs, and should already been discoverable via
a link on a Phabricator task.
If there is no Phabricator task, they are an unloved patch for an
undefined problem.
Hiding them as a draft un-licks them.

But that depends on https://phabricator.wikimedia.org/T63124 , and
probably an upgrade of Gerrit, which will take forever because of the
plan to switch to using Phabricator for code review.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Proposal to invest in Phabricator Calendar

[John Mark Vandenberg]

On Tue, May 17, 2016 at 4:42 AM, Federico Leva (Nemo)
 wrote:
>> That said, I think we should be careful with our assumptions about how
>> much influence we can buy with the money we have.
>
> Sure. Let's not make assumptions at all then: what makes someone think that
> calendar is amenable to WMF-mandated development? Already one year ago, I
> proposed that Phacility be hired to upstream our issues (and triage them
> upstream).
> https://lists.wikimedia.org/pipermail/teampractices/2015-March/000642.html
>
> The reason is that so far I'm not aware of a single person in Wikimedia
> being able to talk with upstream (as opposed to talking past each other).
> https://phabricator.wikimedia.org/project/board/6/query/all/ seems to prove
> none exists, as the "Solved upstream" column lists a whopping 2 issues out
> of 500+ upstream issues.

And of those two tasks, one is a regression and the other is an
exception. i.e. not functional improvements.

I have added T109956 to that column, and would do more to help
categorisation of solved tasks, but I note that "Solved upstream" is
not explained at the documentation
https://www.mediawiki.org/wiki/Phabricator/Code , so I wonder if I am
doing something wrong.

It should be noted that we have successful built some local
extensions/customisations, such as the Sprint.
If we are having difficulty providing improvements to the core
product, then funding incomplete extensions (like Calendar) is a good
approach.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Reducing the environmental impact of the Wikimedia movement

[John Mark Vandenberg]

On Mon, May 16, 2016 at 1:42 AM, Tim Landscheidt  
wrote:
> Lukas Mezger  wrote:
>
>> With the help of Juliet Barbara and Gregory Varnum, we now have detailed
>> public figures regarding the energy use and energy sources of the Wikimedia
>> servers: As of May 2016, the servers use 222 kW, summing up to about 2 GWh
>> of electrical energy per year. For more information, please see
>> https://meta.wikimedia.org/wiki/Environmental_impact.
>
>> The next step would be to figure out the cost and feasibility of having the
>> servers run on 100% renewable energy. I'd appreciate it if someone could
>> help me find out how this works. As a European consumer, I can order
>> renewable energy for my house simply by calling my energy company on the
>> phone, with the price difference being negligible. I assume it is not just
>> as easy in our case, right?
>
> At Hawaii consumer prices, 2 GWh equals less than
> US-$ 800,000; that would be roughly 1 % of the Wikimedia
> Foundation budget.  Don't you think it would be much better
> for *actually* reducing the environmental impact to start on
> the 99 % (or probably more like 99.5 %)?  It would certainly
> be cheaper than paying *more* for energy.

What is an energy consumption estimate of the other 99% of budget expenditure?

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] PyWikibot reviewers needed!

[John Mark Vandenberg]

No!
Quality rapidly decreases when you force unpaid people to rush reviewing so
the queue looks better.
Last time people rushed pywikibot reviewers, lots of junk was committed,
creating lots of bugs, and the tree has been red for months now, making it
more difficult to merge old large changesets. Busy work. By unpaid people.
Because paid people complain about the queue.
We recently lost a very good reviewer & +2'er, putting a large spanner in
the system. But other than that you will see pywikibot has been very active
.
On 13 May 2016 04:17, "Jon Robson"  wrote:

> I'm keen for us give attention to people who have open patchsets that
> they are looking for review. We had the first code review office hours
> today and I hope in the long term during this hour we can work on
> reviewing patchsets from this search:
>
> https://gerrit.wikimedia.org/r/#/q/(+label:Code-Review%253D0+OR+NOT+(label:Code-Review%253D-1+OR+label:Code-Review%253D-2)+),n,z
>
> Pywikibot has 145 open patches on Gerrit.
> You can see them all here:
>
> https://gerrit.wikimedia.org/r/#/q/status:open+project:pywikibot/core+(+label:Code-Review%253D0+OR+NOT+(label:Code-Review%253D-1+OR+label:Code-Review%253D-2)+),n,z
>
> Can someone who is familiar with this codebase commit to helping
> getting all these reviewed?
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikimediaindia-l] How to click custom html option button via api?

[John Mark Vandenberg]

Pywikibot has a class ProofreadPage which allows modification of level via
simple attributes and methods.

https://github.com/wikimedia/pywikibot-core/blob/master/pywikibot/proofreadpage.py#L67

If you dont want to use Pywikibot, its code can be helpful and it can be
copied using very unrestrictive MIT license (and the authors would likely
be happy to relicense it to another license if that was useful.

Regards,
John Vandenberg
On 27 Apr 2016 21:21, "Shrinivasan T"  wrote:

> I am trying to work on https://ta.wikisource.org with python API.
>
> This tamil wikisource has custom html option buttons to mention the edit
> status.
>
> See a screenshot here.
> https://snag.gy/f3aTBn.jpg
>
> How to enable these custom option buttons via API?
>
> Thanks.
>
> --
> Regards,
> T.Shrinivasan
>
>
> My Life with GNU/Linux : http://goinggnu.wordpress.com
> Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com
>
> Get Free Tamil Ebooks for Android, iOS, Kindle, Computer :
> http://FreeTamilEbooks.com
>
> ___
> Wikimediaindia-l mailing list
> wikimediaindi...@lists.wikimedia.org
> To unsubscribe from the list / change mailing preferences visit
> https://lists.wikimedia.org/mailman/listinfo/wikimediaindia-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [WikimediaMobile] The future of Related Pages feature

[John Mark Vandenberg]

This feature appears to be an automated edition of the "See also" section
on English Wikipedia. Having both Related Articles and See also feels like
a usability issue.

Has there been any discussions on the wikis about this overlap?
On 24 Mar 2016 05:18, "Moushira Elamrawy"  wrote:

> Hello everyone,
>
> Related pages feature has been in beta for over two months now,  the
> future of the feature depends on our discussions.  While we currently don't
> have a clear process for deciding collaboratively on an all languages
> product, Alsee and the reading team have put together this document on meta
> [0], as a request for comment, seeking comments and ideas on modifications
> required, and how to further test the feature.  In fact, we are not sure if
> an rfc is the best strategy to move forward with product decisions, but
> lets see how the discussion evolves, and we might explore the need for a
> different process, as we move on with this one.
>
> We managed to translate a brief introduction about the topic, please feel
> free to fully translate the document and/or further promote the discussion
> on your wiki.  We are trying hard to avoid having an English centric
> discussion for a feature that could be available across all language
> projects, and while we don't have a clear solution for this, we are trying
> this method as an experiment, where at least our communities can leave
> comments in their preferred language if they aren't comfortable writing in
> English but they can understand it.
>
> Please check the page, help with translation or promotion in your
> Wikipedia, and most importantly, comment on how you think it can evolve. :)
>
> Lets see how this works!
>
>
> All the best,
> M
>
> [0] https://meta.wikimedia.org/wiki/Requests_for_comment/Related_Pages
>
> ___
> Mobile-l mailing list
> mobil...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Small Amendment to User-Agent Policy

[John Mark Vandenberg]

On Tue, Mar 22, 2016 at 12:44 AM, Marcel Ruiz Forns
 wrote:
> ...
> I think adding the word bot to the user-agent of bot-like programs is a
> widely adopted convention. Actually, the word bot is already (for a long
> time now) being parsed and used to tag requests as bot-originated in our
> jobs that process requests into pageviews stats, because many external bots
> include it in their user-agent. See:
> http://www.useragentstring.com/pages/Crawlerlist/

The algorithm has been imperfect for a long time.  How long and how
imperfect doesnt matter.  Analytics is all about making good use of
imperfect algorithms to provide reasonable approximations.

However, I expect it is the role of Analytics is to improve the
definitions and implementation over time, not force a bad algorithms
into policy.

Pywiki*bot* has the string 'bot' in its useragent, because it is part
of the product name.
However, not all usage of Pywikibot is a crawler or even a bot, in any
sensible definition of those concepts.
Pywikibot is a *user agent* that knows how to be a client of the
*MediaWiki API*.  It can be used for "in-situ human consumption" or
not.

It is no different from a web browser in how it *may* be used,
although of course typically the primary goal of using Pywikibot
instead of a Web browser is to reduce the amount of human consumption
and decision making needed to perform a task.  But that is no
different to Gadgets written using the JavaScript libraries that run
in the Web browser.

It can function *exactly* like a web browser reading a special:search
results page, viewing some of those page in the search results, and
making edits to some of them.  Each page may be viewed by a real
human, who is making decisions throughout the entire process about
which pages to view and which pages to edit.

Or it can function *exactly* like a crawler, spider, bot, etc., with
zero human consumption.

Almost every script that is packaged with Pywikibot has an automatic
and non-automatic mode of operation.
Should we change our user-agent to "Pywikihuman" when in non-automatic
mode of operation, so that it isnt considered to be a bot by
Analytics?

Using the string 'bot' in the user-agent may be a useful approximation
for Analytics to use circa 2010, but it is bad policy, and Analytics
can and should do much better than that in 2016 now that API usage is
in focus.

> There is very little information at
>> https://meta.wikimedia.org/wiki/Research:Page_view or elsewhere (that
>> I can see) regarding what use of the API is considered to be a
>> **page** view.  For example, is it a page view when I ask the API for
>> metadata only of the last revision of a page -- i.e. the page/revision
>> text is not included in the response?
>
> You're right, and this is a very good question. I fear the only ways to
> look into this are browsing the actual code in:
> https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java

I am not very interested in the code, which is at best an attempt at
implementing the API page view definition.  I'd like to understand the
high level goal.

However, having read that file, and the accompanying test suite, it is
my understanding that there is no definition of an API page view.
i.e. all requests to api.php , excepting api.php usage by the
Wikipedia App (i.e. with user-agent "WikipediaApp", used by the iOS
and Android Apps), is classified as *not a page view*.

fwiw, rather than reading the source, this test data file with
expected results is a simpler way to see the current status.

https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/test/resources/pageview_test_data.csv

> or asking the Research team, who owns the definition.

Could the Research team please publish their definition of API
(api.php) page views, like they do for Web (index.php) page views.

Without this, it is hard to have a serious conversation about how
changing the user-agent policy might be helpful to achieve the goal of
better classifying API page views.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Small Amendment to User-Agent Policy

[John Mark Vandenberg]

On Mon, Mar 21, 2016 at 12:37 PM, Marcel Ruiz Forns
 wrote:
> Hi wikitech-l,
>
> After the discussion in analytics-l [1][2] and Phabricator [3], the
> Analytics team added a small amendment [4] to Wikimedia's user-agent policy
> [5] with the intention of improving the quality of WMF's pageview
> statistics.
>
> The amendment asks Wikimedia bot/framework maintainers to optionally add
> the word *bot* (case insensitive) to their user-agents. With that, the
> analytical jobs that process request data into pageview statistics will be
> capable of better identifying traffic generated by bots, and thus of better
> isolating traffic originated by humans (corresponding code is already in
> production [6]). The convention is optional, because modifications to the
> user-agent can be a breaking change.

As asked on the talk page over a month ago with no response...
https://meta.wikimedia.org/wiki/Talk:User-Agent_policy#bot

How does adding 'bot' help over and above including email addresses
and URLs in the User-Agent?
Are there significant cases of human traffic browsers including email
addresses and URLs in the User-Agent?

If not, I am struggling to understand how the addition of 'bot'
assists better isolating traffic originated by humans.

Or, is adding 'bot' an alternative to including email addresses and
URLs?  This will also introduce some false positives, as 'bot' is a
word and word-part with meanings other than the English meaning. See
https://en.wiktionary.org/wiki/bot ,
https://en.wiktionary.org/wiki/Special:Search/intitle:bot and
https://en.wiktionary.org/wiki/Talk:-bot#Fish_suffix

> Targets of this convention are: bots/frameworks that can generate Wikimedia
> pageviews [7] to Wikimedia sites and/or API and are not for in-situ human
> consumption. Not targets: bots/frameworks used to assist in-situ human
> consumption, and bots/frameworks that are otherwise well known and
> recognizable like WordPress, Scrapy, etc. Note that there are many editing
> bots that also generate pageviews, like when trying to copy content from
> one page to another the source page is requested and the corresponding
> pageview is generated.

I appreciate this attempt to classify devise a clearer "target" for
when a client needs to follow this new convention from the analytics
team, as requested during the discussion on the analytics list.

Regarding "Wikimedia pageviews [7] to Wikimedia sites and/or API ..
[7] https://meta.wikimedia.org/wiki/Research:Page_view;

There is very little information at
https://meta.wikimedia.org/wiki/Research:Page_view or elsewhere (that
I can see) regarding what use of the API is considered to be a
**page** view.  For example, is it a page view when I ask the API for
metadata only of the last revision of a page -- i.e. the page/revision
text is not included in the response?

"in-situ human consumption" is an interesting formula.
"in situ human" strongly implies a human is directly accessing the
content that caused the page view.

But how much 'consumption' is required?  This was briefly discussed
during the analytics list discussion, and it would be good to bring
the wider audience into this discussion.

Obviously 'Navigation popups'/Hovercards is definitely "in-situ human
consumption".

But what about gadgets Twinkle's "unlink" feature and Cat-a-lot (on
Wikimeda Commons)?  They do batch modifications to pages, and the
in-situ human does not see the pages fetched by the JavaScript.  Based
on your responses in analytics mailing list discussion, and this new
terminology "in-situ human consumption", I believe that these gadgets
would be considered subject to the bot user-agent policy.
It would be good to identify a list of gadgets which need to be
updated to comply with the new user agent policy.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Interest in contributing to Wikimedia Foundation

[John Mark Vandenberg]

On Fri, Mar 4, 2016 at 6:34 AM, Amate Yolande  wrote:
> Hello,
>
> My name is Yolande Amate from the University of Buea Cameroon. I am
> proficient in C, C++, PHP and Python. I am new to open-source and I would
> like to participate in the Google Summer of Code 2016 / Outreachy Round 12
> with the Wikimedia Foundation.
> I am interested in working on the project "Python library for quiz data,
> with serialisation".

Welcome and good luck.  I've added cc for the two main mailing lists
relevant for your project.

For anyone interested, this project is for Wikiversity & Pywikibot:

https://phabricator.wikimedia.org/T89761

As you know PHP, you may also want to look at adding GIFT support to
the Quiz extension, as that would also allow more interoperability
with other MOOC systems that support quiz functionality.
https://phabricator.wikimedia.org/T24475

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] bluejeans

[John Mark Vandenberg]

On Wed, Mar 2, 2016 at 1:59 PM, Erik Bernhardson
<ebernhard...@wikimedia.org> wrote:
> On Tue, Mar 1, 2016 at 9:00 PM, John Mark Vandenberg <jay...@gmail.com>
> wrote:
>> If BNJ isnt actually open source, here is an open source solution that
>> we could use and help fund as required (e.g. buying their commercial
>> offerings so that WMF Engineering/Ops doesnt need to support it)
>>
>> http://bigbluebutton.org/
>>
>> From the FAQ: If you have a session with 20 users and all share their
> webcam (yes, this is possible) will generate 400 streams

Do you mean 20 concurrent screen sharing is a requirement for any
replacement of Blue Jeans?
That sounds like an usual meeting ;-)
Anyway, in 2010, they had that configuration:
http://bigbluebutton.org/2010/11/22/193-simultaneous-users/

My guess is that it is better in 2016.  Anyway, best we ask them, if
we can give them a list of requirements and a budget.

http://blindsidenetworks.com/hosting

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] bluejeans

[John Mark Vandenberg]

On Wed, Mar 2, 2016 at 2:23 PM, Stas Malyshev  wrote:
> Hi!
>
>> If BNJ isnt actually open source, here is an open source solution that
>> we could use and help fund as required (e.g. buying their commercial
>> offerings so that WMF Engineering/Ops doesnt need to support it)
>>
>> http://bigbluebutton.org/
>
> I've checked their demo and it uses Flash. Which is very iffy from
> security standpoint, may lead to various issues on platforms that don't
> support it or where support is sketchy and is not a good idea in general
> long-term since Flash is on its way out as a technology.
>
> While I'm all for supporting open-source, both by using it and
> contributing to it, in this particular case it doesn't look like viable
> solution to me.

The demo has a html5 client, and the demo asks me whether I want to
use Flash or HTML5.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] bluejeans

[John Mark Vandenberg]

On Wed, Mar 2, 2016 at 4:21 AM, Brion Vibber  wrote:
> BlueJeans is "open source" though I can't offhand find their source code by
> googling for "bluejeans source". ;)

I've dug around a bit, and also cant find anything to suggest that
they are open sourcing any parts of their solution, or anything
resembling bjn providing development or support for existing open
source technology components.

Their own java classes are using namespace "vc.bjn", and googing that
doesnt lead to anything.

Here is their recent github organisation:

https://github.com/BlueJeansNetwork

This looks like an older one:

https://github.com/bluejeansnet

And here is the only use of "vc.bjn" in github, by an employee

https://github.com/Aldaviva/tailor

Other employees have "operations"-like repos

https://github.com/sczizzo

If the organisation is a credible contributor to open source, I havent
found it yet.  I hope that WMF included "open sourciness" in their
product selection criteria & evaluation, so their should be some
document describing how BJN is contributing to open source.  It would
be great if that can be shared.

> It's being used mostly for larger meetings because a) it has a larger limit
> for number of participants than Google Hangout and b) it seems to be more
> open than Google Hangout.
>...
> Can't answer on if anybody's paying anything as I don't know, but
> personally I would hope we are helping to fund their open source
> development. ;)

If BNJ isnt actually open source, here is an open source solution that
we could use and help fund as required (e.g. buying their commercial
offerings so that WMF Engineering/Ops doesnt need to support it)

http://bigbluebutton.org/

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Google Code-in 2015 is over. Congratulations everybody!

[John Mark Vandenberg]

On 29 Jan 2016 4:49 am, "Andre Klapper"  wrote:
>
> Google Code-in 2015 has come to an end.
>
> Thanks to our students for resolving 461 Wikimedia tasks. Thanks to our
> 35 mentors for being available, also on weekends & holidays. Thanks to
> everybody on IRC for your friendliness, patience, and help provided to
> new contributors.
>
> Some more achievements, apart from those already mentioned in
> https://lists.wikimedia.org/pipermail/wikitech-l/2015-December/084421.html
:
>
>  * The CommonsMetadata extension parses vcards in the src field
>  * The MediaWiki core API exposes "actual watchers" as in "action=info"
>  * MediaWiki image thumbnails are interlaced whenever possible
>  * Kiwix is installable/moveable to the SD card, automatically opens
>the virtual keyboard for "find in page", (re)starts with the last
>open article
>  * imageinfo queries in MultimediaViewer are cached
>  * Twinkle's set of article maintenance tags was audited and its XFD
>module has preview functionality
>  * The RandomRootPage extension got merged into MediaWiki core
>  * One can remove items from Gather collections
>  * A new MediaWiki maintenance script imports content from text files
>  * Pywikibot has action=mergehistory support implemented

The same developer created the MediaWiki API endpoint mergehistory. Im not
sure whether it has been merged yet.

The big pywikibot bullet point is all use of  urllib2/httplib has been
replaced with requests.

The other significant pywikibot changes are:
* customisable fake user-agent used for web scrapping metadata of references
* deprecation decorator automatically updating the published documentation.

--
John
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Google Code-in 2015 is over. Congratulations everybody!

[John Mark Vandenberg]

On Wed, Feb 3, 2016 at 11:36 AM, Brad Jorsch (Anomie)
<bjor...@wikimedia.org> wrote:
> On Tue, Feb 2, 2016 at 4:52 PM, John Mark Vandenberg <jay...@gmail.com>
> wrote:
>
>> The same developer created the MediaWiki API endpoint mergehistory. Im not
>> sure whether it has been merged yet.
>>
>
> https://gerrit.wikimedia.org/r/#/c/261617/
>
> Not quite. I found one bug while testing it earlier.

Thanks for checking Brad.  Then the mergehistory change to MediaWiki
core and Pywikibot shouldnt be highlighted for GCI until they are
merged.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Could someone relatively new to Python please QA the Accuracy review login framework?

[John Mark Vandenberg]

On Tue, Jan 19, 2016 at 4:22 PM, James Salsman  wrote:
>> Have you looked at using OAuth for authentication?
>
> Yes; the modules in use support OAuth but we made a conscious decision to
> support anonymity. Lack of anonymity can interfere with the operation of the
> reviewer reputation database.

I'd love to read the background discussion that led to that decision.

Could you identify which part of MediaWiki's OAuth implementation has
unacceptable problems regarding anonymity?

If you are setting high standards/promises in that regard, your
alternative implementation of user authentication will need to be
extremely carefully written (as will your entire codebase need very
good security auditing).

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Could someone relatively new to Python please QA the Accuracy review login framework?

[John Mark Vandenberg]

On Wed, Jan 13, 2016 at 9:25 AM, James Salsman  wrote:
>..
> please test her user authentication and login framework?

Have you looked at using OAuth for authentication?  There are numerous
OAuth providers, and using them removes the largest possible problem
from the app.

> It's built for PythonAnywhere because it shouldn't run on Wikimedia
> servers, because of the safe harbor DMCA provisions precluding editorial
> control by web hosts.

IMO it should be set up on Tool labs, where more people can play with
it.  It isnt editorial control if it uses logic to *identify*
potential problems in content.  That isn't exerting editorial control.
Editorial decisions are being made by reviewers who are not the WMF
webhost.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Image editor prototype

[John Mark Vandenberg]

On Wed, Dec 9, 2015 at 12:36 PM, Luis Villa  wrote:
> Very cool!
>
> Totally personal thing: a task I've done several times on-wiki is rotate an
> image by <90 degrees just to make it level (e.g., 1
> ,
> 2
> ).
> Any chance that is in the roadmap?

fwiw that is https://phabricator.wikimedia.org/T34875

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Yandex?

[John Mark Vandenberg]

On Wed, Nov 11, 2015 at 5:47 AM, Legoktm  wrote:
> On 07/02/2015 12:55 PM, Legoktm wrote:
>> On 07/01/2015 06:50 PM, Ricordisamoa wrote:
>>> Il 02/07/2015 03:28, Legoktm ha scritto:
 I noticed: "Yandex coming up soon!" under ContentTranslation. Are there
 more details about what this means?
>>>
>>> https://phabricator.wikimedia.org/T89844 I think
>>
>> Thanks for the pointer. After some more digging, I found
>> .
>>
>> So it appears that ContentTranslation will be contacting a third-party,
>> closed source service? Are users going to be informed that this is the
>> case? What data is being sent?
>>
>
> It appears[1] this has quietly gone ahead without any response here,
> which is disappointing.
>
> [1]
> https://www.mediawiki.org/w/index.php?title=Content_translation/Documentation/FAQ=1935992=1780766

As the user is isolated from the communication with Yandex , I don't
see it as a huge problem.  Using Qualtrics seems to be a much more
serious problem, and nobody seems to care about that.

Yandex is sort of similar to a "MP4 upload only" support, only without
the patent concerns.  Relying on it comes at the risk that the service
stops, but the free content created is not infected.  More likely,
Yandex will start asking WMF for money, and WMF decides to pay because
it is 'easier' than terminating using the service.

Anyway, I've added it to

https://meta.wikimedia.org/wiki/Open_source

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] AWS usage

[John Mark Vandenberg]

On Thu, Oct 8, 2015 at 1:54 AM, Petr Bena  wrote:
> Hi,
>
> Why do you even care? Is this question directed to foundation only or
> community developers as well? "Used by wikimedia" is very broad term.

Pywikibot uses Travis-CI, and I was looking at their S3 artefacts
integration for storage of coverage data.
It turns out to be quite a bit more expensive than I had assumed.
https://phabricator.wikimedia.org/T74863#1710830

Also of interest was that given some people are dead against allowing
Travis-CI to do builds for free, it appears parts of our community are
using AWS directly instead of Labs, which is not free.

I used the broad term "Wikimedia", as it is the broader community that
could be using Labs.  It seems to be worth understanding why they are
choosing to not use Labs, and how much it is costing "us".

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] AWS usage

[John Mark Vandenberg]

Is AWS (Amazon Web Services) being used by Wikimedia directly; US
Foundation, or by other affiliates?

While trawling around for AWS related tasks, I saw on T74501 that Sage
Ross' team was using AWS late last year, perhaps only temporarily due
to the bug.
https://phabricator.wikimedia.org/T74501
(I assume from the date the task was raised that it was a team at Wiki
Education Foundation).

All use of Travis-CI is indirectly using AWS.  Are there other indirect uses?

Is it still in use at all in a direct manner, i.e. using our own account?

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Add a Gerrit label "WIP" to mark changes as work in progress

[John Mark Vandenberg]

+1

This will be very useful for the Gerrit cleanup day!


On Fri, 11 Sep 2015 16:23 Florian Schmidt
 wrote:
>
> Hi,
>
> that sounds like really good idea. +1 for all your mentioned reasons! :)
>
> Another use case is the time, a project's reviewers needs to review a change 
> (I remember we have a statistic for it somewhere? But I found only [1]), 
> where WIP changes could be ignored (they usually don't get reviews but count 
> as changes, that aren't reviewed for a long time).
>
> [1] http://korma.wmflabs.org/browser/gerrit_review_queue.html
>
> Best,
> Florian
>
> -Original-Nachricht-
> Betreff: [Wikitech-l] Add a Gerrit label "WIP" to mark changes as work in 
> progress
> Datum: Thu, 10 Sep 2015 18:50:08 +0200
> Von: Tim Landscheidt 
> An: wikitech-l@lists.wikimedia.org
>
> Hi,
>
> currently change owners use various ways to mark changes
> that are not yet ready for review.  Recurring patterns are
> commit messages beginning with "[WIP]" or "DO NOT MERGE"
> and/or -1 votes by the change owner.  A common problem with
> these solutions is that they cannot be used in Gerrit
> searches, for example if someone is looking for open changes
> to review, they must filter the results manually.  This also
> affects scripts & Co. like the Wikimedia Dashboard at
> http://korma.wmflabs.org/browser/ that need to use
> heuristics to determine if a change is a work in progress
> and thus should be ignored for statistical purposes.
>
> There was a bug (https://phabricator.wikimedia.org/T52842)
> to implement a "Work in progress" button/status with the
> underlying goal to prevent dashboards/queues from the added
> noise of "draft changes"/"[WIP]" changes, but it was
> declined because a button is not going to be added.
>
> I want to suggest to add a new label "WIP", inspired by
> OpenStack's "Workflow" label.  Its "neutral" value is 0
> ("Ready for reviews").  If it is "voted" to -1 ("Work in
> progress"), the change cannot be submitted.  This vote will
> stick with new uploads until it is changed back to 0.
>
> For searches, this will allow Gerrit users to restrict
> search results by adding "label:WIP+0" to their filters.
>
> Untested, the change would be something like:
>
> | diff --git a/project.config b/project.config
> | index 151eebd..93291e1 100644
> | --- a/project.config
> | +++ b/project.config
> | @@ -12,6 +12,7 @@
> | owner = group ldap/ops
> | label-Code-Review = -2..+2 group Project Owners
> | label-Code-Review = -1..+1 group Registered Users
> | +   label-WIP = -1..+0 group Registered Users
> | create = group Project Owners
> | editTopicName = group Registered Users
> | viewDrafts = group JenkinsBot
> | @@ -78,6 +79,11 @@
> | value = +2 Looks good to me, approved
> | copyAllScoresOnTrivialRebase = true
> | copyAllScoresIfNoCodeChange = true
> | +[label "WIP"]
> | +   function = AnyWithBlock
> | +   value = -1 Work in progress
> | +   value =  0 Ready for reviews
> | +   copyMinScore = true
> |  [access "refs/meta/dashboards/*"]
> | create = group Project Owners
> | create = group platform-engineering
>
> Tim
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Wikidata API breaking changes

[John Mark Vandenberg]

The merged changeset included changes which were not advertised,
causing pywikibot to break.  See T110559

The wbgetentities JSON 'entities' is now an array/list of entities
instead of a mapping/dictionary of 'Qd' => entity.

On Fri, Aug 28, 2015 at 12:33 AM, Addshore  wrote:
> Hi all!
>
>
> We have some breaking API changes that will soon be deployed to wikidata.org.
>
>
> The deployment date should be: 9th September 2015 (just under 2 weeks)
>
>
> The change making the breaks an be found at:
>
> https://gerrit.wikimedia.org/r/#/c/227686/
>
>
> The breaking changes are:
>
>  - XML output aliases are now grouped by language
>  - XML output may no longer give elements when they are empty
>  - XML any claim, qualifier, reference or snak elements that had an
> '_idx' element will no longer have it
>  - ALL output may now give empty elements, ie. labels when an entity has none
>
>
> If you want to see a wikipage explaining these changes take a look at:
>
> https://www.wikidata.org/wiki/User:Addshore/API_Break_September_2015
>
>
> If you have any questions regarding these breaking changes please ask!
>
>
> Addshore
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upcoming SyntaxHighlight_GeSHi changes

[John Mark Vandenberg]

On Mon, Jul 13, 2015 at 1:30 PM, John Mark Vandenberg jay...@gmail.com wrote:
 On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh o...@wikimedia.org wrote:
 Hello,

 Over the course of the next two days, a major update to the
 SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
 change swaps geshi, the unmaintained PHP library which performs the lexical
 analysis and output formatting of code, for another library, called
 Pygments.

 The roll-out will remove support for 31 languages while adding support for
 several hundred languages not previously supported, including Dart, Rust,
 Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
 others. See https://people.wikimedia.org/~ori/geshi_changes.txt for a
 full list. The languages that will lose support are mostly obscure, with
 the notable exception of ALGOL68, Oz, and MMIX.

 I was surprised to find other languages not in your text file that
 appear to no longer be supported.

 I've gone through the geshi php files looking for assembler languages
 only so far:
 https://gerrit.wikimedia.org/r/224379

 How/Why were these excluded in your list?

I've encountered more of these on Wikipedia, so, ...

Here is a list of 59 geshi supported languages which were omitted from
the above list of 31 languages being de-supported by the switch to
Pygments.

4cs
6502kickass
6502tasm
aimms
apt_sources
autoconf
caddcl
cfdg
cuesheet
dcl
dcpu16
dcs
epc
ezt
f1
falcon
fo
freeswitch
gdb
genero
genie
gettext
hicest
hq9plus
icon
inno
intercal
ispfpanel
jcl
kixtart
klonec
klonecpp
lb
lotusformulas
lotusscript
lscript
lsl2
m68k
magiksf
nagios
objeck
oxygene
parasail
per
pf
pixelbender
powerbuilder
proftpd
providex
rbs
scl
spark
stonescript
uscript
vbscript
vedit
whitespace
xorg_conf
z80

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upcoming SyntaxHighlight_GeSHi changes

[John Mark Vandenberg]

On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh o...@wikimedia.org wrote:
 Hello,

 Over the course of the next two days, a major update to the
 SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
 change swaps geshi, the unmaintained PHP library which performs the lexical
 analysis and output formatting of code, for another library, called
 Pygments.

 The roll-out will remove support for 31 languages while adding support for
 several hundred languages not previously supported, including Dart, Rust,
 Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
 others. See https://people.wikimedia.org/~ori/geshi_changes.txt for a
 full list. The languages that will lose support are mostly obscure, with
 the notable exception of ALGOL68, Oz, and MMIX.

I was surprised to find other languages not in your text file that
appear to no longer be supported.

I've gone through the geshi php files looking for assembler languages
only so far:
https://gerrit.wikimedia.org/r/224379

How/Why were these excluded in your list?

I've found reasonable fallbacks for many of the other unsupported
languages, and as previously mentioned they have added support for
some of these languages.
Is Wikimedia going to only use the official releases of Pygments?
Their last release was six months ago.
Is someone in communication with them about pushing out a new release?
If they dont release them soon, will Wikimedia maintain its own
version of the software?

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month

[John Mark Vandenberg]

On Wed, Jun 24, 2015 at 2:43 AM, Andre Klapper aklap...@wikimedia.org wrote:
 As there are seven days left, does someone have time / capacity to
 provide an updated list of bots that likely haven't seen updates yet?

 Also, in [1], Sitic came up with a query of de.wp *gadgets* affected:
 https://de.wikipedia.org/w/index.php?title=Spezial:Suchelimit=100offset=0ns2=1ns8=1search=%22query-continue%22+-%22rawcontinue%22+intitle%3A*.js

 Has there been on-wiki outreach to *gadget* maintainers?

 Thanks in advance!
 andre

 [1] 
 https://de.wikipedia.org/wiki/Wikipedia:Bots/Notizen#Hinweis_an_alle_Bot-Betreiber_.28API-.C3.84nderung.29

Did the change go live?  Did the wikis fall over?

Is there any ongoing analysis of the number of user agents (bots /
gadgets / etc) still not specifying continue method?

Is there a common approach for gadgets to identify themselves to the
API - e.g. setting a custom user-agent?

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Upcoming SyntaxHighlight_GeSHi changes

[John Mark Vandenberg]

On Tue, Jun 23, 2015 at 10:48 AM, Ori Livneh o...@wikimedia.org wrote:
 Hello,

 Over the course of the next two days, a major update to the
 SyntaxHighlight_GeSHi extension will be rolled out to Wikimedia wikis. The
 change swaps geshi, the unmaintained PHP library which performs the lexical
 analysis and output formatting of code, for another library, called
 Pygments.

 The roll-out will remove support for 31 languages while adding support for
 several hundred languages not previously supported, including Dart, Rust,
 Julia, APL, Mathematica, SNOBOL, Puppet, Dylan, Racket, Swift, and many
 others. See https://people.wikimedia.org/~ori/geshi_changes.txt for a
 full list.

A very welcome list of additions!

 The languages that will lose support are mostly obscure, with
 the notable exception of ALGOL68, Oz, and MMIX.

There are a few more exceptions, such as bnf, dot, pcre, email, and bibtex.

Hmm, algol support was recently added to Pygments.

https://bitbucket.org/birkenfeld/pygments-main/issue/1090/update-to-m2-lexer-in

Perhaps it needs to be backported into their stable branch, and a new
minor release pushed out for use on Wikimedia?

Since it is short, here is the full list of languages being de-supported.

6502acme
68000devpac
algol68
arm
avisynth
bibtex
bnf
cil
dot
e
email
euphoria
gml
ldif
lolcode
mirc
mmix
mpasm
oz
parigp
pcre
pic16
pli
q
robots
sas
teraterm
typoscript
unicon
whois
xpp

Of the English Wikipedia articles about those concept that I have
looked at, so far they all use source with the appropriate language
set, so they will all regress down to plain monospaced text.

Have you identified how many times each of these have been used on
Wikimedia projects?  Perhaps that might help us identify the priority
of languages needing to be added to Pygments (and which ones are
entirely useless.

Some of them have bugs raised in Pygments
https://bitbucket.org/birkenfeld/pygments-main/issue/1024/add-dot-lexer-graphviz-language

 Lastly, the way the extension handles unfamiliar languages will change.
 Previously, if the specified language was not supported by the extension,
 instead of a code block, the extension would print an error message. From
 now on, it will simply output a plain, unhighlighted block of monospaced
 code.

Ugh, is there a way to configure pygments to have fallbacks for
languages which are substantially based on another?  e.g. xpp is
basically java, and looks quite good when I tested it on betalabs.  I
am sure that Pygments has some parser close to 'email', as they do
support a 'http session' language.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month

[John Mark Vandenberg]

On Fri, Jun 19, 2015 at 7:56 AM, S Page sp...@wikimedia.org wrote:
 On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle bgers...@wikimedia.org
 wrote:

 I guess it comes down to is this: if we're going to continue supporting
 old behavior, they should be accessible via the same old requests.  *This
 removes the need for existing clients to be updated in the first place*.
 If we eventually want to delete the old code keeping the old behavior
 separated from the new will make it clear  explicit what's being dropped
 and what to use instead. ...

 Lastly, changing the default behavior to make things sane for new
 developers is, IMO, a bad trade-off


 That seems the crux of it. Because the MediaWiki API isn't versioned (and
 there are good reasons for it), the recommended best practices form of an
 API request evolves, something like

 api.php?continue= formatversion=2 utf8= *your_actual_params*

Why doesnt formatversion=2 assume continue= and utf8= ?

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month

[John Mark Vandenberg]

On Thu, Jun 18, 2015 at 12:13 PM, Yuri Astrakhan
yastrak...@wikimedia.org wrote:
 On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg jay...@gmail.com
 wrote:


 The API currently emits a warning if a query continuation mode isnt
 selected.

 I guess on July 1 the API could emit an error, and not return any query
 data.
 Then the data isnt going to cause weird behaviour - it will break,
 properly.


 Hmm, this is actually an interesting idea - would it make sense to error on
 missing continue or rawcontinue for all action=query for about a month
 or two, so that everyone notices it right away and gets updated, and than
 resume with the new behavior?

Not 'all', please, but perhaps all queries where there is actual
continuation data like is used now for the existing warning.  I would
really to avoid the API failing at all, ever, for
?action=querymeta=siteinfo - that is how clients get the API
generator version, and only when we have the API generator version do
we know what features are supported by the API.

Or just not flick the switch on July 1, and only default to the new
continuation mode for formatversion=2.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month

[John Mark Vandenberg]

On Wed, Jun 17, 2015 at 9:15 PM, Marco mai...@live.de wrote:
 On Wed, 03 Jun 2015 18:29:08 +0700, John Mark Vandenberg wrote:

 On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie)
 bjor...@wikimedia.org wrote:
 ...
 I've compiled a list of bots that have hit the deprecation warning more
 than 1 times over the course of the week May 23–29. If you are
 responsible for any of these bots, please fix them. If you know who is,
 please make sure they've seen this notification. Thanks.

 Thank you Brad for doing impact analysis and providing a list of the 71
 bots with more than 10,000 problems per week.  We can try to solve those
 by working with the bot operators.

 If possible, could you compile a list of bots affected at a lower
 threshold - maybe 1,000.  That will give us a better idea of the scale
 of bots operators that will be affected when this lands - currently in
 one months time.

 Will the deploy date be moved back if the impact doesnt diminish by bots
 being fixed?

 Should someone contact those bots on their talk page to notify the owners or
 do we hope everyone reads this mailing list?

I see Whatamidoing (WMF) has already done this for many bots.

I saw one of the problematic bots says they are using AutoWikiBrowser.
maybe the bot is using an old version of AWB?

 Also, this change will affect not only bots but also every piece of software
 or tool that was written or published prior to the change and relies on the
 API to fetch data.
 So there are two issues with that:
 * The maintainer of the tool has no time/interest to compile and publish new
 releases of versions which were considered stable. This means the tool will
 die if the source code is not available or no one wants to take over.
 * Previously published programs that were considered stable and are executed
 after July 1st. The user may not be aware that this tool is no longer stable
 and requires an update. As far as I understood this API change: The results
 from the API will just be somewhat different and not produce an error. So
 the software will not crash but may produce weird behavior instead. Is there
 any solution to this?

The API currently emits a warning if a query continuation mode isnt selected.

I guess on July 1 the API could emit an error, and not return any query data.
Then the data isnt going to cause weird behaviour - it will break, properly.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month

[John Mark Vandenberg]

On Tue, Jun 9, 2015 at 6:04 AM, Ilya Korniyko intra...@gmail.com wrote:
 It would be nice if MediaWiki API  _AND_ pywikipedia bot do not deprecate
 at once.

 Now it looks as
 API:  we are deprecating what we promised to deprecated long ago - ok
 pywikipedia compat:  did not handle the deprecation of API before, and are
 not going to fix copy-pasted in tens of places (not one place, it's never
 that simple) query builders to support rawcontinue, we announce compat as
 discontinued together with the old style API.
 API deprecation was not coordinated with client library deprecation - not ok

 If there is one year gap between two deprecations - ok,  bot writers can
 choose either compat or core, and their bots can still work.
 Most users don't use APi directly so it should be the problem of
 coordination between API and clients developers.

On the pywikipedia side, it has been unofficially deprecated for a
while, and the intention was to decommission compat as gracefully as
possible, with Wikimania as the final killing ground.

The pywikibot developers roughly hashed out a decommissioning plan at
the Lyon hackathon, and worked with WMF staff to start doing impact
analysis.
https://phabricator.wikimedia.org/T101214

Then the MW API continuation breakage was announced post Lyon. Hmm.

The continuation problem in pywikipedia / compat is not so much those
50 or so occurrences where continuation bugs appear to exist, but
1. testing those 50 or so occurrences
2. finding the other less obvious occurrences
3. scripts people have written using compat's query.GetData may also
be broken, as it doesnt do continuation.

With a lot of wasted effort, 1  2 might be resolved so that 'compat'
code works, and someone could create a hack on query.GetData which
adds continuation for scripts not yet adapted to do continuation.
However the active pywikibot developers (approx. 5 people?) are
focused on making core better , including about 10 complex patches
under review that improve its query continuation algorithms, and
making core more compatible with 'compat' to ease the pain of
switching to core.

If anyone submits patches for compat continuation bugs affecting them,
they will be reviewed (usually by people familiar with compat) with
the presumption that the patch author has tested it, and merged if
there are no major problems.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month

[John Mark Vandenberg]

On Wed, Jun 3, 2015 at 3:42 AM, Brad Jorsch (Anomie)
bjor...@wikimedia.org wrote:
 ...
 I've compiled a list of bots that have hit the deprecation warning more
 than 1 times over the course of the week May 23–29. If you are
 responsible for any of these bots, please fix them. If you know who is,
 please make sure they've seen this notification. Thanks.

Thank you Brad for doing impact analysis and providing a list of the
71 bots with more than 10,000 problems per week.  We can try to solve
those by working with the bot operators.

If possible, could you compile a list of bots affected at a lower
threshold - maybe 1,000.  That will give us a better idea of the scale
of bots operators that will be affected when this lands - currently in
one months time.

Will the deploy date be moved back if the impact doesnt diminish by
bots being fixed?

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Simplifying the WMF deployment cadence

[John Mark Vandenberg]

On Fri, May 29, 2015 at 9:36 PM, Brad Jorsch (Anomie)
bjor...@wikimedia.org wrote:
 On Thu, May 28, 2015 at 2:39 PM, John Mark Vandenberg jay...@gmail.com
 wrote:

 [T96942 https://phabricator.wikimedia.org/T96942] was reported by
 pywikibot devs almost as soon as we detected that
 the test wikis were failing in our travis-ci tests.


 At 20:22 in the timezone of the main API developer (me).

 It was 12 hours before a MediaWiki API fix was submitted to Gerrit,


 09:31, basically first thing in the morning for the main API developer.
 There's really not much to complain about there.

In the proposed deploy sequence, 12 hours is a serious amount of time.

So if a shorter deploy process is implemented, we need to find ways to
get bug reports to you sooner, and ensure you are not the only one who
can notice and fix bugs related to API breakages, etc.

 and it took four additional *days* to get merged.


 That part does suck.

On a positive note, the API breakage this week was rectified much
quicker and pywikibot test builds are green again.

https://phabricator.wikimedia.org/T100775
https://travis-ci.org/wikimedia/pywikibot-core/builds/64631025

 This also doesnt give clients sufficient time to workaround
 MediaWiki's wonderful intentional API breakages. e.g. raw continue,
 which completely broke pywikibot and needed a large chunk of code
 rewritten urgently, both for pywikibot core and the much older and
 harder to fix pywikibot compat, which is still used as part of
 processes that wiki communities rely on.


 The continuation change hasn't actually broken anything yet.

Hmm.  You still don't appreciate that you actually really truly
fair-dinkum broke pywikibot?  warnings are part of the API, and
adding/changing them can break clients.  warnings are one of the more
brittle part of the API.

The impact on pywikibot core users wasnt so apparent, as the pywikibot
core devs fixed the problem when it hit the test servers and it was
merged before it hit production servers.  Not all users had 'git pull'
the latest pywikibot core code, and they informed us their bots were
broken, but as far as I am aware we didnt get any pywikibot core bug
reports submitted because (often after mentioning problems on IRC)
their problems disappeared after they ran 'git pull'.

However pywikipedia / compat isnt actively maintained, and it broke
badly in production, with some scripts being broken for over a month:

https://phabricator.wikimedia.org/T74667
https://phabricator.wikimedia.org/T74749

pywikibot core is gradually improving its understanding of the API
warning system, but it isnt well supported yet.  As a result,
generally pywikibot reports warnings to the user.
IIRC your API warnings system can send multiple distinct warnings as a
single string, with each warning separated by only a new line, which
is especially nasty for user agents to 'understand'. (but this may
only be in older versions of the API - I'm not sure)

So adding a new warning to the API can result in the same warning
appearing many many times on the user console / logs, and thousands of
warnings on the screen sends the users into panic mode.

I strongly recommend fixing the warning system before using it again
aggressively like was done for rawcontinue.  e.g. It would be nice if
the API emitted codes for each warning scenario (dont reuse codes for
similar scenarios), so we don't need to do string matching to detect 
discard expected warnings, and you can i18n those messages without
breaking clients. (I think there is already a phab task for this.)

I also strongly recommend that Wikimedia gets heavily involved in
decommissioning pywikibot compat bots on Wikimedia servers, and any
other very old unmaintained clients, so that the API can be
aggressively updated without breaking the many bots still using
compat.  pywikibot devs did some initial work with WMF staff at Lyon
on this front, and we need to keep that moving ahead.

 Unless pywikibot was treating warnings as errors and that's what broke it?

Yes, some of the code was raising an exception when it detected an API warning.

However another part of the breakage was that the JSON structure of
the new rawcontinue warnings was not what was expected.  Some
pywikipedia / compat code assumed that the presence of warnings
implied that there would be a warning related to the module used, e.g.
result['warnings']['allpages'] , but that didnt exist because this
warning was in result['warnings']['query'].

https://gerrit.wikimedia.org/r/#/c/176910/4/wikipedia.py,cm
https://gerrit.wikimedia.org/r/#/c/170075/2/wikipedia.py,cm

 It's coming
 soon though. Nor should a large chunk of code *need* rewriting, just add
 one parameter to your action=query requests.

I presume you mean we could have just added rawcontinue='' .  Our very
limited testing at the time (we didnt have much time to fix the bugs
before it would hit production), and subsequent testing, indicates
that the rawcontinue parameter can be used even

[Wikitech-l] Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

[John Mark Vandenberg]

On Fri, May 29, 2015 at 2:09 AM, Brian Wolff bawo...@gmail.com wrote:
 Shouldnt such [pywikibot] tests be run against beta wiki not testwiki?

Primarily this hasnt happened because pywikibot doesnt have a family
file for the beta wiki, but that is our issue, so I've done a simple
test run on beta wiki

four initial beta wiki problems:

- no https

(not nice - that means test accounts must be created and accessed
using passwords that are sent in essentially cleartext - so sharing
passwords with the same account name on the real wikis is a security
risk)

- invalid siteinfo interwiki map; includes entries to wikis that do
not exist.  That means pywikibot cant parse wikitext links, as it cant
distinguish between iwprefix: and namespace: (pywikibot team might be
able to reduce the impact of this wiki config problem)

- paramInfo failure that breaks pywikibot's dynamic api detection;
pywikibot batch loads the paraminfo for all modules, for example
issuing the following API as part of the initialisation, and it is a
503 Service Unavailable
http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfomodules=abusefiltercheckmatch|abusefilterchecksyntax|abusefilterevalexpression|abusefilterunblockautopromote|antispoof|block|bouncehandler|centralauthtoken|centralnoticebannerchoicedata|centralnoticequerycampaign|checktoken|cirrus-config-dump|cirrus-mapping-dump|cirrus-settings-dump|clearhasmsg|compare|createaccount|cxconfiguration|cxdelete|cxpublish|delete|deleteglobalaccount|echomarkread|echomarkseen|edit|editlist|editmassmessagelist|emailuser|expandtemplates|fancycaptchareload|featuredfeed|feedcontributions|feedrecentchanges|feedwatchlist|filerevert|flowthank|globalblock|globaluserrights|help|imagerotate|import|jsonconfig|languagesearch|login|logout|managetags|massmessage|mobileview|move|opensearchmaxlag=5format=json

A little digging shows that the problematic module is fancycaptchareload

http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfomodules=fancycaptchareloadmaxlag=5format=json

Our servers are currently experiencing a technical problem. This is
probably temporary and should be fixed soon. Please try again in a few
minutes.

If you report this error to the Wikimedia System Administrators,
please include the details below.
Request: GET 
http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfomodules=fancycaptchareloadmaxlag=5format=json,
from 127.0.0.1 via deployment-cache-text02 deployment-cache-text02
([127.0.0.1]:3128), Varnish XID 855090368
Forwarded for: [your IP], 127.0.0.1
Error: 503, Service Unavailable at Fri, 29 May 2015 07:57:09 GMT 

Could someone investigate this / fix that module?

Pywikibot devs could also help reduce the impact of this type of
problem in the future, by falling back from batch mode fetch to
loading individual modules in order to skip buggy modules.

- no SUL with the real wikis

(probably the best choice given no https on the beta cluster, but it
complicates adding beta wiki to our existing Travis-CI test matrix
which includes real wikis)

actual results at
https://travis-ci.org/jayvdb/pywikibot-core/builds/64532033

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

[John Mark Vandenberg]

On Fri, May 29, 2015 at 3:14 PM, John Mark Vandenberg jay...@gmail.com wrote:
 ..
 - paramInfo failure that breaks pywikibot's dynamic api detection;
 pywikibot batch loads the paraminfo for all modules, for example
 issuing the following API as part of the initialisation, and it is a
 503 Service Unavailable
 http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfomodules=abusefiltercheckmatch|abusefilterchecksyntax|abusefilterevalexpression|abusefilterunblockautopromote|antispoof|block|bouncehandler|centralauthtoken|centralnoticebannerchoicedata|centralnoticequerycampaign|checktoken|cirrus-config-dump|cirrus-mapping-dump|cirrus-settings-dump|clearhasmsg|compare|createaccount|cxconfiguration|cxdelete|cxpublish|delete|deleteglobalaccount|echomarkread|echomarkseen|edit|editlist|editmassmessagelist|emailuser|expandtemplates|fancycaptchareload|featuredfeed|feedcontributions|feedrecentchanges|feedwatchlist|filerevert|flowthank|globalblock|globaluserrights|help|imagerotate|import|jsonconfig|languagesearch|login|logout|managetags|massmessage|mobileview|move|opensearchmaxlag=5format=json

 A little digging shows that the problematic module is fancycaptchareload

 http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfomodules=fancycaptchareloadmaxlag=5format=json

 Our servers are currently experiencing a technical problem. This is
 probably temporary and should be fixed soon. Please try again in a few
 minutes.

 If you report this error to the Wikimedia System Administrators,
 please include the details below.
 Request: GET 
 http://en.wikipedia.beta.wmflabs.org/w/api.php?action=paraminfomodules=fancycaptchareloadmaxlag=5format=json,
 from 127.0.0.1 via deployment-cache-text02 deployment-cache-text02
 ([127.0.0.1]:3128), Varnish XID 855090368
 Forwarded for: [your IP], 127.0.0.1
 Error: 503, Service Unavailable at Fri, 29 May 2015 07:57:09 GMT 

 Could someone investigate this / fix that module?

After pywikibot devs cleared some other test breakages (our problems)
we see this problem is also affecting the test wikis, which I assume
means this bug will be affecting real wikis soon, so I've created an
Unbreak Now! ticket for it and cc'd @greg.

https://phabricator.wikimedia.org/T100775

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Testing pywikibot on beta wikis (Was: Simplifying the WMF deployment cadence)

[John Mark Vandenberg]

On Fri, May 29, 2015 at 3:14 PM, John Mark Vandenberg jay...@gmail.com wrote:
 On Fri, May 29, 2015 at 2:09 AM, Brian Wolff bawo...@gmail.com wrote:
 Shouldnt such [pywikibot] tests be run against beta wiki not testwiki?

 Primarily this hasnt happened because pywikibot doesnt have a family
 file for the beta wiki, but that is our issue, so I've done a simple
 test run on beta wiki

I've create a task for this little project:

https://phabricator.wikimedia.org/T100796

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Simplifying the WMF deployment cadence

[John Mark Vandenberg]

On Fri, May 29, 2015 at 2:07 AM, Greg Grossmeier g...@wikimedia.org wrote:
 quote name=John Mark Vandenberg date=2015-05-29 time=01:39:52 +0700
 It was reported by pywikibot devs almost as soon as we detected that
 the test wikis were failing in our travis-ci tests.  It was 12 hours
 before a MediaWiki API fix was submitted to Gerrit, and it took four
 additional *days* to get merged.  The Phabricator task was marked
 Unbreak Now! all that time.

 Which shows the tooling works, but not the social aspects. The backport
 process (eg SWAT and related things) will improve soon as well which
 should address much of this.

Your tooling depends on pywikibot developers (all volunteers) merging
a patch within your branch-deploy cycle, which fires off a Travis-CI
build of *pywikibot* unit tests which runs some tests against
test.wikipedia.org and test.wikidata.org ?  And your proposing to
shorten the window in which all this can happen and get useful bug
reports out.

A little crazy but OK.  The biggest problem with that approach is
Travis-CI is not very reliable - often they are backlogged and tests
are not run for days.  So I suggest that you arrange to run the
pywikibot tests daily (or more regularly) on WMF test/beta servers,
and the unit tests of any other client which is a critical part of
processes on the Wikimedia wikis.

 Not-a-great-response-but: can you specifically ping me in phabricator
 (I'm @greg) for issues like that above?

That is a process problem.  The MediaWiki ops  devs need to detect 
escalate massive API breakages, especially after creating the fix
which needs to be code reviewed.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Simplifying the WMF deployment cadence

[John Mark Vandenberg]

On Fri, May 29, 2015 at 12:17 AM, Legoktm legoktm.wikipe...@gmail.com wrote:
 On 05/27/2015 01:19 PM, Greg Grossmeier wrote:
 Hi all,

 New cadence:
 Tuesday: New branch cut, deployed to test wikis
 Wednesday: deployed to non-wikipedias
 Thursday: deployed to Wikipedias

 This means that if we/users spot a bug once the train hits Wikipedias,
 or the bug is in an extension like PageTriage which is only used on the
 English Wikipedia, we have to: rush to make the 4pm SWAT window, deploy
 on Friday, or wait until Monday; which from what I remember were similar
 reasons from when we moved the train from Thursday to Wednesday.

Recent API breakages suggest that this doesnt give enough time for
client tests to be run, bugs reported, fixed and merged.

https://phabricator.wikimedia.org/T96942 was an API bug last month
which completely broke pywikibot.  All wikis; all use cases.

It was reported by pywikibot devs almost as soon as we detected that
the test wikis were failing in our travis-ci tests.  It was 12 hours
before a MediaWiki API fix was submitted to Gerrit, and it took four
additional *days* to get merged.  The Phabricator task was marked
Unbreak Now! all that time.

This also doesnt give clients sufficient time to workaround
MediaWiki's wonderful intentional API breakages. e.g. raw continue,
which completely broke pywikibot and needed a large chunk of code
rewritten urgently, both for pywikibot core and the much older and
harder to fix pywikibot compat, which is still used as part of
processes that wiki communities rely on.

Another example is the action=help rewrite not being backwards
compatible. pywikibot wasnt broken, as it only uses the help module
for older MW releases; but it wouldnt surprise me if there are clients
that were parsing the help text and they would have been broken.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Regarding gsoc 2015

[John Mark Vandenberg]

On Sat, Mar 21, 2015 at 7:26 PM, Arindam Padhy b113...@iiit-bh.ac.in wrote:
 why is that its always necessary that you have to solve  task in order to
 get the project.you should also look at the student's idea,before rejecting
 him.j

Completing microtasks is a way for applicants to demonstrate that they
are able to understand the codebase relevant to the project.
Mentors (especially if they are volunteers) dont have spare time to
allocate to student ideas where the applicant cant prove that they can
code by solving some simple bugs.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Regarding gsoc 2015

[John Mark Vandenberg]

On Sat, Mar 21, 2015 at 8:59 PM, Arindam Padhy b113...@iiit-bh.ac.in wrote:
 but on 11th march the mentor said in his comment that he has already found
 some strong students for this project.that means from the beginning only
 you guys have fixed who will work for you,that is before the student
 applications begin.
 he also mentioned in his statement that even if you solve microtasks you
 would not be selected.I dont deny the fact that the students he has
 selected may be good ,but they are not the only potential candidates, there
 will always be a lot of candidates better than him. of course the  mentor
 is not a god that he will know frm the beginning that who is the best
 candidate for this project without actually looking at al the proposals .
 even in the official gsoc site in their time line it is mentioned that the
 mentors are giving a time period where they look at all the proposals and
 find for any duplication.i know you guys are developers but without even
 considering a student's proposal you are telling him that you will not be
 selected.this can only happen if the either the candidate is very close to
 the mentor i.e from his nation or from his family.

If the project already has a very good candidate, then you have two options:

1. be a better candidate.

(you never know your luck, the other very good candidate may not
even apply before the deadline)

2. apply for a different project.

Whatever option you take, you need to write a very good application,
be active in the IRC channel, and fixing relevant bugs will boost your
luck in being selected.

There are 11 projects 'Featured for GSoC and Outreachy' status on
https://phabricator.wikimedia.org/tag/possible-tech-projects/

There are an additional 7 in 'Missing Mentors' status.  If you write a
good application for one of those, we'll try harder to find a mentor.

In google-melange, I see only three projects which have a candidate
application, so that leaves 15 projects without any candidates.

There are also 38 possible projects in other status.  If you think you
have the skills to do them, start a discussion on IRC and/or the
Phabricator task to attempt to push the project forward.

If you have an idea which isn't on the 'possible tech projects' board,
you should search Phabricator extensively, as Phabricator is full of
good ideas - almost 15 years worth of good ideas are in the database.
Find similar ideas in Phabricator.  Talk about your idea on IRC.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Global user pages deployed to all wikis

[John Mark Vandenberg]

On Fri, Feb 20, 2015 at 9:22 PM, David Gerard dger...@gmail.com wrote:
 Nice one!

 Will anyone be porting all the Babel templates? That's a seriously
 useful thing that should go there, and they aren't on Meta yet.

Eh? I think Babel templates are from last decade.
See https://meta.wikimedia.org/wiki/User_language

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Boil the ocean, be silly, throw the baby out with bathwater, demolish silos, have fun

[John Mark Vandenberg]

On Sat, Feb 14, 2015 at 2:42 PM, Ricordisamoa
ricordisa...@openmailbox.org wrote:
 Can I just say that I hate it?
 It does not even give access to the talk page!

Looks like here is the best place to discuss design decisions:
https://www.mediawiki.org/wiki/Talk:Beta_Features/Minerva

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Google thinks the Anonymous entry on enwp is hacked

[John Mark Vandenberg]

Google is showing their This site may be hacked. note when showing
the enwp article about Anonymous, with a link to
https://support.google.com/websearch?p=ws_hacked

https://www.google.com/search?q=Anonymous+group

Found via

http://thestack.com/anonymous-this-site-may-be-hacked-wikipedia-120215

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Google thinks the Anonymous entry on enwp is hacked

[John Mark Vandenberg]

On Sat, Feb 14, 2015 at 4:41 PM, Dan Garry dga...@wikimedia.org wrote:
 When I logged into the webmaster tools and followed their instructions to
 resolve the issue it said that en.wikipedia.org had no security issues.

 I guess whatever happened has been fixed and we just need to wait for it to
 resolve.

Dan,

Martin Anderson says (in the article I linked to) this has been
occurring for several days, and it points out that Google has no
problems with the site in general.
http://www.google.com/safebrowsing/diagnostic?site=en.wikipedia.org

A comment on that article suggests that maybe bad links on the page
are the cause..?

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Google thinks the Anonymous entry on enwp is hacked

[John Mark Vandenberg]

Thanks James.  Do you recall what the previous articles were?

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikimedia-l] Congratulations to MediaWiki Farmers User Group for being approved as a Wikimedia User Group

[John Mark Vandenberg]

*sigh* I was hoping the farmers had united 

but good luck to the virtual farmers of ~85% men

On Sun, Jan 18, 2015 at 10:54 AM, Gregory Varnum
gregory.var...@gmail.com wrote:
 Greetings,

 Please join the Affiliations Committee in congratulating the MediaWiki
 Farmers User Group on their official approval as a Wikimedia User Group:
 https://meta.wikimedia.org/wiki/Affiliations_Committee/Resolutions/MediaWiki_Farmers_User_Group_-_Liaison_approval,_January_2015

 The MediaWiki Farmers User Group is A user group of third-party developers
 who work on wiki farms. Our mission is to improve and standardize the way
 MediaWiki wiki farms are setup and run.

 Anyone interested in more information about the group can visit:
 https://www.mediawiki.org/wiki/Project:MediaWiki_Farmers_user_group

 Again - congratulations on the recognition and best wishes for the group's
 future work!
 -greg aka varnent
 Vice Chair, Wikimedia Affiliations Committee
 ___
 Wikimedia-l mailing list, guidelines at: 
 https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
 wikimedi...@lists.wikimedia.org
 Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
 mailto:wikimedia-l-requ...@lists.wikimedia.org?subject=unsubscribe



-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Our CAPTCHA is very unfriendly

[John Mark Vandenberg]

On Fri, Dec 5, 2014 at 7:48 AM, Comet styles cometsty...@gmail.com wrote:
 Until they fix captcha or allow Global Filters to become truly global,
 there will always be a risk of spambots. For someone who deals with
 these on a regular basis and has been doing it for years, making the
 Captcha system more friendly to users also means making it more
 'friendly' to spam bot masters...the only alternative is to make
 captchas more trickier or harderAs i mentioned a few times on IRC,
 every 3rd account created on wikimedia is a spambot and thats just
 accounts, not taking into account the thousands of spam edits by IP's
 daily..

Is there bugs raised for global filters?

Also, I suspect that relying on Abuse Filters wrt Wikibase sites is
going to be a bad idea ; the abuse filters would need to parse the
JSON...?  Also it is a very large wiki by content, important for
spammers because it can directly change the content of several wikis,
but still a quite small wiki by dedicated maintenance team, so may be
quickly overwhelmed if included in testing by devs.  That also applies
to the test.wikidata environment, except for the utility of spamming
it, but spammers need a test site too.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Our CAPTCHA is very unfriendly

[John Mark Vandenberg]

On Fri, Dec 5, 2014 at 11:08 AM, MZMcBride z...@mzmcbride.com wrote:
 Chad wrote:
Well that was a fun experiment for an hour. Turns out captchas do actually
stop a non-zero amount of spam on non-test wikis.

Mediawiki.org logs tell the story pretty clearly.

This has been rolled back.

 :-(

 https://www.mediawiki.org/wiki/Special:Log/delete

 I spent a bit of time poking around. The spam seems to primarily be
 related to page creation. A slightly smarter heuristic (such as requiring
 that your edit count be  0 before you can create a page) might mitigate
 this. Disallowing edits that contain a href might also help.

 The more I think about this, the more I wonder whether we should change
 the CAPTCHA model so that instead of applying CAPTCHAs in a blanket manner
 to types of actions (page creation, account creation, etc.), we could
 instead only force users to solve a CAPTCHA when certain abuse filters are
 triggered. Adding this functionality to the AbuseFilter extension is
 tracked at https://phabricator.wikimedia.org/T20110.

 By shifting from the current rigid PHP configuration model to a looser and
 more flexible AbuseFilter model, we could hopefully ensure that
 anti-abuse measures (warning about an action, disallowing an action, or
 requiring a CAPTCHA before allowing an action) are more narrowly tailored
 to address specific problematic behavior. Even triggering AbuseFilter
 warnings that simply add an extra click/form submission for specific
 patterns of problematic behavior might trip up many of these spambots.

Big +1 for this.

Then, if a wiki community really wants to use CAPTCHA on all account
creations or link additions, they can add that as AF rules.

I am quite sure that on English Wikipedia the AF admins would enjoy
defining precise rules and monitor them for effectiveness using the AF
log.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] changing edit summaries

[John Mark Vandenberg]

On Thu, Nov 13, 2014 at 9:57 PM, Yusuke Matsubara w...@whym.org wrote:
 On Thu, Nov 13, 2014 at 9:15 PM, Amir E. Aharoni
 amir.ahar...@mail.huji.ac.il wrote:
 I tried looking for it in Bugzilla; I expected to find a two-digit bug for
 it, but I couldn't find any at all. Of course it's possible that I didn't
 look well enough.

 A bit different, but there is an extension that enables
 supplementing additional non-modifiable edit summaries:
 https://www.mediawiki.org/wiki/Extension:RevisionCommentSupplement

 It was contributed (without a Bugzilla request) by Burthsceh, a
 volunteer at Japanese Wikipedia, prompted by the necessity to fix
 attributions made in edit summaries (for reused texts). [1]  I don't
 think it has been extensively reviewed, though.

 With that approach, you could effectively modify an edit summary by
 appending a modified one and rev-deleting the original one.

 [1] 
 https://ja.wikipedia.org/wiki/Wikipedia:%E4%BA%95%E6%88%B8%E7%AB%AF/subj/%E5%B1%A5%E6%AD%B4%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%80%8B%E3%80%85%E3%81%AE%E7%89%88%E3%81%AB%E5%AF%BE%E3%81%97%E3%81%A6%E8%BF%BD%E5%8A%A0%E3%81%AE%E3%82%B3%E3%83%A1%E3%83%B3%E3%83%88%E3%82%92%E8%A1%A8%E7%A4%BA%E3%81%99%E3%82%8B%E6%96%B9%E6%B3%95%E3%81%AE%E5%B0%8E%E5%85%A5%E3%81%AE%E6%8F%90%E6%A1%88

This is a very good idea, in itself, to help fix problems with
attribution, especially wrt 'print' editions (PDF export).

It might also be used to avoid undesirable attribution notices in the
article body, where the text is a copy of a GFDL/CC external webpage,
to be used for once-off imports (e.g. bio page on personal website),
rather than large scale import/reuse like FOLDOC/EB1911/etc.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Release workflow recommendation] A public releases JSON file.

[John Mark Vandenberg]

On Fri, Sep 19, 2014 at 9:42 AM, Daniel Friesen
dan...@nadir-seen-fire.com wrote:
 The WikiData item doesn't contain enough information, it only lists the
 stable version number. All the situations I've come up with require the
 kind of detailed information we have on
 https://www.mediawiki.org/wiki/Version_lifecycle

 - Major release numbers
 - Minor release numbers
 - What ones are current, legacy, and obsolete
 - Is a release LTS or in development

I think I've added all that information to the Wikidata item.
Anything else you need or want?

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Is simplicity is the key to success of Echo and Watchlist?

[John Mark Vandenberg]

On Wed, Sep 3, 2014 at 1:25 PM, MZMcBride z...@mzmcbride.com wrote:
 When I hop around mediawiki.org currently, I'm being incessantly poked in
 the eye by a very bright 61 because I had the audacity to watch a talk
 page. Clearly this is broken. Of course I also have the string new
 messages (2,312) from LiquidThreads rewritten as noise using per-user
 JavaScript, so at this point I'm just continuing to pray that the Flow
 team has made a best effort to learn from past mistakes.

Mine is 52 and New messages (1,826), which turns me off looking at either.

The UI under the 52 doesnt show me 52 notifications - it shows me
about 20.  It includes a mark all as read link, but I cant guess
whether that will mark the ~20 as read (which would be good, as they
are all Talk:Flow), or all 52, including the 32 I cant see (which
would be bad as an important/interesting notification could be in the
other ~32).

I dont mind 1,826 new messages as much, but would love
[[Special:NewMessages]] to first let me pick which threads I want to
read or mark as read based on only the thread page and 'section' name.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] broken travis-ci builds of mediawiki

[John Mark Vandenberg]

On Tue, Aug 19, 2014 at 12:23 AM, John Mark Vandenberg jay...@gmail.com wrote:
 There are build errors in the mediawiki builds on travis-ci, which
 have been happening for four months.

 https://travis-ci.org/wikimedia/mediawiki-core/builds

Are there any other travis-ci builds running on the wikimedia account
other than mediawiki and pywikibot?

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] broken travis-ci builds of mediawiki

[John Mark Vandenberg]

On Thu, Aug 21, 2014 at 9:58 PM, aude aude.w...@gmail.com wrote:
 On Thu, Aug 21, 2014 at 4:06 PM, John Mark Vandenberg jay...@gmail.com
 wrote:

 On Tue, Aug 19, 2014 at 12:23 AM, John Mark Vandenberg jay...@gmail.com
 wrote:
  There are build errors in the mediawiki builds on travis-ci, which
  have been happening for four months.
 
  https://travis-ci.org/wikimedia/mediawiki-core/builds

 Are there any other travis-ci builds running on the wikimedia account
 other than mediawiki and pywikibot?


 I think folks at least care about the hhvm build for core.

The hhvm build went green in the last two days, so maybe you're right ;-)

 there also are travis builds for Wikibase, which we do care about

As do I ! ;-)

Would it be possible for WMDE to run its builds on the wmde account?

I am told (by someone on #travis-ci , fwiw) that the 'wikimedia'
account has lots of builds happening and that is why pywikibot ones
have been queued for up to an hour. (that is rare, but it was the case
when I sent the initial email to the wikitech-l list)

 and possibly other extensions (?)

It would be nice to get a list of the repos enabled on travis-ci ,
maybe saved on mediawiki.org, as travis-ci doesnt give a list of
builds for the 'wikimedia' account.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] broken travis-ci builds of mediawiki

[John Mark Vandenberg]

There are build errors in the mediawiki builds on travis-ci, which
have been happening for four months.

https://travis-ci.org/wikimedia/mediawiki-core/builds

Is anyone using that to help with development.  I doubt it, given they
are broken.

Could they be disabled, as those jobs are causing other travis-ci jobs
under 'wikimedia' to be delayed.  e.g. the pywikibot-core builds,
where the developer team actively uses the travis builds.

https://travis-ci.org/wikimedia/pywikibot-core/builds

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Superprotect user right, Comming to a wiki near you

[John Mark Vandenberg]

On Tue, Aug 12, 2014 at 12:18 AM, Brian Wolff bawo...@gmail.com wrote:

 Now, having observed that not only user Eloquence (aka Erik Moeller)
 himself engaged in the enforcement of  superprotect right on de.wp
 [1] but soon after a workaround was published a change was deployed
 [2, 3] as counter measurement to block any possible interference can
 no longer be interpret as acting in good faith but rather strikes me
 as a form of oppression (or worst as censorship).


 [Putting the purely mw dev hat on]

 It was a bug in mediawiki, and thus it should be fixed. MediaWiki is used
 by many different groups and in general we [mw devs] do not judge people
 for how they use the software. If some non wmf entity reported the bug, it
 would still be fixed.

 So dont complain that mw fixes a bug in how page protection. If you are
 unhappy with current events you should direct your anger at how the wmf
 decided to use hard security to enforce its dictates, not at the software
 for working.

Sorry Brian, which bug are you referring to?  Could you point me to a
bug report?

Before this, there was no expectation that a page could be protected
such that sysops could not alter the content of the superprotected
page.

Now, the devs/ops have attempted to introduce that capability, and the
new functionality is very likely riddled with holes, some of which
MZMcBride has suggested in the thread 'Options for the German
Wikipedia'.

Moreover the deployed technical change is useless due to design flaws.
What was the goal of this change?  Was it to prevent sysops injecting
JavaScript that logged out user-agents execute?  If that is the
use-case, this patch is a very weak solution from an engineering
perspective.  It was rushed it into a production environment, and
needed a follow up patch almost immediately.  And the bug reports for
this new functionality will surely roll in.

These patches only make it 'forbidden' to deactivate the MediaViewer.
They don't prevent it.  These patches only introduce a new policy,
signalling a new era, and make it technically more challenging to
bypass that new policy.  The policy written says Sysops are not
allowed to inject JavaScript into the reader's user-agent which
interferes with WMF's favoured features.  It is still possible, but
the only thing that is stopping de.wp sysops from deactivating the
MediaViewer some other way is that the WMF has demonstrated it will
make drastic changes to the MediaWiki configuration to take away
capabilities from their community.  Should the community work-around
this change, they are fairly confident that the WMF will desysop
whoeverdoes it, or more configuration changes and superprotection will
occur.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] [Wikimedia-l] Superprotect user right, Comming to a wiki near you

[John Mark Vandenberg]

On Tue, Aug 12, 2014 at 3:49 AM, Brad Jorsch (Anomie)
bjor...@wikimedia.org wrote:
 On Mon, Aug 11, 2014 at 2:01 PM, John Mark Vandenberg jay...@gmail.com
 wrote:

 Before this, there was no expectation that a page could be protected
 such that sysops could not alter the content of the superprotected
 page.


 This is false.

Care to explain?

Was this functionality was ever supported by MediaWiki core?
Could you point me towards some documentation?

 Now, the devs/ops have attempted to introduce that capability, and the
 new functionality is very likely riddled with holes, some of which
 MZMcBride has suggested in the thread 'Options for the German
 Wikipedia'.


 Most of what MZMcBride posted there has nothing to do with actually
 breaking superprotection. Editing a page that isn't superprotected isn't a
 break in the protection feature itself, for example.

Of course it is.  It isnt a 'feature' until it actually works at the
released product level.
Rushing component level hardening changes into production, when
everyone knows how to work around the new 'hardened' code, it very bad
change management.
It likely introduces unforeseen bugs, for no actual gain.

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Superprotect user right, Comming to a wiki near you

[John Mark Vandenberg]

On Sun, Aug 10, 2014 at 9:00 PM, Michał Łazowik mlazo...@me.com wrote:
 Wiadomość napisana przez Nicolas Vervelle nverve...@gmail.com w dniu 10 sie 
 2014, o godz. 15:45:

 I hope it's not an other step from WMF to prevent the application of
 community decisions when they not agree with it. I fear that they will use
 this to bypass community decisions. For example like forcing again VE on
 everyone on enwki: last year, sysop were able to apply community decision
 against Erik wishes only because they had access to site wide js or CSS.

 I'd like to believe that code-reviewing would mean improving code quality, 
 security
 and performance (applies to javascript).

It could mean that, but of course it is actually introduced to prevent
the German community from deactivating the Media Viewer.

https://de.wikipedia.org/w/index.php?title=MediaWiki:Common.jsaction=history

I remember when we used to beg for the WMF to deploy extensions.
Now we really need to beg for the WMF to not deploy extensions ...

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Not logged in page

[John Mark Vandenberg]

On Wed, Jul 16, 2014 at 8:46 AM, Jon Robson jdlrob...@gmail.com wrote:
 This seems like a bit of an edge case. If you are worried about this
 then I would say you want to check the keep me logged in box...

For some of us power users, it will be an edge case we hit annoyingly
regularly.  I've found mediawiki logs me out despite the 'keep me
logged in' box, when logging out on a different device, etc.

Irrespective of that, instead of redirecting to Special:UserLogin, the
login box could appear, with the form submission being processed at
Special:UserLogin.  Crazy idea, I know, but lots of other software
does that.

Another approach, which IMO should be done as a fallback, is to use an
approach similar to 'action=editredlink=1' and 'redirect=no' to state
in the URL that the user doesnt want to login a second time, but would
like to be logged in and return to another page.  i.e.

https://en.wikipedia.org/w/index.php?title=Special:UserLoginreturnto=Special%3AWatchlistskipifloggedin=1

If they reload that URL while logged in, it wont ask for login.

This would be similar to the API assert=bot/user logic.

http://www.mediawiki.org/wiki/API:Assert

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] User agent policy for bots

[John Mark Vandenberg]

On Sat, Jul 12, 2014 at 12:07 AM, Jeremy Baron jer...@tuxmachine.com wrote:
 On Jul 11, 2014 9:45 AM, Marc A. Pelletier m...@uberbox.org wrote:
 On 07/11/2014 09:34 AM, John Mark Vandenberg wrote:
  Could ops confirm they have the username of each logged in edit at
  their finger tips (i.e. roughly as easy to access as the user-agent)?
  Pywikibot doesnt permit logged out edits.

 We do, after the fact, from the same data Checkusers have access to.

 Not if they don't make an edit.

 There's lots of options for bots to cause trouble for ops. (including
 things that effect all wikis on the cluster, not just the specific one they
 were accessing)

And it is quite reasonable to assume bots could be causing problems
when 'reading', as that is a large component of what they do. In the
case of pywikibot, it only knows how to use the API, so I inaccurately
said 'edit' when I meant 'API request'; my apologies for that.

 I'm not sure where that talk occured; I have not been made aware of it
 and it didn't filter through the normal ops channels that I've seen.

 I believe that's referring to the pywikipedia list.

Correct.

 I'm a little surprised by Antoine's suggestion that it is important that
 the bot user's information is in the UA string - it doesn't seem useful
 or necessary to me.  Bots shouldn't be editing while logged out in the
 first place, so the bot account will normally always be plain to see.

 Obviously, having the user account in the UA would help a bit in
 tracking down errant bots when they happen but that should be a rare
 occurance and we have other methods to use in those cases.

 Varnish has access to the cookies, sure. But we log UA string and not
 cookies. Or maybe analytics is doing extra logging I didn't notice?

It would be good to know the answer to whether the username is logged
against API requests.  It seems like a very important piece of
information which should be visible in server ops logging of API
usage.

 If
 you're looking at request logs or varnishtop then UA string is a convenient
 way (and the standard way we've always suggested to not operators) to
 identify the bot.
 Imagine if you've identified a specific type of bad request in logs and
 they're all from one IP and one UA string. Varnish can easily send an error
 for a certain UA string+IP address before it hits the apaches if you need
 it to. But if that UA string is generic then you may end up blocking
 collateral damage instead of just the one broken bot.

I'd like to tease out what is the most useful data for pywikibot to
include in the UA.  Apparently pywikibot needs to add something to the
user-agent to be 'gold standard' for accessing Wikimedia projects, but
pywikibot also needs to avoid collecting and publishing personal
information for wiki that don't require it, or operators who refuse to
disclose.  Many Wikimedia bot operators run a few commands a week, and
it is not likely to be useful to contact them if their script is
mishaving - they are probably scratching their heads too.

And ideally pywikibot does this automatically, or makes it really easy
to set up and/or provides a benefit to the bot operator for having
provided additional information.

username is easy, if it is needed. checking email is enabled is also
easy, and is comparable to sysops being expected to enable email on
many wikis.

pywiki requiring bot operators provide an email address is technically
easy, but I suspect it isnt going to be very successful or
appreciated, esp for non-SSL wikis, or understood as pywiki hasnt put
this info in the user-agent since the new user-agent policy was
introduced, so why now?  It also has data privacy issues, as user
agents appear in logs.  Are the user-agents completely deleted?

If the main source of problems is the 'large' bots, they usually run
many tasks, and it is likely to only be a single task causing
problems.  With these large tasks, ideally they are paused rather than
blocked, in which case we need to introduce a standardised way to
pause a bot.  In these cases, the user agent could mention the task
identifier, and that identifier could be used to pause it until an
operator has checked their email.  The 'pause' command interface could
be IRC or user_talk, or something new based on Flow, or a API response
warning like replag which pywikibot honours.  I appreciate Bináris'
point that some (most?) wikis, especially smaller wikis, do not have
'task approval' processes with a task identifier, so this would need
to be optional.  Large bot operators would use this feature if it
meant that only a single task is paused rather than the bot account
blocked.

For the normal usage of pywikibot, being invoking an existing script
which is maintained by pywikibot, we could include in the user-agent
which script is running (e.g. move.py).

Determining whether the bot code has been modified by the operator is
a bit of work, but I think that is more feasible than attempting to
convince bot operators to add

Re: [Wikitech-l] User agent policy for bots

[John Mark Vandenberg]

On Fri, Jul 11, 2014 at 6:50 PM, Antoine Musso hashar+...@free.fr wrote:
 Le 11/07/2014 01:09, Amir Ladsgroup a écrit :
 Hello,
 As discussions in pywikipedia-l people are not sure whether is necessary to
 add username of bot operator in user agent or not.

 In user agent policy https://meta.wikimedia.org/wiki/User-agent_policy
 it's mentioned that people need to add contacting information, but it's not
 clear it's about contacting the tool-maker or tool-user.

 Can you clarify it?

 Hello,

 As K. Peachey said, the aim is for Wikimedia operators to be able to
 identify the user running the bot.  The bot framework might be useful.

 A suitable user agent could be:

  HasharBot (http://fr.wikipedia.org/wiki/User:Hashar; hashar at free fr)


 We most probably already have the username in our logs, doesn't harm to
 repeat it in the user-agent.  IRC nickname and email would be nice
 additions and probably save time.

Could ops confirm they have the username of each logged in edit at
their finger tips (i.e. roughly as easy to access as the user-agent)?
Pywikibot doesnt permit logged out edits.

There is some talk that if pywikibot doesnt fix its user-agent string,
ops may need to block 'the toolserver' - could ops confirm that they
would usually block a bot account before killing a well known IP range
like 'the toolserver' (or 'the wmf labs')

IMO it is pretty silly to put the username in the User-Agent for
logged in users who are running adhoc tasks using unmodified pywikibot
code, as they are the user, not its agent. In that scenario, a
distinct version of pywikibot is the agent.  And an email address is
even worse in this scenario.

I do appreciate the need to uniquely identify different user agents,
being any customised code.  Pywikibot already detects which (git)
revision it is running, and includes that in the user agent.  It also
checks versions of files, but I dont think it accurately detects I am
a customised bot and definately doesnt include that in the user
agent.  It should.

Also, ideally bots should link to the bot task approval page with
every edit, either in the public edit summary or in the (invisible
except by ops and check-users) user-agent.

Rather than asking bot operators to put an email address in the user
agent, is it OK to have special:emailuser enabled on the bot and
operator?  Or, have a master kill switch on the bot user (task) page?
There is talk of an RFC for 'standardising' how the community can
interact with pywikibot bots, such as disabling bot tasks or the bot
account.
https://gerrit.wikimedia.org/r/#/c/137980/
Checking email is enabled, and ensuring the bot can be easily paused
by 'the community' (inc. ops) strikes me as what is needed, rather
than putting PII into the user *agent*.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] git-review 1.24 broken?

[John Mark Vandenberg]

On Fri, Jul 11, 2014 at 2:36 AM, Adrian Lang adrian.l...@wikimedia.de wrote:
 On Thu, Jul 10, 2014 at 6:33 PM, Moriel Schottlender mor...@gmail.com wrote:
 It seems that the native package doesn't work correctly. Or, if that's no
 longer true, we should update the docs.

 1.23-2 works great for me.

I was using the distro package of git review for both FC and Ubuntu
for a long time without issues, so I think our docs should be updated
to recommend that, and document any issues with (older) distro
versions.

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] git-review 1.24 broken?

[John Mark Vandenberg]

Could it be an OSX specific problem?

git review 1.24 works for me on two linux distros, with different
setuptools installed

fedora-core-19 using distribute 0.6.49
ubuntu-14.04 using setuptools 3.3


On Wed, Jul 9, 2014 at 9:29 PM, Željko Filipin zfili...@wikimedia.org
wrote:

 A few days ago, I got this warning message after pushing a commit to Gerrit
 via git-review:

 ***
 A new version of git-review is available on PyPI. Please
 update your copy with:
   pip install -U git-review
 to ensure proper behavior with gerrit. Thanks!
 ***



 I had version 1.23

 $ git-review --version
 git-review version 1.23



 I have followed the instructions and got version 1.24. (I had to use sudo
 on my Mac to get pip install to work.)

 $ pip install -U git-review



 But then, git-review broke completely:

 $ git review
 (...)
 raise DistributionNotFound(req)  # XXX put more info here
 pkg_resources.DistributionNotFound: git-review



 Since I did not have the time to debug this, I have gone back to 1.23 and
 everything works again. (Do not forget sudo.)

 pip uninstall git-review
 pip install git-review==1.23



 The problem is already reported[1]. The workaround is to upgrade
 setuptools. (Do not forget sudo.)

 $ pip install --upgrade setuptools
 (...)

 $ pip install -U git-review
 (...)

 $ git review --version
 git-review version 1.24



 Željko
 --
 1: https://bugs.launchpad.net/git-review/+bug/1337701
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
John Vandenberg
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] git-review 1.24 broken?

[John Mark Vandenberg]

On Thu, Jul 10, 2014 at 5:50 AM, Moriel Schottlender mor...@gmail.com wrote:
 I'm using Ubuntu 14.04 and I get the same issues.

 Worse, I seem to also get problems uninstalling and reinstalling git-review
 when I try to follow the directions in the first email. I think that part
 might be local to my machine -- but the general git-review 1.24 problem
 doesn't seem to be limited to OSX.

which package version do you have for 'python-setuptools', and what
does this show:
$ /usr/bin/easy_install --version

I reinstalled python-setuptools, and git review 1.24 is still working for me. :/

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] git-review 1.24 broken?

[John Mark Vandenberg]

Thanks Moriel.  Again I cant reproduce that! :-(

And I cant see any reference to pip 1.0 being a dependency in the
git-review code, however it may be implicit (a feature of pip used) or in
pbr somewhere.

Could you provide a complete backtrace for that exception?


On Thu, Jul 10, 2014 at 10:11 AM, Moriel Schottlender mor...@gmail.com
wrote:

 
  which package version do you have for 'python-setuptools', and what
  does this show:
  $ /usr/bin/easy_install --version
 
 I get 'setuptools 3.3'


  I reinstalled python-setuptools, and git review 1.24 is still working for
  me. :/
 

 So, when I try 'sudo pip install -U git-review' I get

 (...)
 File /usr/lib/python2.7/dist-packages/pkg_resources.py, line 628, in
 resolve
 raise DistributionNotFound(req)
 pkg_resources.DistributionNotFound: pip==1.0


 I tried 'sudo pip install --upgrade setuptools' but I get the exact same
 error.

 Git review seems to work, but it's still on version 1.23, and I keep
 getting the upgrade notice all the time.





  --
  John Vandenberg
 
  ___
  Wikitech-l mailing list
  Wikitech-l@lists.wikimedia.org
  https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 



 --
 No trees were harmed in the creation of this post.
 But billions of electrons, photons, and electromagnetic waves were terribly
 inconvenienced during its transmission!
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
John Vandenberg
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] How to Keep Track of Template Values Across a Wikipedia?

[John Mark Vandenberg]

On Jun 14, 2014 4:54 AM, Maximilian Klein isa...@gmail.com wrote:

 Hello All,

 I'm working on the Open-Access Signalling Project[1], which aims to signal
 and badge when a reference in Wikipedia is Open Access source. I'm writing
 the bot at the moment to do this, and I'm encountering a question - how do
 I keep track of the values of the template {{Cite doi | doi=value}}, in as
 close to real-time as possible?

 The most efficient approach I can come up with is to query the SQL servers
 on Labs in constant loop, returning the results of What transcludes {{Cite
 doi}} and seeing if the last_edited timestamp is newer than previous? If
 the last_edit is newer, then get the content of the page and see if the
 {{Cite_doi}} value has changed, checking against a local database.

 This seems horribly inefficient still. Is there a hook to know when a
 template on a page has been edited, rather than having to check every time
 the page has been edited?

The API can provide the list of URLs on the page, which may be enough for you.

It sounds like you want a API hook which returns (only) the reference
metadata for a page.  That would be lovely.  I would love to see an
option to provide those results in Zotero's JSON format.

COinS metadata can be downloaded in structured form.

1. get a list of sections

https://en.wikipedia.org/w/api.php?action=parseprop=sectionspage=COinS

2. fetch the formatted HTML for the relevant section(s)

https://en.wikipedia.org/w/api.php?action=mobileviewprop=textpage=COinSsections=7format=dumpfmnotransform=1

3. extract out the COinS metadata

Look in the page from 2. above - it should contain 'reference-*'

--
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] new terms of use/paid contributions - do they apply to mediawiki

[John Mark Vandenberg]

On Jun 17, 2014 3:25 AM, Brian Wolff bawo...@gmail.com wrote:

 Hi,

 So from what I understand, there's now been an amendment to WMF's
 terms of use to require disclosure of paid contributions [1]. Its a
 little unclear how this applies to MediaWiki as a project, but a
 literal reading of the policy makes it seem like MediaWiki is
 included.

 * MediaWiki is arguably a project of the Wikimedia foundation. The
 foundation's website says as much [2]
 *A commit/patchset certainly seems like a contribution.

 Thus the new policy would require anyone submitting code to use to
 declare who they work for. Personally this seems both unnecessary to
 me, as well as unlikely to be followed. For example, I see no reason
 why some person who uses our software should have to declare who they
 work for when they upstream a bug fix, etc.

 I would suggest we follow commons' lead [3], and declare that we do
 not have disclosure requirements for people giving us code.

 --bawolff

 [1]
https://wikimediafoundation.org/w/index.php?title=Terms_of_Usediff=0oldid=90463
 [2] https://wikimediafoundation.org/wiki/Our_projects
 [3]
https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Alternative_paid_contribution_disclosure_policy

I raised this issue prior to the amendment without getting a response.

https://meta.wikimedia.org/w/index.php?title=Talk:Terms_of_useoldid=7949115#What_services_are_covered_by_the_terms_of_use

--
John
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Replacing LiquidThreads for Flow

[John Mark Vandenberg]

On Jun 11, 2014 2:19 AM, Brian Wolff
 ...
 Id also like a way to turn a flow page back to wikitext for talk pages etc
 in case somebody doesnt like it (i havent been following closely enough to
 know if that is already possible)

Another option would be for existing LQT pages to 'be asked' if they
want to change back to wikitext mode while Flow is being deployed.
E.g. Some users may want to opt out now and add their user_talk on a
list to be migrated from LQT to wikitext and not be Flowified. This
will reduce the impact of they deploy, and may reduce the 'noise',
allowing more focus on issues relating to pages that want/need Flow.

Many LQT pages have never really used the functionality provided, have
only had a few posts each year with little interaction, and wont
benefit from being auto archived in order to have Flow enabled.

Bot operators could do page conversion. Im guessing that would involve
reproducing the LQT content as wikitext into the archive using the
liquidthreads API, and then switching the page back to wikitext mode.
Is page delete, page recreate the only way to switch it back to
wikitext mode?

--
John

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Bot flags and human-made edits

[John Mark Vandenberg]

On May 20, 2014 8:39 AM, Dan Garry dga...@wikimedia.org wrote:

 On 19 May 2014 19:36, Amir Ladsgroup ladsgr...@gmail.com wrote:

  As a bot operator I think API parameter about flagging bot or not is
  necessary
 

 Sure, but as I'm not a bot operator, can you explain why and what you use
 this for, to help me understand? :-)


  Implementing the parameter (whether the current status quo or my
  suggestion) in UI is important, I think adding a mark box besides minor
and
  watch options would be great (for bots, admins and flooders)
 

 I think so too. Display it only to people with the bot flag, and their
 edits will be tagged as bot edits iff they check the box. The API would be
 unaffected, and continue to function as it did before.

 As a product manager I am wary of adding more clutter into the already
busy
 edit interface, but given the scope of the feature (i.e. it only displays
 to people flagged as bots), I think it'd be okay.

If it is role based ('bot'), why not change it (in 1.24?) to always *not*
flag the UI edits as bot edits, displaying a warning to that effect?

Do we have many UI scaping bot frameworks in use these days?

--
John
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Free fonts and Windows users

[John Mark Vandenberg]

On Fri, Apr 11, 2014 at 6:26 AM, Brian Wolff bawo...@gmail.com wrote:
..
 By lower quality I mean both subjectively, but also objectively. For
 example, today I was reading
 https://en.wikinews.org/wiki/Hamilton_wins_%27incredible%27_Bahrain_race,_F1%27s_900th_Grand_Prix
 (enwikinews is one of the few wikis I haven't overridden the font
 changes with css). At first I thought there was a typo in the image
 caption toward the end of the page, an extra space between prote and
 stor. But no, the kerning on the font chosen is just literally that
 bad, that you can't tell if it is an extra space, or just a kerning
 error. I call that objectively bad (There's other things I don't like
 about the font choice, but they are more touchy-feely subjective)

I can reproduce this. It looks _very_ bad on Firefox/Fedora Core 19,
and but it also appears to a lesser degree on Chrome/Fedora Core 19.

Compare it against monobook

https://en.wikinews.org/wiki/Hamilton_wins_%27incredible%27_Bahrain_race,_F1%27s_900th_Grand_Prix?useskin=monobook

See also the ама in the bold title in the first sentence of

https://ru.wikipedia.org/wiki/?oldid=58319520

vs

https://ru.wikipedia.org/wiki/?oldid=58319520useskin=monobook

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Free fonts and Windows users

[John Mark Vandenberg]

On Apr 9, 2014 2:02 PM, Steven Walling steven.wall...@gmail.com wrote:

 On Tue, Apr 8, 2014 at 10:05 PM, Risker risker...@gmail.com wrote:

  It's pretty clear that the objectives of this project are not
successfully
  met at this point, and in fact have caused major problems on non-Latin
  script WMF sites, and significant but less critical problems on Latin
  script sites. Several factors for this have been identified in the
thread -
  including limited attention to the effects on non-Latin script projects,
  the insertion of philosophical principles (FOSS) without a clear
  understanding of the effects this would have on the outcome, and the
  unwillingness to step back when a major change results in loss of
  functionality.
 

 [citation needed]

 Loss of functionality? The functionality we're talking about here is
 reading of Wikimedia content. It's the most core, most basic functionality
 we have. In the case of VisualEditor, which picks up read-mode typography
 styles, it's also editing.

 Did reading suddenly become seriously impaired? No.

Yes, it was seriously impaired. On the scale of impairments possible with a
font stack change, compared to how it was, this change was in the high end
of the scale, next to broken.

I currently visit a lot of language pedias doing category linking on
wikidata, and after the change I noticed many had obviously bad font issues
that I didnt see the day before. Clicking random on a few wikis would have
been enough testing to work that out.

Btw that was using Chrome on Fedora Core 19 without any local mods to font
handling.

The functionality was previously fine. Maybe it could be improved, but that
isnt what happened.

I agree with everything Risker said. I go further and suggest the team
involved stops defending their goals and implementation. The former are not
the issue, and the latter was indefensible. I havent looked at how much
testing was done, or if there was some staging of the rollout, but it is
clear that it wasnt careful enough.

--
John
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Free fonts and Windows users

[John Mark Vandenberg]

On Wed, Apr 9, 2014 at 5:06 PM, David Gerard dger...@gmail.com wrote:
 On 9 April 2014 09:26, John Mark Vandenberg jay...@gmail.com wrote:

  I havent looked at how much
 testing was done, or if there was some staging of the rollout, but it is
 clear that it wasnt careful enough.


 To be fair, it's easy to say that in hindsight. But e.g. who knew so
 many people were running Windows with ClearType off, and that these
 fonts rendered so badly under it? I mean *now* we know to look at that
 specific thing :-)

Did you even read my email?

-- 
John Vandenberg

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] python bot/script with javascript as frontend

[John Mark Vandenberg]

English Wikisource had a bot which would perform OCR on request by a user
when the user pressed a button in the Wikisource website interface.

Is that like what you are trying to achieve?

I think it used categories as a work queue. JavaScript placed the page in a
category, and the bot picked up the page from the category and removed it
from the category once the bot task had completed.

Some tools use a wiki page as a register of bot commands to be executed by
the bot.

If you need more complex interaction between user and bot via JavaScript,
the bot will likely need to provide a http listener that the JS and bot can
communicate via.
 *Hi*

*How do I build a python bot/script with JavaScript for the frontend
(ideally as Wikisouorce gadget https://www.mediawiki.org/wiki/Gadget).*

*Project is for gsoc2014. So it should be something which can be live on
wikisource gadgets.*


*Thanks*

*---*

*Rohit Dua*
*8ohit.dua*
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l