Re: [Wikitech-l] What happened to our user agent requirements?

[Brion Vibber]

I'm not 100% convinced that the UA requirement is helpful, for two reasons:

1) Lots of requests will have default like "PHP" or "Python/urllib" or
whatever from the tool they used to build their bot. These aren't helpful
either as they contain no of how to get in touch.

2) It's trivial to work around the requirement for a non-blank UA by
setting one of the above, or worse -- cut-n-pasting the UA string from a
browser. If someone hacks this up real quick while testing, they may never
bother putting in contact information when their bot moves from a handful
of requests to gazillions.

Auto-throttling super-high-rate API clients (by IP/IP group) and giving
them an explicit "You really should contact us and, better yet, make it
possible for us to contact you" message might be nice.


We may want to seriously think about some sort of API key system... not
necessarily as mandatory for access (we love freedom and convenience!) but
perhaps as the way you get around being throttled for too many accesses.
This would give us a structured way of storing their contact information,
which might be better than unstructured names or addresses in the UA.

Does it make sense to tell people "log in to your bot's account with OAuth"
or is that too much of a pain in the ass versus "add this one parameter to
your requests with your key"? :)

-- brion


On Tue, Sep 1, 2015 at 10:23 AM, Oliver Keyes  wrote:

> Awesome; thanks for the analysis, Krinkle.
>
> Do we want to change this behaviour? From my point of view the answer
> is 'yes, not setting any kind of user agent is a violation of our API
> etiquette and we should be taking steps to alert people that it is'
> but if other people have different perspectives on this I'd love to
> hear them.
>
> On 1 September 2015 at 13:18, Krinkle  wrote:
> > I've confirmed just now that whatever requirement there was, it doesn't
> seem to be in effect.
> >
> > Both omitting the header entirely, sending it with empty string, and
> sending
> > with "-"; – all three result in a response from the MediaWiki API.
> >
> > $ curl -A '' --include -v '
> https://en.wikipedia.org/w/api.php?action=query=json' <
> https://en.wikipedia.org/w/api.php?action=query=json'>
> >> GET /w/api.php?action=query=json HTTP/1.1
> >> Host: en.wikipedia.org
> >> Accept: */*
> > < HTTP/1.1 200 OK
> > ..
> > {"batchcomplete":""}
> >
> >
> > $ curl -A '-' --include -v '
> https://en.wikipedia.org/w/api.php?action=query=json' <
> https://en.wikipedia.org/w/api.php?action=query=json'>
> >> GET /w/api.php?action=query=json HTTP/1.1
> >> User-Agent: -
> >> Host: en.wikipedia.org 
> >> Accept: */*
> > < HTTP/1.1 200 OK
> > ..
> > {"batchcomplete":""}
> >
> > In the past (2012?) these were definitely being blocked. (Ran into it
> from time to time on Toolserver)
> > It seems php file_get_contents('http://...api..' ) is
> also working fine now,
> > without having to init_set a user_agent value first.
> >
> > -- Krinkle
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Oliver Keyes
> Count Logula
> Wikimedia Foundation
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Delays in loading pages on Wikimedia sites

[Pine W]

Yesterday and today, I've experienced a number of delays in getting pages
to load on Wikimedia sites. My connectivity to other websites is normal, so
I think something may be going on internally with Wikimedia connectivity. I
recall seeing discussions during Mexico Wikimania about testing connections
to Wikimedia sites systematically from geographically dispersed locations.
Is that kind of testing being done? Are there any other tools which might
explain the delays in getting connections and loading pages?

Thanks,

Pine
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Oliver Keyes]

Specifically, the hypothesis that people are sending "-"?

On 1 September 2015 at 12:58, Tomasz Finc  wrote:
> Let's get a task in phab for this so that we can triage next steps.
> I'm curious about this as well.
>
> --tomasz
>
> On Tue, Sep 1, 2015 at 9:46 AM, Oliver Keyes  wrote:
>> On 1 September 2015 at 12:42, John  wrote:
>>> Could they be sending a non-standard header of "-"
>>
>> Perfectly possible although also impossible to detect :(
>>
>>>
>>> On Tuesday, September 1, 2015, Chad  wrote:
>>>
 On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes > wrote:

 > Is the
 > blocking of requests absent a user agent simply happening at a
 > 'higher' stage (in mediawiki itself?) and so not registering with the
 > varnishes,


 No, it's not done at the application level.


 > or is sending an /empty/ header simply A-OK?
 >
 >
 Shouldn't be, unless the policy changed...

 -Chad
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org 
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
>>
>> --
>> Oliver Keyes
>> Count Logula
>> Wikimedia Foundation
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Tomasz Finc]

Tracking the overall issue

On Tue, Sep 1, 2015 at 9:59 AM, Oliver Keyes  wrote:
> Specifically, the hypothesis that people are sending "-"?
>
> On 1 September 2015 at 12:58, Tomasz Finc  wrote:
>> Let's get a task in phab for this so that we can triage next steps.
>> I'm curious about this as well.
>>
>> --tomasz
>>
>> On Tue, Sep 1, 2015 at 9:46 AM, Oliver Keyes  wrote:
>>> On 1 September 2015 at 12:42, John  wrote:
 Could they be sending a non-standard header of "-"
>>>
>>> Perfectly possible although also impossible to detect :(
>>>

 On Tuesday, September 1, 2015, Chad  wrote:

> On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes  > wrote:
>
> > Is the
> > blocking of requests absent a user agent simply happening at a
> > 'higher' stage (in mediawiki itself?) and so not registering with the
> > varnishes,
>
>
> No, it's not done at the application level.
>
>
> > or is sending an /empty/ header simply A-OK?
> >
> >
> Shouldn't be, unless the policy changed...
>
> -Chad
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
 ___
 Wikitech-l mailing list
 Wikitech-l@lists.wikimedia.org
 https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>>
>>>
>>>
>>> --
>>> Oliver Keyes
>>> Count Logula
>>> Wikimedia Foundation
>>>
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Oliver Keyes
> Count Logula
> Wikimedia Foundation
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Tomasz Finc]

Let's get a task in phab for this so that we can triage next steps.
I'm curious about this as well.

--tomasz

On Tue, Sep 1, 2015 at 9:46 AM, Oliver Keyes  wrote:
> On 1 September 2015 at 12:42, John  wrote:
>> Could they be sending a non-standard header of "-"
>
> Perfectly possible although also impossible to detect :(
>
>>
>> On Tuesday, September 1, 2015, Chad  wrote:
>>
>>> On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes >> > wrote:
>>>
>>> > Is the
>>> > blocking of requests absent a user agent simply happening at a
>>> > 'higher' stage (in mediawiki itself?) and so not registering with the
>>> > varnishes,
>>>
>>>
>>> No, it's not done at the application level.
>>>
>>>
>>> > or is sending an /empty/ header simply A-OK?
>>> >
>>> >
>>> Shouldn't be, unless the policy changed...
>>>
>>> -Chad
>>> ___
>>> Wikitech-l mailing list
>>> Wikitech-l@lists.wikimedia.org 
>>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
>
>
> --
> Oliver Keyes
> Count Logula
> Wikimedia Foundation
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Trey Jones]

I agree with rate-limiting those without some sort of ID (login or API key).

As Oliver said, big (ab)users can massively skew our stats, often by
themselves. But hordes of upper middle volume bots (way too high for a
human, nowhere near the max for a superstar bot) can have a large
cumulative effect, too. We can't track them down individually, or even
detect that they are there because they are "only" involved in a fraction
of a percent of traffic—but a hundred such bots add up to a significant
skew, and reasonable rate limits could knock them down to manageable levels.

While enforcing UA requirements is inherently reasonable, anyone who
doesn't know to set up a valid UA string may not know to not just copy one
from a browser to make things worse. (I've done that myself in the past
when using curl with an uncooperative site. The shame.) Maybe rate limiting
will be the 80 in the 80/20 solution, and enforcing UA reqs won't be
necessary to control traffic, leaving them as a silly but effective way of
identifying certain kinds of traffic. The flip-side case would be
bajillions of very low volume bots—mimicking roughly human levels of
traffic and so sailing under rate limits—all with blank UAs. But we could
note that after rate limiting slows down the ridiculously heavy hitters and
take action as needed.


Trey Jones
Software Engineer, Discovery
Wikimedia Foundation

On Tue, Sep 1, 2015 at 1:44 PM, Oliver Keyes  wrote:

> If people aren't capable of following UA guidelines I doubt they're
> going to follow voluntary login.
>
> For what it's worth I absolutely support both rate-limiting and login
> to get around this. In fact, I would argue that from an analytics
> point of view rate limiting is probably the most high-profile problem
> we have with incoming data at the moment. It's far, far too common for
> random pieces of automata to set themselves up and massively skew our
> datasets; identifying this in advance is impossible (we don't always
> have IP data) and identifying them post-hoc on an individual basis is
> massively time consuming.
>
> Why don't we have rate limiting + login? Who would work on this? Why
> /should/ we not have rate limiting?
>
> On 1 September 2015 at 13:37, Brion Vibber  wrote:
> > I'm not 100% convinced that the UA requirement is helpful, for two
> reasons:
> >
> > 1) Lots of requests will have default like "PHP" or "Python/urllib" or
> > whatever from the tool they used to build their bot. These aren't helpful
> > either as they contain no of how to get in touch.
> >
> > 2) It's trivial to work around the requirement for a non-blank UA by
> > setting one of the above, or worse -- cut-n-pasting the UA string from a
> > browser. If someone hacks this up real quick while testing, they may
> never
> > bother putting in contact information when their bot moves from a handful
> > of requests to gazillions.
> >
> > Auto-throttling super-high-rate API clients (by IP/IP group) and giving
> > them an explicit "You really should contact us and, better yet, make it
> > possible for us to contact you" message might be nice.
> >
> >
> > We may want to seriously think about some sort of API key system... not
> > necessarily as mandatory for access (we love freedom and convenience!)
> but
> > perhaps as the way you get around being throttled for too many accesses.
> > This would give us a structured way of storing their contact information,
> > which might be better than unstructured names or addresses in the UA.
> >
> > Does it make sense to tell people "log in to your bot's account with
> OAuth"
> > or is that too much of a pain in the ass versus "add this one parameter
> to
> > your requests with your key"? :)
> >
> > -- brion
> >
> >
> > On Tue, Sep 1, 2015 at 10:23 AM, Oliver Keyes 
> wrote:
> >
> >> Awesome; thanks for the analysis, Krinkle.
> >>
> >> Do we want to change this behaviour? From my point of view the answer
> >> is 'yes, not setting any kind of user agent is a violation of our API
> >> etiquette and we should be taking steps to alert people that it is'
> >> but if other people have different perspectives on this I'd love to
> >> hear them.
> >>
> >> On 1 September 2015 at 13:18, Krinkle  wrote:
> >> > I've confirmed just now that whatever requirement there was, it
> doesn't
> >> seem to be in effect.
> >> >
> >> > Both omitting the header entirely, sending it with empty string, and
> >> sending
> >> > with "-"; – all three result in a response from the MediaWiki API.
> >> >
> >> > $ curl -A '' --include -v '
> >> https://en.wikipedia.org/w/api.php?action=query=json' <
> >> https://en.wikipedia.org/w/api.php?action=query=json'>
> >> >> GET /w/api.php?action=query=json HTTP/1.1
> >> >> Host: en.wikipedia.org
> >> >> Accept: */*
> >> > < HTTP/1.1 200 OK
> >> > ..
> >> > {"batchcomplete":""}
> >> >
> >> >
> >> > $ curl -A '-' --include -v '
> >> 

Re: [Wikitech-l] Delays in loading pages on Wikimedia sites

[Keegan Peterzell]

On Tue, Sep 1, 2015 at 2:13 PM, Pine W  wrote:

> Yesterday and today, I've experienced a number of delays in getting pages
> to load on Wikimedia sites. My connectivity to other websites is normal, so
> I think something may be going on internally with Wikimedia connectivity. I
> recall seeing discussions during Mexico Wikimania about testing connections
> to Wikimedia sites systematically from geographically dispersed locations.
> Is that kind of testing being done? Are there any other tools which might
> explain the delays in getting connections and loading pages?
>
> Thanks,
>
> Pine


Hm, I don't see anything alarming here:
http://status.wikimedia.org/8777/169667/https-services---wikipedia


-- 
Keegan Peterzell
Community Liaison, Product
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Delays in loading pages on Wikimedia sites

[Pine W]

OK, thanks.

Pine


On Tue, Sep 1, 2015 at 12:35 PM, Keegan Peterzell 
wrote:

> On Tue, Sep 1, 2015 at 2:13 PM, Pine W  wrote:
>
> > Yesterday and today, I've experienced a number of delays in getting pages
> > to load on Wikimedia sites. My connectivity to other websites is normal,
> so
> > I think something may be going on internally with Wikimedia
> connectivity. I
> > recall seeing discussions during Mexico Wikimania about testing
> connections
> > to Wikimedia sites systematically from geographically dispersed
> locations.
> > Is that kind of testing being done? Are there any other tools which might
> > explain the delays in getting connections and loading pages?
> >
> > Thanks,
> >
> > Pine
>
>
> Hm, I don't see anything alarming here:
> http://status.wikimedia.org/8777/169667/https-services---wikipedia
>
>
> --
> Keegan Peterzell
> Community Liaison, Product
> Wikimedia Foundation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Oliver Keyes]

Awesome; thanks for the analysis, Krinkle.

Do we want to change this behaviour? From my point of view the answer
is 'yes, not setting any kind of user agent is a violation of our API
etiquette and we should be taking steps to alert people that it is'
but if other people have different perspectives on this I'd love to
hear them.

On 1 September 2015 at 13:18, Krinkle  wrote:
> I've confirmed just now that whatever requirement there was, it doesn't seem 
> to be in effect.
>
> Both omitting the header entirely, sending it with empty string, and sending
> with "-"; – all three result in a response from the MediaWiki API.
>
> $ curl -A '' --include -v 
> 'https://en.wikipedia.org/w/api.php?action=query=json' 
> 
>> GET /w/api.php?action=query=json HTTP/1.1
>> Host: en.wikipedia.org
>> Accept: */*
> < HTTP/1.1 200 OK
> ..
> {"batchcomplete":""}
>
>
> $ curl -A '-' --include -v 
> 'https://en.wikipedia.org/w/api.php?action=query=json' 
> 
>> GET /w/api.php?action=query=json HTTP/1.1
>> User-Agent: -
>> Host: en.wikipedia.org 
>> Accept: */*
> < HTTP/1.1 200 OK
> ..
> {"batchcomplete":""}
>
> In the past (2012?) these were definitely being blocked. (Ran into it from 
> time to time on Toolserver)
> It seems php file_get_contents('http://...api..' ) is also 
> working fine now,
> without having to init_set a user_agent value first.
>
> -- Krinkle
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Oliver Keyes]

If people aren't capable of following UA guidelines I doubt they're
going to follow voluntary login.

For what it's worth I absolutely support both rate-limiting and login
to get around this. In fact, I would argue that from an analytics
point of view rate limiting is probably the most high-profile problem
we have with incoming data at the moment. It's far, far too common for
random pieces of automata to set themselves up and massively skew our
datasets; identifying this in advance is impossible (we don't always
have IP data) and identifying them post-hoc on an individual basis is
massively time consuming.

Why don't we have rate limiting + login? Who would work on this? Why
/should/ we not have rate limiting?

On 1 September 2015 at 13:37, Brion Vibber  wrote:
> I'm not 100% convinced that the UA requirement is helpful, for two reasons:
>
> 1) Lots of requests will have default like "PHP" or "Python/urllib" or
> whatever from the tool they used to build their bot. These aren't helpful
> either as they contain no of how to get in touch.
>
> 2) It's trivial to work around the requirement for a non-blank UA by
> setting one of the above, or worse -- cut-n-pasting the UA string from a
> browser. If someone hacks this up real quick while testing, they may never
> bother putting in contact information when their bot moves from a handful
> of requests to gazillions.
>
> Auto-throttling super-high-rate API clients (by IP/IP group) and giving
> them an explicit "You really should contact us and, better yet, make it
> possible for us to contact you" message might be nice.
>
>
> We may want to seriously think about some sort of API key system... not
> necessarily as mandatory for access (we love freedom and convenience!) but
> perhaps as the way you get around being throttled for too many accesses.
> This would give us a structured way of storing their contact information,
> which might be better than unstructured names or addresses in the UA.
>
> Does it make sense to tell people "log in to your bot's account with OAuth"
> or is that too much of a pain in the ass versus "add this one parameter to
> your requests with your key"? :)
>
> -- brion
>
>
> On Tue, Sep 1, 2015 at 10:23 AM, Oliver Keyes  wrote:
>
>> Awesome; thanks for the analysis, Krinkle.
>>
>> Do we want to change this behaviour? From my point of view the answer
>> is 'yes, not setting any kind of user agent is a violation of our API
>> etiquette and we should be taking steps to alert people that it is'
>> but if other people have different perspectives on this I'd love to
>> hear them.
>>
>> On 1 September 2015 at 13:18, Krinkle  wrote:
>> > I've confirmed just now that whatever requirement there was, it doesn't
>> seem to be in effect.
>> >
>> > Both omitting the header entirely, sending it with empty string, and
>> sending
>> > with "-"; – all three result in a response from the MediaWiki API.
>> >
>> > $ curl -A '' --include -v '
>> https://en.wikipedia.org/w/api.php?action=query=json' <
>> https://en.wikipedia.org/w/api.php?action=query=json'>
>> >> GET /w/api.php?action=query=json HTTP/1.1
>> >> Host: en.wikipedia.org
>> >> Accept: */*
>> > < HTTP/1.1 200 OK
>> > ..
>> > {"batchcomplete":""}
>> >
>> >
>> > $ curl -A '-' --include -v '
>> https://en.wikipedia.org/w/api.php?action=query=json' <
>> https://en.wikipedia.org/w/api.php?action=query=json'>
>> >> GET /w/api.php?action=query=json HTTP/1.1
>> >> User-Agent: -
>> >> Host: en.wikipedia.org 
>> >> Accept: */*
>> > < HTTP/1.1 200 OK
>> > ..
>> > {"batchcomplete":""}
>> >
>> > In the past (2012?) these were definitely being blocked. (Ran into it
>> from time to time on Toolserver)
>> > It seems php file_get_contents('http://...api..' ) is
>> also working fine now,
>> > without having to init_set a user_agent value first.
>> >
>> > -- Krinkle
>> > ___
>> > Wikitech-l mailing list
>> > Wikitech-l@lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
>>
>>
>> --
>> Oliver Keyes
>> Count Logula
>> Wikimedia Foundation
>>
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Brad Jorsch (Anomie)]

On Tue, Sep 1, 2015 at 1:18 PM, Krinkle  wrote:

> In the past (2012?) these were definitely being blocked. (Ran into it from
> time to time on Toolserver)
> It seems php file_get_contents('http://...api..' ) is
> also working fine now,
> without having to init_set a user_agent value first.
>

I wonder if it got lost in the move from Squid to Varnish, or something
along those lines.


-- 
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Krinkle]

I've confirmed just now that whatever requirement there was, it doesn't seem to 
be in effect.

Both omitting the header entirely, sending it with empty string, and sending
with "-"; – all three result in a response from the MediaWiki API.

$ curl -A '' --include -v 
'https://en.wikipedia.org/w/api.php?action=query=json' 

> GET /w/api.php?action=query=json HTTP/1.1
> Host: en.wikipedia.org
> Accept: */*
< HTTP/1.1 200 OK
..
{"batchcomplete":""}


$ curl -A '-' --include -v 
'https://en.wikipedia.org/w/api.php?action=query=json' 

> GET /w/api.php?action=query=json HTTP/1.1
> User-Agent: -
> Host: en.wikipedia.org 
> Accept: */*
< HTTP/1.1 200 OK
..
{"batchcomplete":""}

In the past (2012?) these were definitely being blocked. (Ran into it from time 
to time on Toolserver)
It seems php file_get_contents('http://...api..' ) is also 
working fine now,
without having to init_set a user_agent value first.

-- Krinkle
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] New RFC: better JS minification

[MZMcBride]

Ori Livneh wrote:
>On Tue, Sep 1, 2015 at 8:43 AM, Jérémie Roquet 
>wrote:
>> Has the RFC been abandonned because of lack of interest?
>
>Speaking for myself: at the time the RFC was written, I was skeptical that
>the benefits would be worth incurring a dependency on an external service
>(or the requirement of shelling out). But since then we have had a lot of
>discussions (in the RFC meetings) on how to do micro-services correctly,
>and some good abstractions have come out of that. So +1 for reviving it.

I'd personally prefer that we move in the other direction, removing the
minification. I think it's harmful to the open Web to minify, or worse,
obfuscate our code. As expected, we've encountered a number of bugs that
disappear when the debug=true URL parameter is specified. And given other
performance optimization work and existing optimizations such as gzip
compression, I don't see the benefits outweighing the costs.

MZMcBride



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Brandon Black]

On Tue, Sep 1, 2015 at 10:42 PM, Platonides  wrote:
> Brad Jorsch (Anomie) wrote:
>> I wonder if it got lost in the move from Squid to Varnish, or something
>> along those lines.
> That's likely, given that it was enforced by squid.

We could easily add it back in Varnish, too, but I tend to agree with
Brion's points that it's not ultimately helpful.

I really do like the idea of moving towards smarter ratelimiting of
APIs by default, though (and have brought this up in several contexts
recently, but I'm not really aware of whatever past work we've done in
that direction).  From that relatively-ignorant perspective, I tend to
envision an architecture where the front edge ratelimits API requests
(or even possibly, all requests, but we'd probably have to exclude a
lot of common spiders...) via a simple token-bucket-filter if they're
anonymous, but lets them run free if they superficially appear to have
a legitimate cookie or API access token.  Then it's up to the app
layer to enforce limits for the seemingly-identifiable traffic and be
configurable to raise them for legitimate remote clients we've had
contact with, and to reject legitimate-looking tokens/logins that the
edge choses not to ratelimit which aren't actually legitimate.

-- Brandon

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Greg Grossmeier]

> It's been discussed previously both on this list and elsewhere, but for
> better or worse the Wikimedia Foundation has an entire "Mobile apps" team
> that pretty much exclusively works on closed platforms, as I understand
> it. They've gone as far as to abandon Gerrit in favor of GitHub.

For the record, it's just the iOS team that moved to Github. Android is
still in Gerrit. The iOS move was for CI reasons (summary: we (WMF
RelEng) can't support the OSX platform for build and test cases with any
ease, especially vis a vis other priorities).

https://gerrit.wikimedia.org/r/#/q/status:merged+project:apps/android/wikipedia+branch:master,n,z

Greg

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @gregA18D 1138 8E47 FAC8 1C7D |

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] New RFC: better JS minification

[Max Semenik]

On Tue, Sep 1, 2015 at 8:49 AM, Ori Livneh  wrote:

> So +1 for reviving it.
>

Feel free to, however I myself have neither the time nor inclination to
work on this; the components are:
* The extension itself:
https://github.com/wikimedia/mediawiki-extensions-Minifier
* Minifier service in extension's /server/ directory
* Core change https://gerrit.wikimedia.org/r/#/c/74293/

-- 
Best regards,
Max Semenik ([[User:MaxSem]])
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Phabricator monthly statistics - 2015-08

[Quim Gil]

Wikimedia Phabricator will be soon one year old!

On Tue, Sep 1, 2015 at 2:00 AM,  wrote:

>
> Number of accounts created in (2015-08): 288
>

Kind of surprised about the fact that we keep having almost ten new
Phabricator users every day, I have created a graph at
https://www.mediawiki.org/wiki/Community_metrics#New_accounts_in_Phabricator

Pretty impressive. There are about 3640 valid users today.


> Number of active users (any activity) in (2015-08): 831
>

Another very respectable number, that is slightly growing every month:
https://www.mediawiki.org/wiki/Community_metrics#Active_users_in_Phabricator

-- 
Quim Gil
Engineering Community Manager @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Gergo Tisza]

On Tue, Sep 1, 2015 at 4:54 PM, Brandon Black  wrote:

> I really do like the idea of moving towards smarter ratelimiting of
> APIs by default, though (and have brought this up in several contexts
> recently, but I'm not really aware of whatever past work we've done in
> that direction).  From that relatively-ignorant perspective, I tend to
> envision an architecture where the front edge ratelimits API requests
> (or even possibly, all requests, but we'd probably have to exclude a
> lot of common spiders...) via a simple token-bucket-filter if they're
> anonymous, but lets them run free if they superficially appear to have
> a legitimate cookie or API access token.  Then it's up to the app
> layer to enforce limits for the seemingly-identifiable traffic and be
> configurable to raise them for legitimate remote clients we've had
> contact with, and to reject legitimate-looking tokens/logins that the
> edge choses not to ratelimit which aren't actually legitimate.
>

Rate limiting / UA policy enforcement has to be done in Varnish, since API
responses can be cached there and so the requests don't necessarily reach
higher layers (and we wouldn't want to vary on user agent).
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Gabriel Wicke]

On Tue, Sep 1, 2015 at 5:54 PM, Gergo Tisza  wrote:
>
>
> Rate limiting / UA policy enforcement has to be done in Varnish, since API
> responses can be cached there and so the requests don't necessarily reach
> higher layers (and we wouldn't want to vary on user agent).



The cost / benefit trade-offs for Varnish cache hits are fairly different
from those of cache misses. Especially for in-memory (frontend) hits it
might overall be cheaper to send a regular response, rather than adding
rate limit overheads to each cache hit.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Oliver Keyes]

What was the publicising of the campaign prior to its launch?

It should be pretty apparent to people with experience within the
movement that this would be both entirely novel and pretty
controversial. I'd expect some amount of transparency around it (a
phabricator ticket is not, in and of itself, transparency). To
contrast, with search when we make /experimental/ modifications to the
user experience of a tiny sample (through A/B testing) we not only
list those changes in phabricator but also send explicit mailing list
announcements - and those effect a smaller chunk of our user base on a
platform.

On 1 September 2015 at 22:51, Greg Grossmeier  wrote:
> 
>> It's been discussed previously both on this list and elsewhere, but for
>> better or worse the Wikimedia Foundation has an entire "Mobile apps" team
>> that pretty much exclusively works on closed platforms, as I understand
>> it. They've gone as far as to abandon Gerrit in favor of GitHub.
>
> For the record, it's just the iOS team that moved to Github. Android is
> still in Gerrit. The iOS move was for CI reasons (summary: we (WMF
> RelEng) can't support the OSX platform for build and test cases with any
> ease, especially vis a vis other priorities).
>
> https://gerrit.wikimedia.org/r/#/q/status:merged+project:apps/android/wikipedia+branch:master,n,z
>
> Greg
>
> --
> | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
> | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D |
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] What does "Director of Architecture" mean anyway?

[Rob Lanphier]

Hi everyone,

It's been a while since I've been involved here, and I've got a pretty good
excuse.  I'm going to avoid going into too much detail, but in short, I was
involved in a pretty nasty traffic collision, and have only recently
returned to work.  It turned out the way that a lot of bike versus car
collisions go, which is to say, I lost  :-(

The good news (for me at least), I've largely recovered.  The surgeons here
did a remarkable job putting me back together again.  I returned to
part-time work a couple of weeks ago, and I'm getting back up to speed on
things.  I still have a number of personal obligations that keep me from
working full-time, but I'll keep my user page on mediawiki.org
 updated with the latest (or
point you to where I plan to keep that info).

Though we hadn't announced it in April, we had *just* talked about making
my title "Director of Architecture" in the big reorg we had here at WMF.
We hadn't worked out exactly what that meant when I was in the wreck, but I
had a vague idea which I intend to flesh out.  In the short term, I want to
facilitate the activity of the Architecture Committee
.  I've started
chairing the private meetings of that group, and I'm taking on a bunch of
the administrivia.  I'll make things clearer as they become clearer in my
head  :-)  Well, maybe after that, and after I get a chance to write down
what I'm thinking.
I'm really happy to be back involved!  \o/  If you have any questions, or
you simply want to wave back, feel free to drop a comment on my MediaWiki
talk page:
https://www.mediawiki.org/wiki/User_talk:RobLa-WMF

Rob
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[MZMcBride]

Ori Livneh wrote:
>We appear to be running a banner campaign on the mobile web site, driving
>people to download the mobile app:
>
>https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2
>https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1
>
>[...]
>
>This isn't cool. This isn't us. We don't drive people from an open
>platform to a closed one.

It's been discussed previously both on this list and elsewhere, but for
better or worse the Wikimedia Foundation has an entire "Mobile apps" team
that pretty much exclusively works on closed platforms, as I understand
it. They've gone as far as to abandon Gerrit in favor of GitHub. I agree
with the general sentiment of your post, but the issues here are deeper.

MZMcBride



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Gabriel Wicke]

We recently revisited rate limiting in
https://phabricator.wikimedia.org/T107934, but came to similar conclusions
as reached in this thread:


   - Limits for weak identifiers like IPs or user agents would (at least
   initially) need to be high enough to render the limiting borderline useless
   against DDOS attacks.
   - Stronger authentication requirements have significant costs to users,
   and will require non-trivial backend work to keep things efficient on our
   end. I believe we should tackle this backend work in any case, but it will
   take some time.
   - In our benchmarks, most off-the-shelf rate limiting libraries use
   per-request network requests to a central service like Redis, which costs
   latency and throughput, and has some scaling challenges. There are
   algorithms [1] that trade some precision for performance, but we aren't
   aware of any open source implementations we could use.

The dual of rate limiting is making each API request cheaper. We have
recently made some progress towards limiting the cost of individual API
requests, and are working towards making most API end points cacheable &
backed by storage.

Gabriel

[1]:
http://yahooeng.tumblr.com/post/111288877956/cloud-bouncer-distributed-rate-limiting-at-yahoo


On Tue, Sep 1, 2015 at 4:54 PM, Brandon Black  wrote:

> On Tue, Sep 1, 2015 at 10:42 PM, Platonides  wrote:
> > Brad Jorsch (Anomie) wrote:
> >> I wonder if it got lost in the move from Squid to Varnish, or something
> >> along those lines.
> > That's likely, given that it was enforced by squid.
>
> We could easily add it back in Varnish, too, but I tend to agree with
> Brion's points that it's not ultimately helpful.
>
> I really do like the idea of moving towards smarter ratelimiting of
> APIs by default, though (and have brought this up in several contexts
> recently, but I'm not really aware of whatever past work we've done in
> that direction).  From that relatively-ignorant perspective, I tend to
> envision an architecture where the front edge ratelimits API requests
> (or even possibly, all requests, but we'd probably have to exclude a
> lot of common spiders...) via a simple token-bucket-filter if they're
> anonymous, but lets them run free if they superficially appear to have
> a legitimate cookie or API access token.  Then it's up to the app
> layer to enforce limits for the seemingly-identifiable traffic and be
> configurable to raise them for legitimate remote clients we've had
> contact with, and to reject legitimate-looking tokens/logins that the
> edge choses not to ratelimit which aren't actually legitimate.
>
> -- Brandon
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>



-- 
Gabriel Wicke
Principal Engineer, Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Brandon Black]

On Wed, Sep 2, 2015 at 1:21 AM, Gabriel Wicke  wrote:
> On Tue, Sep 1, 2015 at 5:54 PM, Gergo Tisza  wrote:
>>
>>
>> Rate limiting / UA policy enforcement has to be done in Varnish, since API
>> responses can be cached there and so the requests don't necessarily reach
>> higher layers (and we wouldn't want to vary on user agent).
>
>
>
> The cost / benefit trade-offs for Varnish cache hits are fairly different
> from those of cache misses. Especially for in-memory (frontend) hits it
> might overall be cheaper to send a regular response, rather than adding
> rate limit overheads to each cache hit.

Yeah I was mostly thinking of uncacheable API accesses.  If we can
cache it, we don't mind (as much) in terms of load/abuse.  By having
the simpler outer check in varnish, though, it takes the big load from
anonymous spikes away from being handled at the applayer for those
uncacheable hits.

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Christian Aistleitner]

On Tue, Sep 01, 2015 at 12:42:35PM -0400, John wrote:
> Could they be sending a non-standard header of "-"

They could.

But if a request comes in without a User-Agent header, the logging
pipeline silently translates it into "-".

Have fun,
Christian



P.S.: The relevant configuration (for webrequests) is at

https://github.com/wikimedia/operations-puppet/blob/production/modules/role/manifests/cache/kafka/webrequest.pp#L26

That long line contains '%{User-Agent@user_agent}i', which means

  log the request's User-Agent header

but no default value is provided. As no default value is provided,
varnishkafka uses the pre-set default value, which is "-":

https://github.com/wikimedia/varnishkafka/blob/master/varnishkafka.c#L246

This conversion from the empty string to "-" does not kill relevant
information and is useful for some researchers when manually
inspecting TSVs, or manually browsing Hive output.



-- 
 quelltextlich e.U.  \\  Christian Aistleitner 
   Companies' registry: 360296y in Linz
Christian Aistleitner
Kefermarkterstrasze 6a/3 Email:  christ...@quelltextlich.at
4293 Gutau, Austria  Phone:  +43 7946 / 20 5 81
 Fax:+43 7946 / 20 5 81
 Homepage: http://quelltextlich.at/
---


signature.asc
Description: Digital signature
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Collaboration team reprioritization

[Danny Horn]

For a while now, the Collaboration team has been working on Flow, the
structured discussion system. I want to let you know about some changes in
that long-term plan.

While initial announcements about Flow said that it would be a universal
replacement for talk pages, the features that were ultimately built into
Flow were specifically forum-style group discussion tools. But article and
project talk pages are used for a number of important and complex processes
that those tools aren't able to handle, making Flow unsuitable for
deployment on those kinds of pages.

To better address the needs of our core contributors, we're now focusing
our strategy on the curation, collaboration, and admin processes that take
place on a variety of pages. Many of these processes use complex
workarounds -- templates, categories, transclusions, and lots of
instructions -- that turn blank wikitext talk pages into structured
workflows. There are gadgets and user scripts on the larger wikis to help
with some of these workflows, but these tools aren't standardized or
universally available.

As these workflows grow in complexity, they become more difficult for the
next generation of editors to learn and use. This has increased the
workload on the people who maintain those systems today. Complex workflows
are also difficult to adapt to other languages, because a wiki with
thousands of articles may not need the kind of complexity that comes with
managing a wiki with millions of articles. We've talked about this kind of
structured workflow support at Wikimania, in user research sessions, and on
wikis. It's an important area that needs a lot of discussion, exploration,
and work.

Starting in October, Flow will not be in active development, as we shift
the team's focus to these other priorities. We'll be helping core
contributors reduce the stress of an ever-growing workload, and helping the
next generation of contributors participate in those processes. Further
development on these projects will be driven by the needs expressed by wiki
communities.

Flow will be maintained and supported, and communities that are excited
about Flow discussions will be able to use it. There are places where the
discussion features are working well, with communities that are
enthusiastic about them: on user talk pages, help pages, and forum/village
pump-style discussion spaces. By the end of September, we'll have an opt-in
Beta feature available to communities that want it, allowing users to
enable Flow on their own user talk pages.

I'm sure people will want to know more about these projects, and we're
looking forward to those conversations. We'll be reaching out for lots of
input and feedback over the coming months.

Danny Horn
Collaboration team, PM
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Platonides]

Brad Jorsch (Anomie) wrote:

I wonder if it got lost in the move from Squid to Varnish, or something
along those lines.


That's likely, given that it was enforced by squid.


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Phabricator monthly statistics - 2015-08

[Yongmin Hong]

2015. 9. 1. 오후 7:18에 님이 작성:
>
>
> Hi Community Metrics team,
>
> this is your automatic monthly Phabricator statistics mail.
>
> Number of accounts created in (2015-08): 288
> Number of active users (any activity) in (2015-08): 831
> Number of task authors in (2015-08): 481
> Number of users who have closed tasks in (2015-08): 258
>
> Number of projects which had at least one task moved from one column
>  to another on their workboard in (2015-08): 192
>
> Number of tasks created in (2015-08): 3338
> Number of tasks closed in (2015-08): 2625
>
> Number of open and stalled tasks in total: 25324
>
> Median age in days of open tasks by priority:
> Unbreak now: 15
> Needs Triage: 105
> High: 151
> Normal: 301
> Low: 698
> Needs Volunteer: 16678

I thought this "Needs Volunteer" has been renamed to "Lowest" a while ago?

(snip)

--
revi
https://revi.me
-- Sent from Android --
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Phabricator monthly statistics - 2015-08

[communitymetrics]

Hi Community Metrics team,

this is your automatic monthly Phabricator statistics mail.

Number of accounts created in (2015-08): 288
Number of active users (any activity) in (2015-08): 831
Number of task authors in (2015-08): 481
Number of users who have closed tasks in (2015-08): 258

Number of projects which had at least one task moved from one column
 to another on their workboard in (2015-08): 192

Number of tasks created in (2015-08): 3338
Number of tasks closed in (2015-08): 2625

Number of open and stalled tasks in total: 25324

Median age in days of open tasks by priority:
Unbreak now: 15
Needs Triage: 105
High: 151
Normal: 301
Low: 698
Needs Volunteer: 16678
(How long tasks have been open, not how long they have had that priority)

TODO: Numbers which refer to closed tasks might not be correct, as described in 
T1003.

Yours sincerely,
Fab Rick Aytor

(via community_metrics.sh on iridium at Tue Sep  1 00:00:08 UTC 2015)

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Semantic MediaWiki Conference Fall 2015: Call for Participation

[kghbln]

Dear users, developers and all people interested in semantic wikis,

We are very happy to announce that early bird registration to the 12th 
Semantic MediaWiki Conference is now open!


*Important facts reminder:*

 * Dates: October 28th to October 30th 2015 (Wednesday to Friday)
 * Location: Fabra i Coats, Art Factory. Carrer Sant Adrià 20 (Sant
   Andreu), Barcelona, Catalonia, Spain.
 * Conference page: https://semantic-mediawiki.org/wiki/SMWCon_Fall_2015
 * Participants: Everybody interested in semantic wikis, especially in
   Semantic MediaWiki, e.g. users, developers, consultants, business
   representatives and researchers.

*We welcome new contributions from you:*

 * We encourage contributions about applications and development of
   semantic wikis; for a list of topics, see [1].
 * Please propose regular talks, posters or workshops on the conference
   website. We will do our best to consider your proposal in the
   conference program. An interesting variety of talks has already be
   proposed, see [2].
 * Presentations will generally be video and audio recorded and made
   available for others after the conference.
 * If you've already announced your talk it's now time to expand its
   description.

*News on participation and tutorials:*

 * You can now officially register for the conference [3] and benefit
   from early bird fees until October 5, 2015.
 * The tutorial program has been announced and made available [4].

*Organization:*

 * Amical Wikimedia [5] and Open Semantic Data Association e. V. [6]
   have become the official organisers of SMWCon Fall 2015
 * Thanks to Institut de Cultura - Ajuntament de Barcelona [7] for
   providing free access to the conference location and its infrastructure

If you have questions you can contact Lia Veja and Karsten Hoffmeyer 
(Program Chairs), Alina Mierluș (General Chair) or Toni Hermoso (Local 
Chair) per e-mail (Cc).


We will be happy to see you in Barcelona!

Lia Veja, Karsten Hoffmeyer (Program Board)

[1] 
[2] 
[3] 
[4] 
[5] 
[6] 
[7] 

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Collaboration team reprioritization

[Pine W]

Thanks for the update. Discussion is taking place on Wikimedia-l.

Pine
On Sep 1, 2015 2:27 PM, "Danny Horn"  wrote:

> For a while now, the Collaboration team has been working on Flow, the
> structured discussion system. I want to let you know about some changes in
> that long-term plan.
>
> While initial announcements about Flow said that it would be a universal
> replacement for talk pages, the features that were ultimately built into
> Flow were specifically forum-style group discussion tools. But article and
> project talk pages are used for a number of important and complex processes
> that those tools aren't able to handle, making Flow unsuitable for
> deployment on those kinds of pages.
>
> To better address the needs of our core contributors, we're now focusing
> our strategy on the curation, collaboration, and admin processes that take
> place on a variety of pages. Many of these processes use complex
> workarounds -- templates, categories, transclusions, and lots of
> instructions -- that turn blank wikitext talk pages into structured
> workflows. There are gadgets and user scripts on the larger wikis to help
> with some of these workflows, but these tools aren't standardized or
> universally available.
>
> As these workflows grow in complexity, they become more difficult for the
> next generation of editors to learn and use. This has increased the
> workload on the people who maintain those systems today. Complex workflows
> are also difficult to adapt to other languages, because a wiki with
> thousands of articles may not need the kind of complexity that comes with
> managing a wiki with millions of articles. We've talked about this kind of
> structured workflow support at Wikimania, in user research sessions, and on
> wikis. It's an important area that needs a lot of discussion, exploration,
> and work.
>
> Starting in October, Flow will not be in active development, as we shift
> the team's focus to these other priorities. We'll be helping core
> contributors reduce the stress of an ever-growing workload, and helping the
> next generation of contributors participate in those processes. Further
> development on these projects will be driven by the needs expressed by wiki
> communities.
>
> Flow will be maintained and supported, and communities that are excited
> about Flow discussions will be able to use it. There are places where the
> discussion features are working well, with communities that are
> enthusiastic about them: on user talk pages, help pages, and forum/village
> pump-style discussion spaces. By the end of September, we'll have an opt-in
> Beta feature available to communities that want it, allowing users to
> enable Flow on their own user talk pages.
>
> I'm sure people will want to know more about these projects, and we're
> looking forward to those conversations. We'll be reaching out for lots of
> input and feedback over the coming months.
>
> Danny Horn
> Collaboration team, PM
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] RFC meeting this week

[Rob Lanphier]

Hi all,

I'm going to take on the RFC review meeting announcement duties for Tim.

Discussion topic this week:
"RFC: Master/slave datacenter strategy for MediaWiki":


I've put the meeting details in Phab's event tool here:


Borrowing Tim's boilerplate:
The meeting will be on the IRC channel #wikimedia-office on
chat.freenode.net at the following time:

* UTC: Wednesday 21:00
* US PDT: Wednesday 14:00
* Europe CEST: Wednesday 23:00
* Australia AEST: Thursday 07:00

I'm planning to document this and future meetings here somewhere:
https://www.mediawiki.org/wiki/Architecture_meetings

Still TBD, but if that type of thing interests you, weigh in here:
https://www.mediawiki.org/wiki/Talk:Architecture_meetings

Rob

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Ori Livneh]

On Tue, Sep 1, 2015 at 8:30 AM, Ori Livneh  wrote:

> We appear to be running a banner campaign on the mobile web site, driving
> people to download the mobile app:
>

Just in time!
http://techcrunch.com/2015/09/01/death-to-app-install-interstitials/
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Ricordisamoa]

Il 02/09/2015 07:39, Matthew Flaschen ha scritto:

On 09/01/2015 11:30 AM, Ori Livneh wrote:
We appear to be running a banner campaign on the mobile web site, 
driving

people to download the mobile app:

https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2
https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1

Campaign definition:
https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice=noticeDetail=Android_app 



This isn't cool. This isn't us. We don't drive people from an open 
platform

to a closed one.


I don't necessarily think it's a great idea to push people from web to 
apps either, especially when we also have people working on mobile web.


I also do most of my mobile Wikipedia browsing on mobile web.

That said, I think that assessment is overly critical.

* The Android mobile app is fully free and open source (obvious, since 
all of our stuff is, but worth re-iterating).


* They've done a great job on the app.  In particular, they've 
implemented features that are easier on app (or only feasible there), 
like a user-friendly saved pages list and a nice UI in general.


* I don't know this for sure, but I would guess the app works on 
fully-FOSS versions of Android (e.g. Replicant), since an updated 
version is in the fully-free app store 
(https://f-droid.org/wiki/page/org.wikipedia).  If it doesn't work on 
Replicant (or some similar fully-FOSS Android), that does seem like 
something important to address.


* No one is going to install proprietary software as a result of this 
ad.  It only shows to people who are *already* running Android and 
asks them to install free and open source software.


It's no different then recommending to a Windows user that they 
install Inkscape because it's a great piece of free and open source 
software.


Finally, this is indeed only configured for Finland.


Linus' birthplace...



Matt Flaschen

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l



___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Legoktm]

On 09/01/2015 10:37 AM, Brion Vibber wrote:
> I'm not 100% convinced that the UA requirement is helpful, for two reasons:

For those of us who looked for the initial rationale on the UA
requirement, the announcement and resulting discussion is at [1].

[1] http://www.gossamer-threads.com/lists/wiki/wikitech/189275

-- Legoktm

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Matthew Flaschen]

On 09/01/2015 11:30 AM, Ori Livneh wrote:

We appear to be running a banner campaign on the mobile web site, driving
people to download the mobile app:

https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2
https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1

Campaign definition:
https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice=noticeDetail=Android_app

This isn't cool. This isn't us. We don't drive people from an open platform
to a closed one.


I don't necessarily think it's a great idea to push people from web to 
apps either, especially when we also have people working on mobile web.


I also do most of my mobile Wikipedia browsing on mobile web.

That said, I think that assessment is overly critical.

* The Android mobile app is fully free and open source (obvious, since 
all of our stuff is, but worth re-iterating).


* They've done a great job on the app.  In particular, they've 
implemented features that are easier on app (or only feasible there), 
like a user-friendly saved pages list and a nice UI in general.


* I don't know this for sure, but I would guess the app works on 
fully-FOSS versions of Android (e.g. Replicant), since an updated 
version is in the fully-free app store 
(https://f-droid.org/wiki/page/org.wikipedia).  If it doesn't work on 
Replicant (or some similar fully-FOSS Android), that does seem like 
something important to address.


* No one is going to install proprietary software as a result of this 
ad.  It only shows to people who are *already* running Android and asks 
them to install free and open source software.


It's no different then recommending to a Windows user that they install 
Inkscape because it's a great piece of free and open source software.


Finally, this is indeed only configured for Finland.

Matt Flaschen

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Gergo Tisza]

On Tue, Sep 1, 2015 at 10:09 PM, Ori Livneh  wrote:

> Just in time!
> http://techcrunch.com/2015/09/01/death-to-app-install-interstitials/


Interstitials are full-page ads where you have to click a link to get to
the actual content. These are normal banners.
More importantly, as you can see in the Phabricator task, they are an
experiment to measure if it is possible to make more people use the app.
Experiments are good. For one thing, they can turn out negative, in which
case we will have been spared a  philosophical debate about openness.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What does "Director of Architecture" mean anyway?

[Pine W]

Welcome back. It's good to hear of your recovery.

Pine
On Sep 1, 2015 8:29 PM, "Rob Lanphier"  wrote:

> Hi everyone,
>
> It's been a while since I've been involved here, and I've got a pretty good
> excuse.  I'm going to avoid going into too much detail, but in short, I was
> involved in a pretty nasty traffic collision, and have only recently
> returned to work.  It turned out the way that a lot of bike versus car
> collisions go, which is to say, I lost  :-(
>
> The good news (for me at least), I've largely recovered.  The surgeons here
> did a remarkable job putting me back together again.  I returned to
> part-time work a couple of weeks ago, and I'm getting back up to speed on
> things.  I still have a number of personal obligations that keep me from
> working full-time, but I'll keep my user page on mediawiki.org
>  updated with the latest
> (or
> point you to where I plan to keep that info).
>
> Though we hadn't announced it in April, we had *just* talked about making
> my title "Director of Architecture" in the big reorg we had here at WMF.
> We hadn't worked out exactly what that meant when I was in the wreck, but I
> had a vague idea which I intend to flesh out.  In the short term, I want to
> facilitate the activity of the Architecture Committee
> .  I've started
> chairing the private meetings of that group, and I'm taking on a bunch of
> the administrivia.  I'll make things clearer as they become clearer in my
> head  :-)  Well, maybe after that, and after I get a chance to write down
> what I'm thinking.
> I'm really happy to be back involved!  \o/  If you have any questions, or
> you simply want to wave back, feel free to drop a comment on my MediaWiki
> talk page:
> https://www.mediawiki.org/wiki/User_talk:RobLa-WMF
>
> Rob
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Discovery Department A/B testing an alternative to prefix search next week

[Dan Garry]

Hi everyone,

*tl;dr: Discovery Department to run A/B test
 comparing new search suggester
to prefix search, to see if it can reduce zero results rate.*

As I'm sure you're all aware, the search box at the top right of every page
on desktop uses prefix search to generate its results. The main reason for
this is that prefix search is incredibly fast and performant; that search
box sees a lot of traffic, and it's important to keep it scalable.

However, we know that there are numerous problems with prefix search.
Prefix searches are prone to give you no results; if you make even a slight
typo, then you won't get the result you want. And thus a complex system of
manually curated redirects were born to try to alleviate this navigation
issue. Wouldn't it be nice if we could work towards a solution that doesn't
require the manual curation of redirects, thus freeing up Wikimedians to do
other more meaningful tasks? And make search a bit better in the process,
too? That's a long term goal of mine... emphasis on the long. ;-)

The Q1 2015-17 (Jul - Aug 2015) goal of the Search Team in the Discovery
Department is to reduce the zero results rate
.
Amongst other things, we've been working to build an alternative to prefix
search . Documentation on the
API is pretty light right now because we're scrambling to get it up and
running (but there's a task for that!
).

An initial version of the suggestion API is now in production on enwiki and
dewiki [1], but is currently not being used for anything. Our initial tests
 of the API show that it's
incredibly promising for reducing the zero results rate. But we need more
data!

We're planning on running an A/B test on whether this API is better at
reducing zero results. We're targeting beginning on Tuesday 8th September,
for two weeks. This is documented in T111078
.

A very important note here is that we currently have no way of
quantitatively measuring result relevance (although we're working on it
), so this test will be highly
limited in scope, only measuring the zero results rate. Given the limits of
this, even seeing massive success in this test is not enough to deploy this
API as a full replacement of prefix search; we'd need additional data. But,
that's not stopping us from gathering initial data from this test.

As always, if you have any questions, let me know.

Thanks,
Dan

[1]: The API is actually live on all wikis, but we only built the search
indices for enwiki and dewiki since they're our biggest content wikis and
this is an early test. Attempting to use the API on any other wiki will get
you a cirrus backend error.

-- 
Dan Garry
Lead Product Manager, Discovery
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Chad]

On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes  wrote:

> Is the
> blocking of requests absent a user agent simply happening at a
> 'higher' stage (in mediawiki itself?) and so not registering with the
> varnishes,


No, it's not done at the application level.


> or is sending an /empty/ header simply A-OK?
>
>
Shouldn't be, unless the policy changed...

-Chad
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Oliver Keyes]

On 1 September 2015 at 12:41, Chad  wrote:
> On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes  wrote:
>
>> Is the
>> blocking of requests absent a user agent simply happening at a
>> 'higher' stage (in mediawiki itself?) and so not registering with the
>> varnishes,
>
>
> No, it's not done at the application level.
>
>
>> or is sending an /empty/ header simply A-OK?
>>
>>
> Shouldn't be, unless the policy changed...

Well I'm looking at millions of requests from
API-users-who-I-am-not-big-fans-of[0] with a blank UA sooo..

[0] actual term far more expletive-laden than this

>
> -Chad
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[Oliver Keyes]

On 1 September 2015 at 12:42, John  wrote:
> Could they be sending a non-standard header of "-"

Perfectly possible although also impossible to detect :(

>
> On Tuesday, September 1, 2015, Chad  wrote:
>
>> On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes > > wrote:
>>
>> > Is the
>> > blocking of requests absent a user agent simply happening at a
>> > 'higher' stage (in mediawiki itself?) and so not registering with the
>> > varnishes,
>>
>>
>> No, it's not done at the application level.
>>
>>
>> > or is sending an /empty/ header simply A-OK?
>> >
>> >
>> Shouldn't be, unless the policy changed...
>>
>> -Chad
>> ___
>> Wikitech-l mailing list
>> Wikitech-l@lists.wikimedia.org 
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] What happened to our user agent requirements?

[Oliver Keyes]

According to https://meta.wikimedia.org/wiki/User-Agent_policy and the
associated mailing list threads, user agent headers are now required
(and have been for some time) but on the request log side, we see a
lot of requests with the user agent "-" - IOW, an empty field. Is the
blocking of requests absent a user agent simply happening at a
'higher' stage (in mediawiki itself?) and so not registering with the
varnishes, or is sending an /empty/ header simply A-OK?

-- 
Oliver Keyes
Count Logula
Wikimedia Foundation

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] What happened to our user agent requirements?

[John]

Could they be sending a non-standard header of "-"

On Tuesday, September 1, 2015, Chad  wrote:

> On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes  > wrote:
>
> > Is the
> > blocking of requests absent a user agent simply happening at a
> > 'higher' stage (in mediawiki itself?) and so not registering with the
> > varnishes,
>
>
> No, it's not done at the application level.
>
>
> > or is sending an /empty/ header simply A-OK?
> >
> >
> Shouldn't be, unless the policy changed...
>
> -Chad
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org 
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] "Try the free Wikipedia app" banners

[Gergo Tisza]

Probably T103896  but in that
case something was misconfigured (those should only show in Finland).

On Tue, Sep 1, 2015 at 8:30 AM, Ori Livneh  wrote:

> We appear to be running a banner campaign on the mobile web site, driving
> people to download the mobile app:
>
> https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2
> https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1
>
> Campaign definition:
>
> https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice=noticeDetail=Android_app
>
> This isn't cool. This isn't us. We don't drive people from an open platform
> to a closed one.
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] New RFC: better JS minification

[Jérémie Roquet]

Hi everyone,

I was running some experiments with UglifyJS lately and so I stumbled
upon this RFC which shows that if I've not discovered anything new, we
can still get ~15% smaller JS files.

2014-03-19 22:16 GMT+01:00 Max Semenik :
> Hi, I'd like to present a new RFC for your consideration:
> https://www.mediawiki.org/wiki/Requests_for_comment/Minifier
>
> It is about how we can shave 10-15% off the size if JavaScript
> delivered to users.

Now, it's also about supporting newer versions of ECMAScript (if we
don't target full backward compatibility, of course), because it seems
to me that unlike UglifyJS, JSMin+ has not been updated for a while
and could choke on some valid syntax with error messages that are not
exactly… explicit for casual JS developers (I've experimented that a
few weeks ago with local scripts making use of reserved words as
identifiers — not exactly valid code, indeed, but code that had worked
for years in most browsers).

Has the RFC been abandonned because of lack of interest?

Best regards,

-- 
Jérémie

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] "Try the free Wikipedia app" banners

[Ori Livneh]

We appear to be running a banner campaign on the mobile web site, driving
people to download the mobile app:

https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_2
https://en.m.wikipedia.org/wiki/?banner=Aug2015_app_banner_1

Campaign definition:
https://meta.wikimedia.org/w/index.php?title=Special:CentralNotice=noticeDetail=Android_app

This isn't cool. This isn't us. We don't drive people from an open platform
to a closed one.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] New RFC: better JS minification

[Ori Livneh]

On Tue, Sep 1, 2015 at 8:43 AM, Jérémie Roquet 
wrote:

> Has the RFC been abandonned because of lack of interest?
>

Speaking for myself: at the time the RFC was written, I was skeptical that
the benefits would be worth incurring a dependency on an external service
(or the requirement of shelling out). But since then we have had a lot of
discussions (in the RFC meetings) on how to do micro-services correctly,
and some good abstractions have come out of that. So +1 for reviving it.
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l