Re: [Wikitech-l] What happened to our user agent requirements?

I'm not 100% convinced that the UA requirement is helpful, for two reasons: 1) Lots of requests will have default like "PHP" or "Python/urllib" or whatever from the tool they used to build their bot. These aren't helpful either as they contain no of how to get in touch. 2) It's trivial to work

Re: [Wikitech-l] What happened to our user agent requirements?

Specifically, the hypothesis that people are sending "-"? On 1 September 2015 at 12:58, Tomasz Finc wrote: > Let's get a task in phab for this so that we can triage next steps. > I'm curious about this as well. > > --tomasz > > On Tue, Sep 1, 2015 at 9:46 AM, Oliver Keyes

Re: [Wikitech-l] What happened to our user agent requirements?

Tracking the overall issue On Tue, Sep 1, 2015 at 9:59 AM, Oliver Keyes wrote: > Specifically, the hypothesis that people are sending "-"? > > On 1 September 2015 at 12:58, Tomasz Finc wrote: >> Let's get a task in phab for this so that we can triage

Re: [Wikitech-l] What happened to our user agent requirements?

Let's get a task in phab for this so that we can triage next steps. I'm curious about this as well. --tomasz On Tue, Sep 1, 2015 at 9:46 AM, Oliver Keyes wrote: > On 1 September 2015 at 12:42, John wrote: >> Could they be sending a non-standard

Re: [Wikitech-l] What happened to our user agent requirements?

I agree with rate-limiting those without some sort of ID (login or API key). As Oliver said, big (ab)users can massively skew our stats, often by themselves. But hordes of upper middle volume bots (way too high for a human, nowhere near the max for a superstar bot) can have a large cumulative

Re: [Wikitech-l] What happened to our user agent requirements?

Awesome; thanks for the analysis, Krinkle. Do we want to change this behaviour? From my point of view the answer is 'yes, not setting any kind of user agent is a violation of our API etiquette and we should be taking steps to alert people that it is' but if other people have different

Re: [Wikitech-l] What happened to our user agent requirements?

If people aren't capable of following UA guidelines I doubt they're going to follow voluntary login. For what it's worth I absolutely support both rate-limiting and login to get around this. In fact, I would argue that from an analytics point of view rate limiting is probably the most

Re: [Wikitech-l] What happened to our user agent requirements?

On Tue, Sep 1, 2015 at 1:18 PM, Krinkle wrote: > In the past (2012?) these were definitely being blocked. (Ran into it from > time to time on Toolserver) > It seems php file_get_contents('http://...api..' ) is > also working fine now, > without having to

Re: [Wikitech-l] What happened to our user agent requirements?

I've confirmed just now that whatever requirement there was, it doesn't seem to be in effect. Both omitting the header entirely, sending it with empty string, and sending with "-"; – all three result in a response from the MediaWiki API. $ curl -A '' --include -v

Re: [Wikitech-l] What happened to our user agent requirements?

On Tue, Sep 1, 2015 at 10:42 PM, Platonides wrote: > Brad Jorsch (Anomie) wrote: >> I wonder if it got lost in the move from Squid to Varnish, or something >> along those lines. > That's likely, given that it was enforced by squid. We could easily add it back in Varnish,

Re: [Wikitech-l] What happened to our user agent requirements?

On Tue, Sep 1, 2015 at 4:54 PM, Brandon Black wrote: > I really do like the idea of moving towards smarter ratelimiting of > APIs by default, though (and have brought this up in several contexts > recently, but I'm not really aware of whatever past work we've done in > that

Re: [Wikitech-l] What happened to our user agent requirements?

On Tue, Sep 1, 2015 at 5:54 PM, Gergo Tisza wrote: > > > Rate limiting / UA policy enforcement has to be done in Varnish, since API > responses can be cached there and so the requests don't necessarily reach > higher layers (and we wouldn't want to vary on user agent).

Re: [Wikitech-l] What happened to our user agent requirements?

We recently revisited rate limiting in https://phabricator.wikimedia.org/T107934, but came to similar conclusions as reached in this thread: - Limits for weak identifiers like IPs or user agents would (at least initially) need to be high enough to render the limiting borderline useless

Re: [Wikitech-l] What happened to our user agent requirements?

On Wed, Sep 2, 2015 at 1:21 AM, Gabriel Wicke wrote: > On Tue, Sep 1, 2015 at 5:54 PM, Gergo Tisza wrote: >> >> >> Rate limiting / UA policy enforcement has to be done in Varnish, since API >> responses can be cached there and so the requests don't

Re: [Wikitech-l] What happened to our user agent requirements?

On Tue, Sep 01, 2015 at 12:42:35PM -0400, John wrote: > Could they be sending a non-standard header of "-" They could. But if a request comes in without a User-Agent header, the logging pipeline silently translates it into "-". Have fun, Christian P.S.: The relevant configuration (for

Re: [Wikitech-l] What happened to our user agent requirements?

Brad Jorsch (Anomie) wrote: I wonder if it got lost in the move from Squid to Varnish, or something along those lines. That's likely, given that it was enforced by squid. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org

Re: [Wikitech-l] What happened to our user agent requirements?

On 09/01/2015 10:37 AM, Brion Vibber wrote: > I'm not 100% convinced that the UA requirement is helpful, for two reasons: For those of us who looked for the initial rationale on the UA requirement, the announcement and resulting discussion is at [1]. [1]

Re: [Wikitech-l] What happened to our user agent requirements?

On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes wrote: > Is the > blocking of requests absent a user agent simply happening at a > 'higher' stage (in mediawiki itself?) and so not registering with the > varnishes, No, it's not done at the application level. > or is sending

Re: [Wikitech-l] What happened to our user agent requirements?

On 1 September 2015 at 12:41, Chad wrote: > On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes wrote: > >> Is the >> blocking of requests absent a user agent simply happening at a >> 'higher' stage (in mediawiki itself?) and so not registering with the >>

Re: [Wikitech-l] What happened to our user agent requirements?

On 1 September 2015 at 12:42, John wrote: > Could they be sending a non-standard header of "-" Perfectly possible although also impossible to detect :( > > On Tuesday, September 1, 2015, Chad wrote: > >> On Tue, Sep 1, 2015 at 9:24 AM Oliver

[Wikitech-l] What happened to our user agent requirements?

According to https://meta.wikimedia.org/wiki/User-Agent_policy and the associated mailing list threads, user agent headers are now required (and have been for some time) but on the request log side, we see a lot of requests with the user agent "-" - IOW, an empty field. Is the blocking of requests

Re: [Wikitech-l] What happened to our user agent requirements?

Could they be sending a non-standard header of "-" On Tuesday, September 1, 2015, Chad wrote: > On Tue, Sep 1, 2015 at 9:24 AM Oliver Keyes > wrote: > > > Is the > > blocking of requests absent a user agent simply happening at a >