Re: [OSM-talk] Overpass XAPI URL Status

[Roland Olbricht]

Hi Bryce,


I help support a large company that uses OSM data, pulled via a specific
query at  ://www.overpass-api.de/api/xapi
. This query runs about once a month.


thank you for asking back.

First about the status: when I last checked some days ago then there 
have been still almost a million requests for /xapi . This is in 
contrast to 500.000 requests for all other API endpoints combined.
Yesterday figures are 1067494 for /xapi and 937498 for all other API 
call combined.


The bigger surprise came when I have taken a sample of 100 requests from 
xapi calls. I turned out that while they are from various apps still all 
100 of them were less desirable bulk load. At that point I decided to 
wait until somebody comes up with a real request to get the xapi call 
back. This has happened right now.


For that reason I have configured the service now as follows:
- xapi works again
- xapi?map now returns 400 Bad request as I have turned off that in the 
syntax recognition


I think that a longtime solution will be to take the referrer and user 
agent into account when scheduling requests. But this has not been 
implemented yet.


I would like to conclude with some remarks to hedge rumors on this thread:

The OSMF is not in any way involved in operations. Paul's mail describes 
the precise situation. Please note that the whole purpose of third party 
services (like Overpass API) for secondary functionality is to keep that 
burden off the OSMF.


Nothing is held confidentially on purpose. There are two limitations 
that might give this impression:
- I'm on vacation right now and until SotM and relatively unwillig to 
answer questions immediately.

- The log files may contain personal data and thus I won't hand them out

Best regards,

Roland


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Overpass XAPI URL Status

[Michał Brzozowski]

On Wed, Sep 14, 2016 at 8:04 PM, mmd  wrote:
> xapi endpoint is the only service that is blocked right now. That's due
> to one particular mobile app firing off excessive amounts of queries.
> Details are also known to the OSMF board.

Is it, shall I say, classified information?

If the developer can't deal with it for a few weeks (!), I'd just
escalate it at the respective app store, which I presume would happily
take it down if you tell them they perform a DDoS.

Michał

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Overpass XAPI URL Status

[Paul Norman]

On 9/14/2016 11:04 AM, mmd wrote:

Am 14.09.2016 um 09:30 schrieb Bryce Nesbitt:

>I help support a large company that uses OSM data, pulled via a specific
>query at  ://www.overpass-api.de/api/xapi
>. This query runs about once a month.

xapi endpoint is the only service that is blocked right now. That's due
to one particular mobile app firing off excessive amounts of queries.
Details are also known to the OSMF board.


There's been questions about this and I want to clarify that

- Overpass runs on FOSSGIS Germany hardware, not OSMF hardware

- The OSMF board is not involved and has not been sent any details, nor 
do we need to be involved


Even if it were on OSMF hardware, scrapers and badly behaving apps are 
an sysadmin/ops issue and not one the board would normally need to get 
involved with.


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Overpass XAPI URL Status

[mmd]

Hi!


Am 14.09.2016 um 09:30 schrieb Bryce Nesbitt:
> I help support a large company that uses OSM data, pulled via a specific
> query at  ://www.overpass-api.de/api/xapi
> . This query runs about once a month.

xapi endpoint is the only service that is blocked right now. That's due
to one particular mobile app firing off excessive amounts of queries.
Details are also known to the OSMF board.

You basically have two options at this time:

a) Use xapi_meta instead of xapi endpoint
b) Migrate your xapi query to Overpass XML / QL, see [1]

> 
> It recently stopped working apparently because of a hack and because of
> bogus querys:
> http://wiki.openstreetmap.org/wiki/Overpass_API/status

There was no hack in the first place. Reinstalling the server was rather
a precautionary measure.

> The comment is "I'll restore /api/xapi when this problematic access
> pattern has ceased."

The situation was last described in this ticket:

https://github.com/drolbr/Overpass-API/issues/313

I assume Roland will restore access to the xapi endpoint once that
particular app ceases to send significant amounts of queries.

-- 


[1] http://wiki.openstreetmap.org/wiki/Overpass_API/XAPI_Compatibility_Layer


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk