Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[mmd]

Hi,

Am 18.09.2016 um 14:41 schrieb Michael Larsen:
> Hi,
> 
> Last time I tried consuming augmented diffs on a minutely basis, I hit the 
> load-limitations which meant that I could not consume augmented diffs for 
> some 
> time afterwards, i.e. this will lead to black holes in your history.

the value returned by augmented_diff_status corresponds to current
database timestamp (as number of minutes since the license change). It
does not need to increase one-by-one(!), e.g. the database may process
several minutely diffs in one go, due to some backlog. If you always
download the number returned by augmented_diff_stauts, you will indeed
get some holes!

That situation can be avoided, if you keep your own internal sequence
id, and fetch augmented diffs up to and including to the value returned
by augmented_diff_status.

If augmented_diff_status does not return any value (due to overload),
just wait some time and try again. The same applies to downloading
augmented_diffs: you may get HTTP 429 or HTTP 504 in case of overload,
or if you exceed your quota (see /api/status for details). In that case,
don't increase your internal sequence id yet, but try downloading the
same augmented diff again.

> 
> Also, using timestamp start/end to fetch diffs for a given timestamp (like 
> avachi) is problematic with some changesets that stay open for > 1 hours 
> (this 
> happens quite ofter). The live service running on osm.expandable.dk use the 
> API as described previously to get the augmented diff for a changeset. If 
> there where a better way I'm all ears!
> 

I hope the situation will improve a bit once the database has moved to
version 0.7.53 and a different compression. If you're at SotM next week,
you could maybe ask Roland for a current status.

-- 



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Michael Larsen]

Hi,

Last time I tried consuming augmented diffs on a minutely basis, I hit the 
load-limitations which meant that I could not consume augmented diffs for some 
time afterwards, i.e. this will lead to black holes in your history.

Also, using timestamp start/end to fetch diffs for a given timestamp (like 
avachi) is problematic with some changesets that stay open for > 1 hours (this 
happens quite ofter). The live service running on osm.expandable.dk use the 
API as described previously to get the augmented diff for a changeset. If 
there where a better way I'm all ears!

Regards,
MichaelVL

On søndag den 18. september 2016 01.59.57 CEST Stefan Keller wrote:
> Thanks mmd,
> 
> As said, I'd like to fetch the latest available augmented diff and
> keep uptodate.
> So, to understand it correctly:
> Every minute I first call the
> http://overpass-api.de/api/augmented_diff_status which returns an id,
> e.g. 2111988,
> then I fetch the augmented diff file
> http://overpass-api.de/api/augmented_diff?id=2111988
> Just want to be sure that this is the correct way to use the API.
> 
> :Stefan
> 
> 2016-09-17 16:48 GMT+02:00 mmd :
> > Am 15.09.2016 um 22:53 schrieb Stefan Keller:
> >> I'm setting up a Kafka publish-subscribe messaging system delivering
> >> minutely diffs.
> >> 
> >> AFAIK augmented diffs are rather an experimental feature and I'd like
> >> to avoid the latency time and blackouts of overpass which runs in same
> >> server. So I'm concentrating on the main OSM API.
> >> 
> >> Now, osmChange XML like
> >> http://www.osm.org/api/0.6/changeset/42143238/download obviously does
> >> not include all info (e.g. tags) from ref nodes/ways/relations.
> >> 
> >> I'm not looking for specific info but I'd like to get at least all
> >> data about nodes/ways/relations which are created/modified/deleted in
> >> a changeset.
> > 
> > This sounds a lot like what achavi is doing right now.
> > 
> > Note that with the introduction of the famous 'attic' concept, augmented
> > diffs are nowadays simply translated into a plain Overpass QL query and
> > calculated on the fly.
> > 
> > This also means that you don't have to stick to a minute interval or a
> > global scope as before: you're free to provide time intervals matching a
> > changeset (like achavi does), or restrict the adiff to a certain
> > bounding box or even some nodes/ways/relations with certain tags only.
> > 
> > There's no XSD around, the wiki page should answer most questions,
> > though. In general, the Overpass Dev list would be a good place for more
> > details and discussions. [1]
> > 
> > [1] http://listes.openstreetmap.fr/wws/info/overpass
> > 
> > 
> > 
> > 
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/dev
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Stefan Keller]

Thanks mmd,

As said, I'd like to fetch the latest available augmented diff and
keep uptodate.
So, to understand it correctly:
Every minute I first call the
http://overpass-api.de/api/augmented_diff_status which returns an id,
e.g. 2111988,
then I fetch the augmented diff file
http://overpass-api.de/api/augmented_diff?id=2111988
Just want to be sure that this is the correct way to use the API.

:Stefan

2016-09-17 16:48 GMT+02:00 mmd :
> Am 15.09.2016 um 22:53 schrieb Stefan Keller:
>> I'm setting up a Kafka publish-subscribe messaging system delivering
>> minutely diffs.
>>
>> AFAIK augmented diffs are rather an experimental feature and I'd like
>> to avoid the latency time and blackouts of overpass which runs in same
>> server. So I'm concentrating on the main OSM API.
>>
>> Now, osmChange XML like
>> http://www.osm.org/api/0.6/changeset/42143238/download obviously does
>> not include all info (e.g. tags) from ref nodes/ways/relations.
>>
>> I'm not looking for specific info but I'd like to get at least all
>> data about nodes/ways/relations which are created/modified/deleted in
>> a changeset.
>>
>
> This sounds a lot like what achavi is doing right now.
>
> Note that with the introduction of the famous 'attic' concept, augmented
> diffs are nowadays simply translated into a plain Overpass QL query and
> calculated on the fly.
>
> This also means that you don't have to stick to a minute interval or a
> global scope as before: you're free to provide time intervals matching a
> changeset (like achavi does), or restrict the adiff to a certain
> bounding box or even some nodes/ways/relations with certain tags only.
>
> There's no XSD around, the wiki page should answer most questions,
> though. In general, the Overpass Dev list would be a good place for more
> details and discussions. [1]
>
> [1] http://listes.openstreetmap.fr/wws/info/overpass
>
>
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Pierre Béland]

I have already opened a Overpass API Github issue relatd to a restriction with 
the Overpass Api while extracting the Attic data. If many versions of an object 
are available in the time span you specifiy,  the Overpass API will only return 
the first and the last versions of the object.  Then, you wont obtain all 
content included in the minutely diffs.
 
Pierre 


  De : mmd <mmd@gmail.com>
 À : dev@openstreetmap.org 
 Envoyé le : samedi 17 Septembre 2016 10h48
 Objet : Re: [OSM-dev] OSM API lookups to complement minutely diffs?
   
Am 15.09.2016 um 22:53 schrieb Stefan Keller:
> I'm setting up a Kafka publish-subscribe messaging system delivering
> minutely diffs.
> 
> AFAIK augmented diffs are rather an experimental feature and I'd like
> to avoid the latency time and blackouts of overpass which runs in same
> server. So I'm concentrating on the main OSM API.
> 
> Now, osmChange XML like
> http://www.osm.org/api/0.6/changeset/42143238/download obviously does
> not include all info (e.g. tags) from ref nodes/ways/relations.
> 
> I'm not looking for specific info but I'd like to get at least all
> data about nodes/ways/relations which are created/modified/deleted in
> a changeset.
> 

This sounds a lot like what achavi is doing right now.

Note that with the introduction of the famous 'attic' concept, augmented
diffs are nowadays simply translated into a plain Overpass QL query and
calculated on the fly.

This also means that you don't have to stick to a minute interval or a
global scope as before: you're free to provide time intervals matching a
changeset (like achavi does), or restrict the adiff to a certain
bounding box or even some nodes/ways/relations with certain tags only.

There's no XSD around, the wiki page should answer most questions,
though. In general, the Overpass Dev list would be a good place for more
details and discussions. [1]

[1] http://listes.openstreetmap.fr/wws/info/overpass




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev


   ___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[mmd]

Am 15.09.2016 um 22:53 schrieb Stefan Keller:
> I'm setting up a Kafka publish-subscribe messaging system delivering
> minutely diffs.
> 
> AFAIK augmented diffs are rather an experimental feature and I'd like
> to avoid the latency time and blackouts of overpass which runs in same
> server. So I'm concentrating on the main OSM API.
> 
> Now, osmChange XML like
> http://www.osm.org/api/0.6/changeset/42143238/download obviously does
> not include all info (e.g. tags) from ref nodes/ways/relations.
> 
> I'm not looking for specific info but I'd like to get at least all
> data about nodes/ways/relations which are created/modified/deleted in
> a changeset.
> 

This sounds a lot like what achavi is doing right now.

Note that with the introduction of the famous 'attic' concept, augmented
diffs are nowadays simply translated into a plain Overpass QL query and
calculated on the fly.

This also means that you don't have to stick to a minute interval or a
global scope as before: you're free to provide time intervals matching a
changeset (like achavi does), or restrict the adiff to a certain
bounding box or even some nodes/ways/relations with certain tags only.

There's no XSD around, the wiki page should answer most questions,
though. In general, the Overpass Dev list would be a good place for more
details and discussions. [1]

[1] http://listes.openstreetmap.fr/wws/info/overpass




___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Stefan Keller]

2016-09-16 5:59 GMT+02:00 Paul Norman :
> No, particularly if your code takes off and multiple people start running
> it. I would guess that it would hit rate limits and be automatically
> blocked.

Got that - though the pub-sub messaging system in fact could cache the
data thus taking over the load from subscribers.
I'll try augmented diffs.

:Stefan

2016-09-16 5:59 GMT+02:00 Paul Norman :
> On 9/15/2016 1:53 PM, Stefan Keller wrote:
>>
>> Is it OK to do API lookups like this
>> https://www.osm.org/api/0.6/nodes?nodes=59906080,4400821613  even for
>> minutely diffs? Any alternatives?
>
>
> No, particularly if your code takes off and multiple people start running
> it. I would guess that it would hit rate limits and be automatically
> blocked.
>
> If all you need is node positions this can be done fairly efficiently for
> the entire planet with something like osm2pgsql flat-nodes. If you need full
> information this takes more space. But you should consider if you're just
> rebuilding augmented diffs.
>
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Stefan Keller]

2016-09-16 1:30 GMT+02:00 Frederik Ramm :
> This reads to me like:
>
> "There is a third-party service that solves the problem I have, but
> because it is experimental, I would like to build my own experimental
> third-party service instead" ;)

You convinced me to use augmented diffs with your irresistible
argumentation :-).
Although the very nature of experiments (including mine) is such that
they are possibly transient.

Is there any XML Schema (xsd) of the augmented diffs - or is the code
[2] the documentation [1] ;-) ?

:Stefan

[1] http://wiki.openstreetmap.org/wiki/Overpass_API/Augmented_Diffs
[2] https://github.com/drolbr/Overpass-API/



2016-09-16 1:30 GMT+02:00 Frederik Ramm :
> Hi,
>
> On 09/15/2016 10:53 PM, Stefan Keller wrote:
>> AFAIK augmented diffs are rather an experimental feature and I'd like
>> to avoid the latency time and blackouts of overpass which runs in same
>> server. So I'm concentrating on the main OSM API.
>
> This reads to me like:
>
> "There is a third-party service that solves the problem I have, but
> because it is experimental, I would like to build my own experimental
> third-party service instead" ;)
>
> The augmented diffs were made to solve exactly the problem you have. The
> underlying idea that Overpass implements - load updates from OSM into
> its own full history database and generate augmented diffs using that
> database - is by far the best solution in terms of load caused on the
> central servers. It also scales nicely; if a thousand people decide to
> run Overpass instances to produce their own augmented diffs then the API
> won't suffer.
>
> If you cannot work with overpass but must implement your own version of
> that, then you should at least copy the principle. The central OSM API
> is meant to support editing activity and doesn't scale to support use
> cases like yours (especially if others come after you and decide they
> need their own system because yours is experimental ;)
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
>
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Paul Norman]

On 9/15/2016 1:53 PM, Stefan Keller wrote:

Is it OK to do API lookups like this
https://www.osm.org/api/0.6/nodes?nodes=59906080,4400821613  even for
minutely diffs? Any alternatives?


No, particularly if your code takes off and multiple people start 
running it. I would guess that it would hit rate limits and be 
automatically blocked.


If all you need is node positions this can be done fairly efficiently 
for the entire planet with something like osm2pgsql flat-nodes. If you 
need full information this takes more space. But you should consider if 
you're just rebuilding augmented diffs.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Frederik Ramm]

Hi,

On 09/15/2016 10:53 PM, Stefan Keller wrote:
> AFAIK augmented diffs are rather an experimental feature and I'd like
> to avoid the latency time and blackouts of overpass which runs in same
> server. So I'm concentrating on the main OSM API.

This reads to me like:

"There is a third-party service that solves the problem I have, but
because it is experimental, I would like to build my own experimental
third-party service instead" ;)

The augmented diffs were made to solve exactly the problem you have. The
underlying idea that Overpass implements - load updates from OSM into
its own full history database and generate augmented diffs using that
database - is by far the best solution in terms of load caused on the
central servers. It also scales nicely; if a thousand people decide to
run Overpass instances to produce their own augmented diffs then the API
won't suffer.

If you cannot work with overpass but must implement your own version of
that, then you should at least copy the principle. The central OSM API
is meant to support editing activity and doesn't scale to support use
cases like yours (especially if others come after you and decide they
need their own system because yours is experimental ;)

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Ian Dees]

On Thu, Sep 15, 2016 at 4:53 PM, Stefan Keller  wrote:

> I'm setting up a Kafka publish-subscribe messaging system delivering
> minutely diffs.
>
> AFAIK augmented diffs are rather an experimental feature and I'd like
> to avoid the latency time and blackouts of overpass which runs in same
> server. So I'm concentrating on the main OSM API.
>

I've also experimented with this sort of thing and ran into similar hiccups
as you.

Also, if you want to further reduce latency don't forget about the
streaming replication service:
http://wiki.openstreetmap.org/wiki/Osmosis/Replication#Streaming_Replication_Wire_Protocol

Last I checked it wasn't running, and I think TomH would be hard pressed to
get it to work again, but if you end up building something that would be
useful for a wide array of people it might be worth revisiting that.


> Now, osmChange XML like
> http://www.osm.org/api/0.6/changeset/42143238/download obviously does
> not include all info (e.g. tags) from ref nodes/ways/relations.
>
> I'm not looking for specific info but I'd like to get at least all
> data about nodes/ways/relations which are created/modified/deleted in
> a changeset.
>
> Is it OK to do API lookups like this
> https://www.osm.org/api/0.6/nodes?nodes=59906080,4400821613 even for
> minutely diffs? Any alternatives?


This is where I stopped working on this idea. I ended up making *lots* of
requests to the API server to fill in missing data and I didn't think it
would be very nice to my fellow mappers if I scaled that up and made it run
24/7.

At this point I started experimenting with ways of maintaining a full
history database to help make these sorts of lookups more performant, but
then ran out of time to work on this project.

I think a more interesting path forward for this kind of problem is to
resurrect the aforementioned streaming replication protocol and modify it
to include changeset and "augmented diff"-style information. My intuition
says that making smart SQL queries against a read replica database rather
than adding API load might be more efficient and useful for lots of
consumers. I'm happy to be proven wrong, though.
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] OSM API lookups to complement minutely diffs?

[Stefan Keller]

I'm setting up a Kafka publish-subscribe messaging system delivering
minutely diffs.

AFAIK augmented diffs are rather an experimental feature and I'd like
to avoid the latency time and blackouts of overpass which runs in same
server. So I'm concentrating on the main OSM API.

Now, osmChange XML like
http://www.osm.org/api/0.6/changeset/42143238/download obviously does
not include all info (e.g. tags) from ref nodes/ways/relations.

I'm not looking for specific info but I'd like to get at least all
data about nodes/ways/relations which are created/modified/deleted in
a changeset.

Is it OK to do API lookups like this
https://www.osm.org/api/0.6/nodes?nodes=59906080,4400821613 even for
minutely diffs? Any alternatives?

:Stefan

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev