Re: [OSM-dev] Multipolygon relation way member without role

[Michael Zangl]

Am 19.09.2016 um 05:50 schrieb patrick keshishian:
> It seems that this association is a mistake. If so, is there a
> better place to report these sort of things as I come across them?
> As I am still learning my way through the data, I don't wish to
> do wholesale edits.

You can add a note about this:

http://www.openstreetmap.org/note/new#map=19/21.59793/-158.10305 ;-)

Michael

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] Multipolygon relation way member without role

[patrick keshishian]

Greetings,

Sorry about the subject line. I couldn't come up with anything better.

Continuing to learn my way through "relations" and assorted "tags",
I came across relation=3489649 [0], island of Oahu, HI, USA. What
interests me is its "way" member=428362225 (Way: War Memorial World
War II) [1] with role="".

It seems that this association is a mistake. If so, is there a
better place to report these sort of things as I come across them?
As I am still learning my way through the data, I don't wish to
do wholesale edits.

Question: Is there a general rule used when processing role=""?
OSM wiki for Relation [3] indicates that roles are optional. It
looks that OSM is rendering way=428362225 as an area [4].

Thanks in advance,
--patrick


[0] http://www.openstreetmap.org/relation/3489649
[1] http://www.openstreetmap.org/way/428362225
[3] http://wiki.openstreetmap.org/wiki/Relation
[4] http://www.openstreetmap.org/relation/3489649#map=19/21.59777/-158.10302

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [overpass] OSM API lookups to complement minutely diffs?

[mmd]

Am 18.09.2016 um 15:27 schrieb Stefan Keller:

> 
> 0. The typical delay (augmented diff id) compared to current time is 1
> to 4 minutes.

Right, you can see the current delay in munin [1], and the current
timestamp via a dedicated API call [2].

[1]
http://overpass-api.de/munin/localdomain/localhost.localdomain/osm_db_lag.html
[2] http://overpass-api.de/api/timestamp

> 1. The id is always increasing but sometimes an id (or more) is/are
> left out between to consecutive deliveries.

At first sight, IDs may appear to be left out. But as the ID just
corresponds exactly to the database timestamp, there may be some
non-consecutive increases.

However, that's not really an issue. The reason for the larger gaps is
the way how minutely diffs are processed: if there's a backlog of a few
minutes, several diffs will be processed as one package to speed up
processing and the timestamp increases by more than a minute.

From the outside, augmented_diff_status appears to have jumped by
several numbers, but you still have to query every number in between,
even if it wasn't announced by augmented_diff_status.

> 2. The time interval for a new diff can be more than a minute
> (sometimes hours in blackouts).

Please remember that augmented diffs are calculated on the fly. Based on
the sequence id you provide, data from an interval of exactly one minute
is returned.

It also doesn't matter how the minutely diffs were originally posted to
the database. Augmented diffs just follow a fixed 60 second grid, with a
1:1 relationship between the id and the respective timeframe.

For reference, here's the respective code snippet. I think a few lines
of code tell a thousand words ;)

https://github.com/drolbr/Overpass-API/blob/master/src/cgi-bin/augmented_diff#L40-L48

> So there may be still missing diff files when fetching up (which means
> that an internal sequence id is not enough; there is also a need for a
> list of missing id's)?

There are really no missing ids, as mentioned, augmented_diff_status
just returns the maximum available number to be queried. You still need
to query every single id up to and including that number.

> 
> And the client can poll every minute (sometimes longer if client is
> busy or diff downloading takes time) without hitting load-limitations?
> 

Well, http://overpass-api.de/api/status should have a line containing
"slots available now" in there, otherwise, all slots are currently busy
and you will hit a HTTP 429 error. IIRC, Roland explained that idea
somewhere on the Overpass Dev list before...

-- 



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] [overpass] OSM API lookups to complement minutely diffs?

[Stefan Keller]

Thanks mmd: That was exactly what I tried to summarize after some observations:

0. The typical delay (augmented diff id) compared to current time is 1
to 4 minutes.
1. The id is always increasing but sometimes an id (or more) is/are
left out between to consecutive deliveries.
2. The time interval for a new diff can be more than a minute
(sometimes hours in blackouts).
3. The interval delivering new diffs can be less than a minute (esp.
also when previous left out's are handed in later).

So a client polls augmented_diff_status and
a. waits to download diffs if there's no new id number from
augmented_diff_status,
b. or it fetches diffs up to and including to the augmented_diff_status id

So there may be still missing diff files when fetching up (which means
that an internal sequence id is not enough; there is also a need for a
list of missing id's)?

And the client can poll every minute (sometimes longer if client is
busy or diff downloading takes time) without hitting load-limitations?

:Stefan

P.S. I'm at SoTM too this week!


2016-09-18 14:58 GMT+02:00 mmd :
> Hi,
>
> Am 18.09.2016 um 14:41 schrieb Michael Larsen:
>> Hi,
>>
>> Last time I tried consuming augmented diffs on a minutely basis, I hit the
>> load-limitations which meant that I could not consume augmented diffs for 
>> some
>> time afterwards, i.e. this will lead to black holes in your history.
>
> the value returned by augmented_diff_status corresponds to current
> database timestamp (as number of minutes since the license change). It
> does not need to increase one-by-one(!), e.g. the database may process
> several minutely diffs in one go, due to some backlog. If you always
> download the number returned by augmented_diff_stauts, you will indeed
> get some holes!
>
> That situation can be avoided, if you keep your own internal sequence
> id, and fetch augmented diffs up to and including to the value returned
> by augmented_diff_status.
>
> If augmented_diff_status does not return any value (due to overload),
> just wait some time and try again. The same applies to downloading
> augmented_diffs: you may get HTTP 429 or HTTP 504 in case of overload,
> or if you exceed your quota (see /api/status for details). In that case,
> don't increase your internal sequence id yet, but try downloading the
> same augmented diff again.
>
>>
>> Also, using timestamp start/end to fetch diffs for a given timestamp (like
>> avachi) is problematic with some changesets that stay open for > 1 hours 
>> (this
>> happens quite ofter). The live service running on osm.expandable.dk use the
>> API as described previously to get the augmented diff for a changeset. If
>> there where a better way I'm all ears!
>>
>
> I hope the situation will improve a bit once the database has moved to
> version 0.7.53 and a different compression. If you're at SotM next week,
> you could maybe ask Roland for a current status.
>
> --
>
>

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[mmd]

Hi,

Am 18.09.2016 um 14:41 schrieb Michael Larsen:
> Hi,
> 
> Last time I tried consuming augmented diffs on a minutely basis, I hit the 
> load-limitations which meant that I could not consume augmented diffs for 
> some 
> time afterwards, i.e. this will lead to black holes in your history.

the value returned by augmented_diff_status corresponds to current
database timestamp (as number of minutes since the license change). It
does not need to increase one-by-one(!), e.g. the database may process
several minutely diffs in one go, due to some backlog. If you always
download the number returned by augmented_diff_stauts, you will indeed
get some holes!

That situation can be avoided, if you keep your own internal sequence
id, and fetch augmented diffs up to and including to the value returned
by augmented_diff_status.

If augmented_diff_status does not return any value (due to overload),
just wait some time and try again. The same applies to downloading
augmented_diffs: you may get HTTP 429 or HTTP 504 in case of overload,
or if you exceed your quota (see /api/status for details). In that case,
don't increase your internal sequence id yet, but try downloading the
same augmented diff again.

> 
> Also, using timestamp start/end to fetch diffs for a given timestamp (like 
> avachi) is problematic with some changesets that stay open for > 1 hours 
> (this 
> happens quite ofter). The live service running on osm.expandable.dk use the 
> API as described previously to get the augmented diff for a changeset. If 
> there where a better way I'm all ears!
> 

I hope the situation will improve a bit once the database has moved to
version 0.7.53 and a different compression. If you're at SotM next week,
you could maybe ask Roland for a current status.

-- 



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] OSM API lookups to complement minutely diffs?

[Michael Larsen]

Hi,

Last time I tried consuming augmented diffs on a minutely basis, I hit the 
load-limitations which meant that I could not consume augmented diffs for some 
time afterwards, i.e. this will lead to black holes in your history.

Also, using timestamp start/end to fetch diffs for a given timestamp (like 
avachi) is problematic with some changesets that stay open for > 1 hours (this 
happens quite ofter). The live service running on osm.expandable.dk use the 
API as described previously to get the augmented diff for a changeset. If 
there where a better way I'm all ears!

Regards,
MichaelVL

On søndag den 18. september 2016 01.59.57 CEST Stefan Keller wrote:
> Thanks mmd,
> 
> As said, I'd like to fetch the latest available augmented diff and
> keep uptodate.
> So, to understand it correctly:
> Every minute I first call the
> http://overpass-api.de/api/augmented_diff_status which returns an id,
> e.g. 2111988,
> then I fetch the augmented diff file
> http://overpass-api.de/api/augmented_diff?id=2111988
> Just want to be sure that this is the correct way to use the API.
> 
> :Stefan
> 
> 2016-09-17 16:48 GMT+02:00 mmd :
> > Am 15.09.2016 um 22:53 schrieb Stefan Keller:
> >> I'm setting up a Kafka publish-subscribe messaging system delivering
> >> minutely diffs.
> >> 
> >> AFAIK augmented diffs are rather an experimental feature and I'd like
> >> to avoid the latency time and blackouts of overpass which runs in same
> >> server. So I'm concentrating on the main OSM API.
> >> 
> >> Now, osmChange XML like
> >> http://www.osm.org/api/0.6/changeset/42143238/download obviously does
> >> not include all info (e.g. tags) from ref nodes/ways/relations.
> >> 
> >> I'm not looking for specific info but I'd like to get at least all
> >> data about nodes/ways/relations which are created/modified/deleted in
> >> a changeset.
> > 
> > This sounds a lot like what achavi is doing right now.
> > 
> > Note that with the introduction of the famous 'attic' concept, augmented
> > diffs are nowadays simply translated into a plain Overpass QL query and
> > calculated on the fly.
> > 
> > This also means that you don't have to stick to a minute interval or a
> > global scope as before: you're free to provide time intervals matching a
> > changeset (like achavi does), or restrict the adiff to a certain
> > bounding box or even some nodes/ways/relations with certain tags only.
> > 
> > There's no XSD around, the wiki page should answer most questions,
> > though. In general, the Overpass Dev list would be a good place for more
> > details and discussions. [1]
> > 
> > [1] http://listes.openstreetmap.fr/wws/info/overpass
> > 
> > 
> > 
> > 
> > ___
> > dev mailing list
> > dev@openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/dev
> 
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev