Re: [OSM-dev] planet.openstreetmap.org/replication policy

[mmd]

Am 28.11.2016 um 15:28 schrieb Martin Koppenhoefer:
> 
> 2016-11-28 14:33 GMT+01:00 Oliver Tonnhofer  >:
> 
> It would also not reduce the bandwidth by much, as it still needs to
> download the same data.
> 
> 
> 
> it surely will use a lot fewer connections, but also the amount of data
> to download can be significantly smaller, depending how often the same
> objects get touched within the same day.
> 

I don't think that's the case. Daily diffs contain every single version
of an object as long as it has been created/deleted/modified on that
particular day. Maybe you're somehow confusing this with the Overpass
API augmented diffs, which in fact leave out all 'intermediate' versions.

You can easily check this for yourself, see relation id 3227136 or way
id 448670397 in:

http://planet.openstreetmap.org/replication/day/000/001/514.osc.gz

I think the only difference with minutely diffs vs. daily diffs is the
fix cost associated to transferring a single file. While that's not much
of a concern for a single daily diff file, downloading many small
minutely diff will for sure accumulate lots of fix cost and make the
whole process quite slow. Also, it creates more server load.

-- 



___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org/replication policy

[Yves]

Martin is right,  using daily and hourly diffs you'll save bandwidth and reduce 
the number of transactions on your side to update Imposm DB.

Yves

Le 28 novembre 2016 15:28:36 GMT+01:00, Martin Koppenhoefer 
 a écrit :
>2016-11-28 14:33 GMT+01:00 Oliver Tonnhofer :
>
>> It would also not reduce the bandwidth by much, as it still needs to
>> download the same data.
>
>
>
>it surely will use a lot fewer connections, but also the amount of data
>to
>download can be significantly smaller, depending how often the same
>objects
>get touched within the same day.
>
>Cheers,
>Martin

-- 
Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org/replication policy

[Martin Koppenhoefer]

2016-11-28 14:33 GMT+01:00 Oliver Tonnhofer :

> It would also not reduce the bandwidth by much, as it still needs to
> download the same data.



it surely will use a lot fewer connections, but also the amount of data to
download can be significantly smaller, depending how often the same objects
get touched within the same day.

Cheers,
Martin
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org/replication policy

[Oliver Tonnhofer]

Hi,

> On 28.11.2016, at 13:44, Yves  wrote:
> 
> I think you could take daily and hourly diffs first to cope with the import 
> and last planet delay. 

That would make the implementation much more complex (what is the first hourly 
diff after a complete day?). It would also not reduce the bandwidth by much, as 
it still needs to download the same data. The code also uses keep-alive 
connections during the catch-up phase to reduce the load. 

The code is already working and behaves similar to --read-replication-interval 
from osmosis (as far as I unterstand). So I'm just asking what is acceptable: 
10 requests/s? 100 requests/s?

Regards,
Oliver

-- 
Oliver Tonnhofer  | Omniscale GmbH & Co KG  | https://omniscale.com
OpenStreetMap WMS and tile services | https://maps.omniscale.com
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org/replication policy

[Tom Hughes]

On 28/11/16 12:24, Oliver Tonnhofer wrote:


I'm the author of Imposm 3 (https://github.com/omniscale/imposm3/) and I'm 
working on a new command that will automatically download and import diff files 
from planet.openstreetmap.org as they appear.

Normally, it should only make two requests per minute when using minutely 
replication. One for the state and one for the osc.gz file. But after the 
initial import it will download the diff files as fast as Imposm can import 
them till it catches up with the live updates.

A fast server should be able to process 100 diffs per second and more, 
especially when only a smaller extract is imported. My question: Is this OK, or 
should I add a throttle for this?


As Yves said the best plan would be to use daily diffs until you get to 
the current day, then hourlies and only switch to minutelies when you 
get to the last hour.


Tom

--
Tom Hughes (t...@compton.nu)
http://compton.nu/

___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org/replication policy

[Komяpa]

For prior implementations, try looking at osmupdate:
https://wiki.openstreetmap.org/wiki/Osmupdate

пн, 28 нояб. 2016 г. в 15:47, Yves :

> I think you could take daily and hourly diffs first to cope with the
> import and last planet delay.
> Yves
>
>
> Le 28 novembre 2016 13:24:23 GMT+01:00, Oliver Tonnhofer 
> a écrit :
>
> Hi,
>
> I'm the author of Imposm 3 (https://github.com/omniscale/imposm3/) and I'm 
> working on a new command that will automatically download and import diff 
> files from planet.openstreetmap.org as they appear.
>
> Normally, it should only make two requests per minute when using minutely 
> replication. One for the state and one for the osc.gz file. But after the 
> initial import it will download the diff files as fast as Imposm can import 
> them till it catches up with the live updates.
>
> A fast server should be able to process 100 diffs per second and more, 
> especially when only a smaller extract is imported. My question: Is this OK, 
> or should I add a throttle for this?
>
> PS: The User-Agent is set to "Imposm 3 x.x.x".
>
>
> Regards,
> Oliver
>
>
> --
> Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.
> ___
> dev mailing list
> dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev
>
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] planet.openstreetmap.org/replication policy

[Yves]

I think you could take daily and hourly diffs first to cope with the import and 
last planet delay. 
Yves

Le 28 novembre 2016 13:24:23 GMT+01:00, Oliver Tonnhofer  a 
écrit :
>Hi,
>
>I'm the author of Imposm 3 (https://github.com/omniscale/imposm3/) and
>I'm working on a new command that will automatically download and
>import diff files from planet.openstreetmap.org as they appear.
>
>Normally, it should only make two requests per minute when using
>minutely replication. One for the state and one for the osc.gz file.
>But after the initial import it will download the diff files as fast as
>Imposm can import them till it catches up with the live updates.
>
>A fast server should be able to process 100 diffs per second and more,
>especially when only a smaller extract is imported. My question: Is
>this OK, or should I add a throttle for this?
>
>PS: The User-Agent is set to "Imposm 3 x.x.x".
>
>
>Regards,
>Oliver
>
>-- 
>Oliver Tonnhofer  | Omniscale GmbH & Co KG  | https://omniscale.com
>OpenStreetMap WMS and tile services |
>https://maps.omniscale.com
>
>
>
>
>
>___
>dev mailing list
>dev@openstreetmap.org
>https://lists.openstreetmap.org/listinfo/dev

-- 
Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

[OSM-dev] planet.openstreetmap.org/replication policy

[Oliver Tonnhofer]

Hi,

I'm the author of Imposm 3 (https://github.com/omniscale/imposm3/) and I'm 
working on a new command that will automatically download and import diff files 
from planet.openstreetmap.org as they appear.

Normally, it should only make two requests per minute when using minutely 
replication. One for the state and one for the osc.gz file. But after the 
initial import it will download the diff files as fast as Imposm can import 
them till it catches up with the live updates.

A fast server should be able to process 100 diffs per second and more, 
especially when only a smaller extract is imported. My question: Is this OK, or 
should I add a throttle for this?

PS: The User-Agent is set to "Imposm 3 x.x.x".


Regards,
Oliver

-- 
Oliver Tonnhofer  | Omniscale GmbH & Co KG  | https://omniscale.com
OpenStreetMap WMS and tile services | https://maps.omniscale.com





___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev