Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Jochen Topf
On Sun, Oct 26, 2008 at 06:11:04PM -0700, Michal Migurski wrote: What is the difference between osmosis and osm2pgsql, with regards to postGIS? osm2pgsql creates the structure needed for Mapnik. Osmosis creates a structure more simliar to the one in the OSM central database. If I've been

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Tom Hughes
Shaun McDonald wrote: On 27 Oct 2008, at 00:50, Michal Migurski wrote: Planet dumps are not snapshots - they do not represent a consistent view at any particular point in time because they take a number of hours to generate, during which time new changes are constantly being made to the

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Jochen Topf
On Mon, Oct 27, 2008 at 08:22:32AM +, Tom Hughes wrote: Shaun McDonald wrote: On 27 Oct 2008, at 00:50, Michal Migurski wrote: Planet dumps are not snapshots - they do not represent a consistent view at any particular point in time because they take a number of hours to generate,

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
Others have already commented on most of your points but I'll add my thoughts in case there's some gaps. Michal Migurski wrote: Hi, I've been trying to keep up to date with the dumps and diffs from http://planet.openstreetmap.org/ , and I'm running into a number of bugs related to cutoff

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
Jochen Topf wrote: If the planet dump plus the diff from the same day is what everybody wants anyway, why not do this on the server side and hold the planet back after the first diff is available, run this over the planet and then publish that as the planet? It would add delay to the

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Frederik Ramm
Hi, Brett Henderson wrote: Brett Henderson has offered to look into creating the dailies from history as well, but I don't know about the status of that. Are you referring to the daily changesets? [...] Or did you mean planets instead of dailies? Mix-up on my part, sorry, yes I meant

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Michal Migurski
Yep, as others have commented there are two tables types in the osm database; current tables, and history tables. The planet dumper just reads current tables which is the fastest approach. Unfortunately the current tables change constantly during the planet generation process

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Martijn van Oosterhout
On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski [EMAIL PROTECTED] wrote: I'm liking Jochen Topf's suggestion here: If the planet dump plus the diff from the same day is what everybody wants anyway, why not do this on the server side and hold the planet back after the first diff is

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Martijn van Oosterhout
On Mon, Oct 27, 2008 at 9:40 PM, Michal Migurski [EMAIL PROTECTED] wrote: Now that I think about it though, I think what I did was take one of the planet dumps from http://hypercube.telascience.org/planet/ (which *are* consistant snapshots), and run the dailies from there. Is there any reason

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
On Tue, Oct 28, 2008 at 7:39 AM, Michal Migurski [EMAIL PROTECTED] wrote: Finally, the boundaries between the hourlies and dailies seem misaligned. This shouldn't be the case. After running the remaining hourlies for the 22nd, I attempted to pick up on the 23rd with a daily. The

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Michal Migurski
On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski [EMAIL PROTECTED] wrote: I'm liking Jochen Topf's suggestion here: If the planet dump plus the diff from the same day is what everybody wants anyway, why not do this on the server side and hold the planet back after the first diff

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Tom Hughes
Michal Migurski wrote: The final event in each weekly planet dump does not fall on an even day boundary. In the case of the most recent Oct. 22nd planet.osm, it was necessary to experiment with hourly diffs from that day to find that the boundary was approx. 2:00pm. Hourlies up to and

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Frederik Ramm
Hi, Michal Migurski wrote: I've noticed some misalignments between the data in the dumps and the osm2pgsql importer that leads to unavoidable holes in the data. As TomH has already said, this is not a bug, it stems from the fact that the full planet export reads the current tables and as

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Michal Migurski
The final event in each weekly planet dump does not fall on an even day boundary. In the case of the most recent Oct. 22nd planet.osm, it was necessary to experiment with hourly diffs from that day to find that the boundary was approx. 2:00pm. Hourlies up to and including

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Michal Migurski
On Oct 26, 2008, at 5:50 PM, Frederik Ramm wrote: Brett Henderson has offered to look into creating the dailies from history as well, but I don't know about the status of that. If you use osmosis, it is safe (and in fact recommended) that, after loading the database with a planet file

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Shaun McDonald
On 27 Oct 2008, at 00:50, Michal Migurski wrote: The final event in each weekly planet dump does not fall on an even day boundary. In the case of the most recent Oct. 22nd planet.osm, it was necessary to experiment with hourly diffs from that day to find that the boundary was approx.