A quick summary of the OSM data model: Nodes: the only entity with spatial information. Ways: an ordered list of nodes
With that knowledge we can determine why osm2pgsql (and other geometry-building conversion apps take a while to update): On first import osm2pgsql stores all nodes and their locations in memory as they are read in (since nodes come first in an OSM XML file). After parsing nodes, osm2pgsql must build the geometries for ways. On first import, the geometry information for the nodes are stored in memory and so can be retrieved quickly when building way geometry. On subsequent updates osm2psgql does not have node information in memory anymore, so it must request the node information from PostgreSQL. This takes orders of magnitudes longer to do than a hit to memory. To speed up, osm2pgsql could probably run as a daemon (rather than once a minute) and cache these node locations in memory. The hit rate might not be spectacular but it could speed up a little bit. If the server has boatloads of memory or only needs to update a small chunk of the world then it could pre-load the memory cache with that region's nodes beforehand. -Ian On Tue, Apr 5, 2011 at 3:58 AM, Markus Wagner <[email protected]> wrote: > Hey everybody, > > I hope this list suits the problem I'm trying to solve - searching the > lists, forums, former posts, etc. didn't help. > > I followed (roughly) this > http://wiki.openstreetmap.org/wiki/DE:HowTo_minutely_hstore > guide to setup a PostGIS server; did a full planet import (took 16.x days) > and now I want to catch up by importing the diffs as described there. > Problem is, that a 5min fragment takes 7-11 minutes. (The same factor > applies to imports of an hour, half day, etc.) > > How can I investigate what the bottleneck is? So far, I can see, that there > is a lot of I/O going on and not much CPU usage. The diff import is stuck > upon prompting "Going over pending relations","processing relation (...)". > > Some specs of my setup: > -Debian 64Bit > -PostgreSQL 9.0.3, PostGIS 1.5.2; having its own physical drive with XFS > -Intel(R) Core(TM)2 Quad CPU Q9505 @ 2.83GHz > -8 GB physical RAM > -The initial (and diff) imports have been using the default.style and > keeping keeping coastlines > -a GIN Index has been created, turning its fastupdate option to off didn't > help (rather, it even prolonged the append run) > > Any pointers, tipps are welcome! > > Thank you, > Markus > -- > Markus Wagner > [email protected] > > > > _______________________________________________ > dev mailing list > [email protected] > http://lists.openstreetmap.org/listinfo/dev >
_______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
