Re: [OSM-dev] Osm2pgsql and failed planet import
On 30 August 2011 20:47, Frederik Ramm frede...@remote.org wrote: I guess this needs some thorough investigation and I always hoped that someone more familar with that part of the code (hint, hint) would find the time to have a look ;) I guess it would be helpful if someone with this error could actually look in the tables and see how many ways are pending (is it all of them?). I guess it has to be ways, because I can't imagine there being enough relations to cause this problem. In any case, I had a look at the code and there aren't many ways that a way can be marked pending. However, a way can be marked pending on insert (I can't remember de rationale for that). That happens when the fourth argument for pgsql_ways_set is true, which is on the condition (!filter polygon) and polygon is set in the pgsql_filter_tags function. That code has a comment about that it used to stop early but doesn't any more. As far as I can see the FLAG_POLYGON is now always set to true which means every way is marked pending on insert. Which would cause the symptoms describes. So: 1. Looks like the restructuring of pgsql_filter_tags missed a case. Not sure what the problem was with the way it was. 2. I don't recall the reason for marking polygons as pending. Does anybody else? Have a nice day, -- Martijn van Oosterhout klep...@gmail.com http://svana.org/kleptog/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
On 08/30/2011 11:50 AM, John Smith wrote: osm2pgsql doesn't have any code to check for memory allocation failures and to deal with it in a sane way, it just assumes all allocations are fine until it checks the nodes when going over pending ways etc. Anthony posted a patch a couple of months back, I didn't hear if the patch was added to the svn version(s). i'm not aware of any other patch, but i changed the cache allocation code quite recently to allocate the full configured cache size up front instead of doing so in blocks of 8KB. The main reason for this was that although all cache blocks get freed once the cache is no longer needed all this memory is very unlikely to get returned to the OS due to heap fragmentation. Allocating everything up front has the advantage that malloc() will use mmap() to map the cache memory into the process memory space instead of putting it on the heap, and so it can later be free()d without being affected by potential heap fragmentation. As a side effect this also allows to check up front that there is sufficient memory for the configured cache size instead of running into out-of-memory situations only after the import had already been going for potentially quite a while. As the memory is really freed before the index building step now it is possible to configure larger work_mem and maintenance_work_mem buffers on the postgresql side without having to wait for the unused-but-still-allocated osm2pgsql process memory gradually being pushed out to swap over time. Current code can be found here: https://github.com/hholzgra/osm2pgsql/tree/freeable_cache -- hartmut ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Osm2pgsql and failed planet import
Hi everyone, I've been attempting to perform a planet import for mapnik rendering for the past few weeks without success. I've setup my PostGreSQL database with PostGis correctly, including the various parameter tweaks. I setup PostGreSQL on a separate Windows Server 2008 box with 8GB RAM, in order to separate processes and not have osm2pgsql have missing memory issues. This box has plenty of space for the final database. I setup a virtual box on our virtualization server using a CentOS distribution, obtained osm2pgsql from source, turned on the 64bit compilation flags and compiled it. This virtual box has 16GB memory. I've attempted several times to import, without success. The last attempt I made was using a boundary box, hoping osm2pgsql would have an easier time just importing North America. I'm using slim mode and a 14GB cache. Here is the command line and the output: ./osm2pgsql64 -c -s -m -C 14000 --bbox -165,13,-45,90 -U gis -W -H 192.168.1.131 -P 5432 -S ./default.style ./planet-110810.osm.bz2 osm2pgsql SVN version 0.80.0 (64bit id space) Password: Using projection SRS 900913 (Spherical Mercator) Applying Bounding box: -165.00,13.00 to -45.00,90.00 Setting up table: planet_osm_point NOTICE: table planet_osm_point_tmp does not exist, skipping Setting up table: planet_osm_line NOTICE: table planet_osm_line_tmp does not exist, skipping Setting up table: planet_osm_polygon NOTICE: table planet_osm_polygon_tmp does not exist, skipping Setting up table: planet_osm_roads NOTICE: table planet_osm_roads_tmp does not exist, skipping Mid: pgsql, scale=100, cache=14000MB, maxblocks=1792001*8192 Setting up table: planet_osm_nodes NOTICE: table planet_osm_nodes does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index planet_osm_nodes_pkey for table planet_osm_nodes Setting up table: planet_osm_ways NOTICE: table planet_osm_ways does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index planet_osm_ways_pkey for table planet_osm_ways Setting up table: planet_osm_rels NOTICE: table planet_osm_rels does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index planet_osm_rels_pkey for table planet_osm_rels Reading in file: ./planet-110810.osm.bz2 Unknown node type 8 Processing: Node(1173423k) Way(104443k) Relation(1070274) parse time: 175361s Node stats: total(1173423751), max(1393174131) Way stats: total(104443727), max(125432779) Relation stats: total(1070274), max(1706327) Going over pending ways processing way (26160k)way_done failed: ERROR: out of memory DETAIL: Failed on request of size 419430400. (7) Arguments were: 75277817, Error occurred, cleaning up Now the error occurs with or without 64bits, with or without the boundary box. It seems like the pending ways section of the code is behaving oddly, or I've setup something incorrectly. The size of the memory request is odd as well. Anybody have any ideas? Best regards to all, Francisco -- View this message in context: http://gis.638310.n2.nabble.com/Osm2pgsql-and-failed-planet-import-tp6729516p6729516.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
On 27 August 2011 03:18, Francisco.Matamala francisco.matam...@weblakes.com wrote: Hi everyone, I've been attempting to perform a planet import for mapnik rendering for the past few weeks without success. I've setup my PostGreSQL database with PostGis correctly, including the various parameter tweaks. I setup PostGreSQL on a separate Windows Server 2008 box with 8GB RAM, in order to separate processes and not have osm2pgsql have missing memory issues. This box has plenty of space for the final database. I setup a virtual box on our virtualization server using a CentOS distribution, obtained osm2pgsql from source, turned on the 64bit compilation flags and compiled it. This virtual box has 16GB memory. osm2pgsql doesn't have any code to check for memory allocation failures and to deal with it in a sane way, it just assumes all allocations are fine until it checks the nodes when going over pending ways etc. Anthony posted a patch a couple of months back, I didn't hear if the patch was added to the svn version(s). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
Hi, On 08/30/11 11:50, John Smith wrote: osm2pgsql doesn't have any code to check for memory allocation failures and to deal with it in a sane way The error message discussed here is not a segfault, but an out of memory condition reported by the PostgreSQL client library. It has nothing to do with osm2pgsql's own memory allocation. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
On 30 August 2011 12:07, Frederik Ramm frede...@remote.org wrote: Hi, On 08/30/11 11:50, John Smith wrote: osm2pgsql doesn't have any code to check for memory allocation failures and to deal with it in a sane way The error message discussed here is not a segfault, but an out of memory condition reported by the PostgreSQL client library. It has nothing to do with osm2pgsql's own memory allocation. This indicates the code should be changed to retrieve the data in blocks using a cursor. Mvg, -- Martijn van Oosterhout klep...@gmail.com http://svana.org/kleptog/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
I will look into the code that pre-allocates memory, this definitely sounds like a good idea. Out of curiosity, has anyone else reported problems with full planet imports lately? Thank you to everyone for their input and time. -- View this message in context: http://gis.638310.n2.nabble.com/Osm2pgsql-and-failed-planet-import-tp6729516p6743304.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
Hi, On 08/30/2011 07:43 PM, Martijn van O wrote: The error message discussed here is not a segfault, but an out of memory condition reported by the PostgreSQL client library. It has nothing to do with osm2pgsql's own memory allocation. This indicates the code should be changed to retrieve the data in blocks using a cursor. Yes, that would be fixing the symptom. But the heart of the problem seems to be (as you said yourself a while ago in this discussion) that the whole dirty mark scheme should not really be used during an initial import at all, or should it? I guess this needs some thorough investigation and I always hoped that someone more familar with that part of the code (hint, hint) would find the time to have a look ;) Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql and failed planet import
On 31 August 2011 02:44, Hartmut Holzgraefe hartmut.holzgra...@gmail.com wrote: i'm not aware of any other patch, but i changed the cache allocation http://lists.openstreetmap.org/pipermail/dev/2011-June/023002.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
