Re: [OSM-dev] Osm2pgsql seg fault
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I just wanted to give a short update on the issue. I did an update of osm2pgsql to 0.84 and tried again to import the diffs (minutely mapnik in conjunction with osmosis), but no prevail. Interestingly, the crash is related to a certain day, 2014-01-05, after setting the state.txt to the next day, imports were again processed. But the problem seems to go a bit deeper. On my database server I am experiencing the same problem while importing the full planet (planet-140226.osm.pbf) into the database. Importing smaller extracts work though. I've tested Luxembourg and that finishes in 81s, right now I'm trying the Europe extract from Geofabrik. For the planet dump I've already checked the md5 Here's the last output from osm2pgsql before the crash, the gdb bt is the same as with my first email concerning the crash: Reading in file: planet-140226.osm.pbf Processing: Node(2219907k 62.8k/s) Way(219165k 7.62k/s) Relation(741130 33.82/s)Segmentation fault (core dumped) I tried it now for several times, and osm2pgsql exits always at the same relation (if that's possible): 741130 Any ideas? Am 2014-02-03 23:35, schrieb Paul Norman: From: Frank Broniewski [mailto:b...@metrico.lu] Sent: Monday, February 03, 2014 2:13 AM To: OSM Dev List Subject: [OSM-dev] Osm2pgsql seg fault I am having segmentation faults with osm2pgsql and I am having trouble to identify the reason. I presume it is related to some updates of base libraries on my system but I am not really sure if that is the case. I am running osm2pgsql on FreeBSD 9.2-RELEASE-p3. I tried already to delete it and reinstall it, but this did not help. Pkg_libchk does not yield any problems. Osm2pgsql is: osm2pgsql --version osm2pgsql SVN version 0.81.0 (64bit id space) This is an old version. Does the problem still occur on 0.84 or later? If it does, you'll have to provide more information like the command you used to call osm2pgsql. - -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTHWEAAAoJEHopqW0d1cQVWXMH/jTzSE8IV7rf6RaPike3Sr15 bae9/43B02n6n2onisPgj+EPIAe2mwP1uDuTl/ZdXQehB8VlAzvIUfb2K0k0SBee JHcEeRWjAE8rtcOkLMNl0tqcxUjjtC/y3B3/s8v9ideckDfp27bUzdc8OISVvKeA 169t7tfU+cV5SX193Et5nT6QfE/J9yrc/qjY2uoSnRz6uOrPGqW02YU0csWDlD9O v4bVVkrPNz36ws2e+I3t59+mjO7vgk+747nD+u7JcEOaS8Pc6iFAqgF0ctXcu7dR jOXU8ymZaeQO/TyTYCuc/nA1mB2yhpH2B7NA+iGUDFK+221biCdSiy9//C+ZrIc= =EsBB -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Osm2pgsql seg fault
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I am having segmentation faults with osm2pgsql and I am having trouble to identify the reason. I presume it is related to some updates of base libraries on my system but I am not really sure if that is the case. I am running osm2pgsql on FreeBSD 9.2-RELEASE-p3. I tried already to delete it and reinstall it, but this did not help. Pkg_libchk does not yield any problems. Osm2pgsql is: osm2pgsql --version osm2pgsql SVN version 0.81.0 (64bit id space) Running gdb on the core file gets me the following results gdb /usr/local/bin/osm2pgsql osm2pgsql.core (gdb) bt full #0 0x00080299bed8 in snprintf () from /lib/libc.so.7 No symbol table info available. #1 0x0040e8a1 in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #2 0x0040ef05 in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #3 0x0040a7e7 in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #4 0x0040b7bd in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #5 0x00417820 in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #6 0x0041798d in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #7 0x004129e3 in std::vectorstd::string, std::allocatorstd::string ::_M_insert_aux () No symbol table info available. #8 0x00404e71 in ?? () No symbol table info available. #9 0x00080064a000 in ?? () No symbol table info available. #10 0x in ?? () No symbol table info available. (gdb) Does anyone have a hint for me what actually might go wrong here? Thanks, Frank - -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS72uNAAoJEHopqW0d1cQVePoH/A0WqK2AuuAmTbAq4wow3Y81 0SMAdUK27ckJCgrNl0Ny2jiR5nNxCCdfJ6W1W3R89JWEJgI0Y4Qxerz18vs9ixkP NXdgL7/KO1ulFiHa2ku7a1CTKyAflTQdnbiG/KLAtPmVs83cQWdRpIFl/eap4Qa2 uxDebUyzcCYiggVeg94+HgfkGPLZXHC0XK8SXrB1b4rlh/X81VFXU3sgVkM5It29 tpl/Uqb5n1PZFvnFtPzocgPCn8YneOj6FNVzjoOgZ/Lb1AdSHdEaFxSeQghbxov+ XNUWXXAXjhEc75oG+idEuokcgML23E643qXg0dzHxmf1SQtgGK2fWDfgVNQ+CZ4= =vMtO -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql seg fault
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Paul, thanks for your answer. You're right of course, I should have added the command to my first email immediately, so I'll do it now: /usr/local/bin/osm2pgsql --number-processes 4 --append --slim --cache 4096 --proj 3857 --prefix planet_osm --style /usr/home/brfr/osm/default.style --host 10.0.0.2 --database osm - --username osm2pgsql --hstore-all --verbose /tmp/osm-load-next.65646.osc And I'll investigate the process with a newer version of osm2pgsql and report back on the results. Frank Am 2014-02-03 23:35, schrieb Paul Norman: From: Frank Broniewski [mailto:b...@metrico.lu] Sent: Monday, February 03, 2014 2:13 AM To: OSM Dev List Subject: [OSM-dev] Osm2pgsql seg fault I am having segmentation faults with osm2pgsql and I am having trouble to identify the reason. I presume it is related to some updates of base libraries on my system but I am not really sure if that is the case. I am running osm2pgsql on FreeBSD 9.2-RELEASE-p3. I tried already to delete it and reinstall it, but this did not help. Pkg_libchk does not yield any problems. Osm2pgsql is: osm2pgsql --version osm2pgsql SVN version 0.81.0 (64bit id space) This is an old version. Does the problem still occur on 0.84 or later? If it does, you'll have to provide more information like the command you used to call osm2pgsql. - -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS8JfAAAoJEHopqW0d1cQVFNwH+QGE9bt8c8Stx5yR4ByjihWM /KQ/Nt9YMupj3RQXTnNoY3H3ifQAp2ZFMjZrdKkN08/DfxyN8jkzdGmTgWNtWbhe EYLSH9tYwb4CwtJF9CsTFQyJV5t7Gv4TZmaQEVQz15q4FXjGhnSLhRFcc9Jy9afh 437DFDSXR8ST4NSeVPZMxDUHajyFzJrKP1TIhAo5/BKlGeKMgCC0gqAstzWD8CYI aaOuJfWh9z5eY7Do2QvrfDyaxp5s9I0oo3sG3RJQ6TUvTD7K8SMJffs4yZphUjsb voIYUCYWfXjVh4q8UuxEth0Z8kdQJV97YaKbTUk8x18e+lUrKLLRPDBjjZsuWGs= =eA+j -END PGP SIGNATURE- ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Website to generate polygons from OSM relations
I tried that once too, but then I stopped using a osmosis based DB and switched to osm2pgsql. But I had a somewhat 90% working solution ... I've uploaded my SQL files to my repo [1], maybe you find some ideas on how to treat inner and outer polygons. There're no comments on the SQL so it might be hard to grasp, but you can find your way around. Frank [1] https://bitbucket.org/metrico/osm-sql Am 2013-06-12 13:37, schrieb Jocelyn Jaubert: On Wed, Jun 12, 2013 at 08:41:55AM +0200, Frank Broniewski wrote: nice work! Your service is really running quite fast. How do you determine what a polygon relation is? Do you have a list of keys and values, like osm2pgsql does? Or do you check if a geometry is closed and produce polygons from them? I check if a geometry is closed, and generate polygon from it. For example, if you try on a bus line, you will get the errors generated from postgresql, with information on where the disconnection might be: http://polygons.openstreetmap.fr/?id=36052 NOTICE: missing connexion at point -0.1406638f 51.5131253f NOTICE: missing connexion at point -0.135611f 51.5099509f This is done by plgsql function create_polygon() here: https://github.com/jocelynj/osm-polygons-generation/blob/master/init.sql#L24 Note that I think that it doesn't correctly handle inner members of relations. -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Website to generate polygons from OSM relations
Hey Jocelyn, nice work! Your service is really running quite fast. How do you determine what a polygon relation is? Do you have a list of keys and values, like osm2pgsql does? Or do you check if a geometry is closed and produce polygons from them? This was one of my biggest issues when I tried to convert a osmosis geometry to polygons. Good work, Frank Am 2013-06-11 23:45, schrieb Jocelyn Jaubert: Hi, I have put into production a website to generate various outputs from OSM relations, by taking into account inclusion of other relations. It is available at: http://polygons.openstreetmap.fr/ What it can do: - generate whole geometry from an OSM relation ID, with handling of sub-relations - this works especially well with the recursive French model for administrative relations, and can be used as a sanity check. - generate simplified geometries, even bigger or smaller - this is useful to cut a .osm file with a good .poly file. - export WKT, .poly, GeoJSON, or even a picture. - import a user .poly file, and then do an Union from an OSM relation. I used this to fix some polygons used for Geofabrik extracts. A few examples: OSM relation for Corsica: http://polygons.openstreetmap.fr/?id=76910 http://polygons.openstreetmap.fr/get_image.py?id=76910params=0.02-0.005000-0.005000 Same thing for France (image is a little longer to generate) http://polygons.openstreetmap.fr/?id=1362232 Union of Geofabrik and OSM relation for PACA, a french region: http://polygons.openstreetmap.fr/get_image.py?id=8654params=0.02-0.005000-0.005000name=provence-alpes-cote-d-azur_5k3mbx (you can see that some part of the region were missing from the yellow polygon) The source code is available at: https://github.com/jocelynj/osm-polygons-generation This uses an osmosis database, containing the whole word, and some Postgis functions. If you have any suggestion, don't hesitate to propose them ! Thanks, Jocelyn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Is there a way to use simple schema without hstore
Am 18.11.2010 10:18, schrieb Andreas Kalsch: Is there a way to use simple schema in Osmosis without hstore? And why was this changed? A separate table for tags can more easily be indexed. I think it is not a good idea to use hstore because then we can drop SQL, use NoSQL for storing data and use PostGIS/Postgres for Geometry only. What do you think? Best, Andi ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev Hello Andi, You can create an index for the tags column. hstore supports gist and gin indexes and plus it saves you a m:n join. And I don't see why using hstore data type is like using NoSQL? You can still extract the tags into a seperate table, if you like of course ;-) Frank ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Relation member_roles from Osmosis import
Cool, thanks for sharing the link, I didn't knew it. It's really helpful. I know that the data in OpenStreetMap may have issues - my questions simply arrive while I explore the osmosis import in my postgis database. And I'm one of the issues myself :D because I still learning the OpenStreetMap data format. But seeing the discussion that arose from my question concerning relations and super relations and such this still seems to be an evolving area ... Many thanks Frank Am 29.10.2010 10:14, schrieb Jochen Topf: Common. There are many, many broken multipolygon relations. See http://tools.geofabrik.de/osmi/debug.html?view=multipolygonlon=13.04883lat=44.58577zoom=5 Whatever you do with OSM data you have to take into account that its not perfect and work around that. Jochen On Thu, Oct 28, 2010 at 02:48:12PM +0200, Frank Broniewski wrote: Date: Thu, 28 Oct 2010 14:48:12 +0200 From: Frank Broniewskib...@metrico.lu To: Jochen Topfjoc...@remote.org CC: dev@openstreetmap.org Subject: Re: [OSM-dev] Relation member_roles from Osmosis import Hello Jochen, thanks for your answer, that makes it clear. I have another question about relations. I found a relation, a lake in Germany, (id=1104680, Schweriner See (Außensee)), which has serveral inner relations to ways. Some of these ways form a closed ways, so it's easy to create a polygon from it. But other ways with the relation role inner are not closed, but they form together an island in the lake. Should't those form a relation of themselves before relating them to the lake relation with an inner relation? The ways I am speaking of are 24563810 70191810 70191809 70191807 Luckily those ways form an island but if there would be another island formed by ways without relating them together it is quite difficult to recreate them properly. Is this situation an exception or is this a common situation? Frank Am 27.10.2010 11:01, schrieb Jochen Topf: On Wed, Oct 27, 2010 at 10:39:38AM +0200, Frank Broniewski wrote: During my examinations from an OSM import (osmosis pbf to postgis) I found some strange looking entries in the relation_members table. My goal is to make (GIS) polygons from the linestrings table and therefore I examined the relations a little closer. Doing a select distinct relation_id, member_role from relation_members I find these entries in the result, which I suspect to be wrong relation_id, member_role 22852;forward_stop_0 22852;forward_stop_11 22852;forward_stop_19 22852;forward_stop_9 22852;stop_1 22852;stop_10 22852;stop_12 22852;stop_13 22852;stop_14 300710;backward_stop_%n 207675;forward_stop_%n I think that numbering the member_role is not the intended way ;-) Thats a left-over from the time when relation members were not ordered. Jochen ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Relation member_roles from Osmosis import
Hello Jochen, thanks for your answer, that makes it clear. I have another question about relations. I found a relation, a lake in Germany, (id=1104680, Schweriner See (Außensee)), which has serveral inner relations to ways. Some of these ways form a closed ways, so it's easy to create a polygon from it. But other ways with the relation role inner are not closed, but they form together an island in the lake. Should't those form a relation of themselves before relating them to the lake relation with an inner relation? The ways I am speaking of are 24563810 70191810 70191809 70191807 Luckily those ways form an island but if there would be another island formed by ways without relating them together it is quite difficult to recreate them properly. Is this situation an exception or is this a common situation? Frank Am 27.10.2010 11:01, schrieb Jochen Topf: On Wed, Oct 27, 2010 at 10:39:38AM +0200, Frank Broniewski wrote: During my examinations from an OSM import (osmosis pbf to postgis) I found some strange looking entries in the relation_members table. My goal is to make (GIS) polygons from the linestrings table and therefore I examined the relations a little closer. Doing a select distinct relation_id, member_role from relation_members I find these entries in the result, which I suspect to be wrong relation_id, member_role 22852;forward_stop_0 22852;forward_stop_11 22852;forward_stop_19 22852;forward_stop_9 22852;stop_1 22852;stop_10 22852;stop_12 22852;stop_13 22852;stop_14 300710;backward_stop_%n 207675;forward_stop_%n I think that numbering the member_role is not the intended way ;-) Thats a left-over from the time when relation members were not ordered. Jochen -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Relation member_roles from Osmosis import
Hello, During my examinations from an OSM import (osmosis pbf to postgis) I found some strange looking entries in the relation_members table. My goal is to make (GIS) polygons from the linestrings table and therefore I examined the relations a little closer. Doing a select distinct relation_id, member_role from relation_members I find these entries in the result, which I suspect to be wrong relation_id, member_role 22852;forward_stop_0 22852;forward_stop_11 22852;forward_stop_19 22852;forward_stop_9 22852;stop_1 22852;stop_10 22852;stop_12 22852;stop_13 22852;stop_14 300710;backward_stop_%n 207675;forward_stop_%n I think that numbering the member_role is not the intended way ;-) Well, I just wanted to mention that, because I am not sure how to correct this or which editor I could use for that Greetings Frank ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql questions
Hello list, I am a new osm2pgsql user and I want to set up a WMS server with osm data. So I loaded the planet.osm into my Postgis database and am applying the daily diffs since the creation date of the planet.osm. Applying a daily diff takes several hours to complete (5~7) and one of my questions is if this is in the standard processing time or still ok. My server is an ubuntu 8.04 amd64 machine with 6GB RAM and two harddrives, one serves the diff files and the database is on the other. Processor is an AMD64 5600+ Dualcore. I ask this question mainly because I found some oddities when examining the diff process by osm2pgsql. Postgresql and Postgis are tuned according to the oms wiki and Postgresql manual ... First is a number of postgresql processes which are idle in transaction, this usually means that there has been a transaction started with BEGIN but not commited. Right now I have 5 processes of this kind and nobody else except osm2pgsql is using the database right now. Maybe it is a flaw/problem in osm2pgsql? At the start of the diff process I get errors in my postgresql log naming that some _tmp tables do not exist and cannot be dropped: 2009-09-30 08:44:16 CEST ERROR: table planet_osm_point_tmp does not exist 2009-09-30 08:44:16 CEST STATEMENT: DROP TABLE planet_osm_point_tmp; 2009-09-30 08:44:16 CEST ERROR: table planet_osm_line_tmp does not exist 2009-09-30 08:44:16 CEST STATEMENT: DROP TABLE planet_osm_line_tmp; 2009-09-30 08:44:16 CEST ERROR: table planet_osm_polygon_tmp does not exist 2009-09-30 08:44:16 CEST STATEMENT: DROP TABLE planet_osm_polygon_tmp; 2009-09-30 08:44:16 CEST ERROR: table planet_osm_roads_tmp does not exist 2009-09-30 08:44:16 CEST STATEMENT: DROP TABLE planet_osm_roads_tmp; I suppose this is probably a leftover of an older version of osm2pgsql and it can be ignored but I am not so sure ... osm2pgsql puts out some messages during import and I wonder what storage efficiency: 11.34%, hit rate: 29.09% mean. Maybe someone can shed some light onto this for me Many thanks Frank -- Frank Broniewski Metrico s.àr.l. ( http://www.metrico.lu ) 36, rue des Romains L-5433 Niederdonven Luxembourg Fon: +352 26 74 94 28 Fax: +352 26 74 94 99 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Default scale / meters per unit ?
On Wednesday 30 September 2009 13:14:08 Jonas Donovan Hansen wrote: When importing openstreetmap with osm2pgsql and default settings (900913) - how many meters is a unit? /Jonas Whatever a unit is, that is totally dependent on scale and your renderer. The data in postgresql is vector data and has no scale. The coordinates of the reference system 900913 are metric, so you can calculate with them. The values of your bounding box are in meters, so you can calculate the current scale of your data with the corner values of the bounding box. Frank ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
