Re: [OSM-talk] Issues with OSM import to postgis
Hello I will check that, thanks for the suggestion On 6/15/2011 6:45 PM, Walter Nordmann wrote: check the existance and/or access right to system table schema_info - has nothing to do with osmosis. regards walter - Wenn du den Wald vor lauter Bäumen nicht siehst, fälle die Bäume und du wirst sehen, dass da kein Wald ist. -- View this message in context: http://gis.638310.n2.nabble.com/Issues-with-OSM-import-to-postgis-tp6469431p6479647.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Issues with OSM import to postgis
Hello I have actually created a simple postGIS template and have created the database from it (using the postgis template), I have done this in order to be compatible with GeoServer I have not done any additional configurations as well as the database is concerned At the moment I am running the import with osm2pgsql and is running from yesterday, I expect to run for a long time since cause of the memory limitation I have used the -s option As soon as this import process is done (successful or unsuccessful) I will give a go with the proposal you have gave me Thanks! On 6/15/2011 6:51 PM, Walter Nordmann wrote: did you create the schema? A schema creation script is available in the osmosis script directory. runnig this: pgsnapshot_schema_0.6.sql - Builds the minimal schema. pgsnapshot_schema_0.6_action.sql - Adds the optional action table which allows derivative tables to be kept up to date when diffs are applied. pgsnapshot_schema_0.6_bbox.sql - Adds the optional bbox column to the way table. pgsnapshot_schema_0.6_linestring.sql - Adds the optional linestring column to the way table. may be thats the problem. - Wenn du den Wald vor lauter Bäumen nicht siehst, fälle die Bäume und du wirst sehen, dass da kein Wald ist. -- View this message in context: http://gis.638310.n2.nabble.com/Issues-with-OSM-import-to-postgis-tp6469431p6479664.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Issues with OSM import to postgis
) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) at org.codehaus.classworlds.Launcher.main(Launcher.java:31) F:\osmosis-latest\osmosis-0.39\bin The database is postGIS. Are we missing something ? On 6/14/2011 12:55 AM, Jon Burgess wrote: On Mon, 2011-06-13 at 12:04 +0200, Zolt Egete wrote: As far as other map files are concerned I have downloaded a few ones but this is the only one which I could unpack (have used pbzunzip2, bzip2, bunzip2) but all the time I have got corrupt archive messages. Also the MD5 sum of the downloaded file where not consistent with the md5 hash sum from the servers (I do not yet know the reason why) I think this is where your problem is. With the recurring checksum issues you are seeing I would suspect that something in your system is causing random data to be corruption. This could be one of a thousand different things: a faulty RAM module, hard drive cable, bad PSU, an over clocked or over heating CPU, bad PCI card etc. Or it could be a software issue, such as a bad driver in the kernel or X11. It can be really hard to identify the real cause but you could try running memtest86+ or swapping around pieces of hardware. I do not believe the files on the server are at fault. If they were then I would expect to see many more complaints. Until you can resolve the underlying data corruption issue then I don't believe it is worth you trying different combinations of options with osm2pgsql. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] server configuration
Hello A quick question I have the following HW configuration RAM: 3GB HDD: 2TB CPU: Intel(R) Pentium(R) 4 CPU 3.20GHz If I succeed to import the planet file to the posgGIS database I have on the mentioned machine, will it be sufficient for map rendering using GeoServer (I only need the WMS to server the base layer -- the planet), or will I need a better HW configuration for that ? ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Issues with OSM import to postgis
Hello I have issues with importing large (planet) OSM files my postGIS database using the osm2pgsql tool. The smaller files like some of the continents I could easily import but with large files I have problems. The hardware configuration is the following RAM: 3GB HDD: 2TB CPU: Intel(R) Pentium(R) 4 CPU 3.20GHz Software: CentOS 5.6 32 bit Postgres 8.4 PostGIS 1.5.2 Geoserver 2.1.0 Java 1.6 update 25 osm2pgsql - latest from the SVN For the import I use the following command ./osm2pgsql -U postgres -s -u -S default.style -d gisa -C 2500 /home/zsolt/tmp/planet-100127.osm The -C parameter I have altered several time and I have tried to lower the values for it but I have had no success The error I get with this particular file is the following Reading in file: /home/zsolt/tmp/planet-100127.osm Processing: Node(225520k) Way(0k) Relation(0)Entity: line 316029280: parser error : Specification mandate value for attribute version5 253 lat=33.5848027 lon=35.4919974 timestamp=2008-04-22T17:21:36Z version5 ^ Entity: line 316029280: parser error : attributes construct error 253 lat=33.5848027 lon=35.4919974 timestamp=2008-04-22T17:21:36Z version5 ^ Entity: line 316029280: parser error : Couldn't find end of Start Tag node 253 lat=33.5848027 lon=35.4919974 timestamp=2008-04-22T17:21:36Z version5 ^ /home/zsolt/tmp/planet-100127.osm : failed to parse Error occurred, cleaning up Since I have seen on several places that messages about faulty XML can be cause the memory I have expanded the swap space I have added a swap file to my system to extend the swap space so now I have 14GB of swap space but I still get the same error. What am I doing wrong ? Are there any alternatives to osm2pgsql maybe something that ? Is my HW configuration to low ? Should I go with a 64 bit system ? Any help would be greatly appreciated. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Issues with OSM import to postgis
Hello Thank you for the quick reply I will give it a try without the -u option not to use the UTF-8 sanitize and will let you know the results as soon as I have some As far as other map files are concerned I have downloaded a few ones but this is the only one which I could unpack (have used pbzunzip2, bzip2, bunzip2) but all the time I have got corrupt archive messages. Also the MD5 sum of the downloaded file where not consistent with the md5 hash sum from the servers (I do not yet know the reason why) I have downloaded the other day the latest planet file which is almost 17GB large and have started to unpack this morning with the following command pbunzip2 -d -k planet-latest.osm.bz2 pbzip2: *ERROR during decompression: -4 and the error have shown after 200GB is unpacked, but I can still see the process going on, despite for the error and now it is on 227 GB Do you think this can impose a problem with the unpacked OSM file ? On 6/13/2011 11:49 AM, M∡rtin Koppenhoefer wrote: 2011/6/13 Zolt Egetexphreaks...@gmail.com: Are there any alternatives to osm2pgsql maybe something that ? There are alternatives to osm2pgsql (e.g. imposm, osmosis) but they do not do the same thing so it depends what you want to do with your database (which scheme you want to have) which you should choose. Actually osm2pgsql should work nonetheless, have you tried another version of the planet or extract? Did you run md5 sum on your data prior to importing it to verify if your file is OK? ./osm2pgsql -U postgres -s -u -S default.style -d gisa -C 2500 /home/zsolt/tmp/planet-100127.osm Maybe the reason is the -u option? Looking at the help this should not be needed since 2007 and maybe even can cause some harm: -u|--utf8-sanitizeRepair bad UTF8 input data (present in planet dumps prior to August 2007). Adds about 10% overhead. cheers, Martin ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Issues with OSM import to postgis
Thanks Will give it a try and will let you know about the results I have got thanks one more for the quick reply On 6/13/2011 12:07 PM, M∡rtin Koppenhoefer wrote: 2011/6/13 Zolt Egetexphreaks...@gmail.com: Hello Thank you for the quick reply I will give it a try without the -u option not to use the UTF-8 sanitize and will let you know the results as soon as I have some As far as other map files are concerned I have downloaded a few ones but this is the only one which I could unpack (have used pbzunzip2, bzip2, bunzip2) but all the time I have got corrupt archive messages. Also the MD5 sum of the downloaded file where not consistent with the md5 hash sum from the servers (I do not yet know the reason why) I have downloaded the other day the latest planet file which is almost 17GB large and have started to unpack this morning with the following command pbunzip2 -d -k planet-latest.osm.bz2 pbzip2: *ERROR during decompression: -4 and the error have shown after 200GB is unpacked, but I can still see the process going on, despite for the error and now it is on 227 GB Do you think this can impose a problem with the unpacked OSM file ? yes, the md5 check should pass OK, otherwise I suspect there is a problem with your download. I suggest you try first a smaller extract to test your setup, e.g. one of geofabrik. You also don't have to unpack the file, you can pipe it with bzcat to osm2pgsql. See the wiki for details. Another option is using pbf files (binary), see the wiki. cheers, Martin ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Issues with OSM import to postgis
Hello As I have promised I have tried to import the old file without the -u option so not to use utf sanitize and I still got a problem ./osm2pgsql -U postgres -v -s -S default.style -d gisa -C 2500 /home/zsolt/tmp/planet-100127.osm osm2pgsql SVN version 0.70.5 Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point NOTICE: table planet_osm_point does not exist, skipping NOTICE: table planet_osm_point_tmp does not exist, skipping Setting up table: planet_osm_line NOTICE: table planet_osm_line does not exist, skipping NOTICE: table planet_osm_line_tmp does not exist, skipping Setting up table: planet_osm_polygon NOTICE: table planet_osm_polygon does not exist, skipping NOTICE: table planet_osm_polygon_tmp does not exist, skipping Setting up table: planet_osm_roads NOTICE: table planet_osm_roads does not exist, skipping NOTICE: table planet_osm_roads_tmp does not exist, skipping Mid: pgsql, scale=100, cache=2500MB, maxblocks=320001*8192 Setting up table: planet_osm_nodes NOTICE: table planet_osm_nodes does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index planet_osm_nodes_pkey for table planet_osm_nodes Setting up table: planet_osm_ways NOTICE: table planet_osm_ways does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index planet_osm_ways_pkey for table planet_osm_ways Setting up table: planet_osm_rels NOTICE: table planet_osm_rels does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index planet_osm_rels_pkey for table planet_osm_rels Reading in file: /home/zsolt/tmp/planet-100127.osm Processing: Node(3670k) Way(0k) Relation(0)Entity: line 21837637: parser error : Unescaped '' not allowed in attributes values 750 lon=10.4204140 timestamp=2006-12-31T06:28:45Z version=1 changeset=17 ^ Entity: line 21837637: parser error : attributes construct error 750 lon=10.4204140 timestamp=2006-12-31T06:28:45Z version=1 changeset=17 ^ Entity: line 21837637: parser error : Couldn't find end of Start Tag node 750 lon=10.4204140 timestamp=2006-12-31T06:28:45Z version=1 changeset=17 ^ /home/zsolt/tmp/planet-100127.osm : failed to parse Error occurred, cleaning up The newest file I have downloaded I could not unpack since there is an archive error I really need help cause I am on a project and I need to set the server up, I am already over my time which was planned for this task. From where can I download a good planet file ? I have tried from the links which are provided from the openstreetmap site but since now only one file have worked for me, only one I could unpack On 6/13/2011 12:07 PM, M∡rtin Koppenhoefer wrote: 2011/6/13 Zolt Egetexphreaks...@gmail.com: Hello Thank you for the quick reply I will give it a try without the -u option not to use the UTF-8 sanitize and will let you know the results as soon as I have some As far as other map files are concerned I have downloaded a few ones but this is the only one which I could unpack (have used pbzunzip2, bzip2, bunzip2) but all the time I have got corrupt archive messages. Also the MD5 sum of the downloaded file where not consistent with the md5 hash sum from the servers (I do not yet know the reason why) I have downloaded the other day the latest planet file which is almost 17GB large and have started to unpack this morning with the following command pbunzip2 -d -k planet-latest.osm.bz2 pbzip2: *ERROR during decompression: -4 and the error have shown after 200GB is unpacked, but I can still see the process going on, despite for the error and now it is on 227 GB Do you think this can impose a problem with the unpacked OSM file ? yes, the md5 check should pass OK, otherwise I suspect there is a problem with your download. I suggest you try first a smaller extract to test your setup, e.g. one of geofabrik. You also don't have to unpack the file, you can pipe it with bzcat to osm2pgsql. See the wiki for details. Another option is using pbf files (binary), see the wiki. cheers, Martin ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
