[OSM-dev] Again: Importing planet file with osmosis into PostgreSQL
Hallo, I still want to setup up an own OSM server as a mirror. Therefore I downloaded the planet file (2009/12/23). A applied all replicate changesets till 2009/12/28 in hope of a consistend planet file. This new file I tried to import into PostgreSQL database by calling bzcat 'planet-20091228.osm.bz2' | osmosis --read-xml-0.6 file='-' --log-progress --write-apidb host=pg database=xxx user=xxx password=xxx dbType=postgresql validateSchemaVersion=no Unfortunately this try failed again but with different error: INFO: Processing completion steps. 02.01.2010 18:58:41 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion SCHWERWIEGEND: Thread for task 1-read-xml-0.6 failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to load current nodes. at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:933) at org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.complete(EntityProgressLogger.java02.01.2010 18:58:41 org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion SCHWERWIEGEND: Thread for task 1-read-xml-0.6 failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to load current nodes. at org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:933) at org.openstreetmap.osmosis.core.progress.v0_6.EntityProgressLogger.complete(EntityProgressLogger.java:61) at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) at java.lang.Thread.run(Thread.java:619) Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint current_nodes_pkey1 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327) at org.postgresql.core02.01.2010 18:58:41 org.openstreetmap.osmosis.core.Osmosis main SCHWERWIEGEND: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed. at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30) Writer.java:930) ... 3 more 02.01.2010 18:58:41 org.openstreetmap.osmosis.core.Osmosis main SCHWERWIEGEND: Execution aborted. org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed. at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30) Now I had a problem with unique node ids. Who can give me a short introduction for creating a 1:1 mirror? I have no ideas any more. :( Thank you in advance! Frank -- Frank Bielig Tel: +49 33398 687848 OneStepAhead AGFax: +49 33398 687849 Research Development Mobil: +49 177 3339868 Leuschnerstr. 45 eMail: frank.bie...@onestepahead.de D-70176 Stuttgart Web: http://www.OneStepAhead.de Firma: OneStepAhead AG Firmensitz/Amtsgericht:Stuttgart, HRB 22686 Vorstand/Vorsitzender: Nihat Kücük Aufsichtsratvorsitzender: Prof. Dr. Reinhold von Schwerin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Again: Importing planet file with osmosis into PostgreSQL
Frank Bielig schrieb: Hallo, I still want to setup up an own OSM server as a mirror. Therefore I downloaded the planet file (2009/12/23). A applied all replicate changesets till 2009/12/28 in hope of a consistend planet file. Just import the planet as you got it and apply the diffs afterwise. I used the scriot located at [1] to setup the planet dump on the wikimedia toolserver and the script at [2] is running there to keep the database up with the minutely replicating diffs. A rendering server setup under debian is documented at [3]. Peter [1]http://svn.toolserver.org/svnroot/mazder/planet-import/ [2]http://svn.toolserver.org/svnroot/mazder/diff-import/ [3]http://svn.toolserver.org/svnroot/mazder/mapnik-in-a-box/docs/install-logbook ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Again: Importing planet file with osmosis into PostgreSQL
Thank you for reply. But your import script uses osm2pgsql. This script expects the simple database schema for storing data. I really want to have an 1:1 mirror using the official schema. The only tool I found for that was osmosis. Frank Hallo, I still want to setup up an own OSM server as a mirror. Therefore I downloaded the planet file (2009/12/23). A applied all replicate changesets till 2009/12/28 in hope of a consistend planet file. Just import the planet as you got it and apply the diffs afterwise. I used the scriot located at [1] to setup the planet dump on the wikimedia toolserver and the script at [2] is running there to keep the database up with the minutely replicating diffs. A rendering server setup under debian is documented at [3]. Peter [1]http://svn.toolserver.org/svnroot/mazder/planet-import/ [2]http://svn.toolserver.org/svnroot/mazder/diff-import/ [3]http://svn.toolserver.org/svnroot/mazder/mapnik-in-a-box/docs/install-logbook ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Frank Bielig Tel: +49 33398 687848 OneStepAhead AGFax: +49 33398 687849 Research Development Mobil: +49 177 3339868 Leuschnerstr. 45 eMail: frank.bie...@onestepahead.de D-70176 Stuttgart Web: http://www.OneStepAhead.de Firma: OneStepAhead AG Firmensitz/Amtsgericht:Stuttgart, HRB 22686 Vorstand/Vorsitzender: Nihat Kücük Aufsichtsratvorsitzender: Prof. Dr. Reinhold von Schwerin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Again: Importing planet file with osmosis into PostgreSQL
Frank Bielig schrieb: Thank you for reply. But your import script uses osm2pgsql. This script expects the simple database schema for storing data. I really want to have an 1:1 mirror using the official schema. The only tool I found for that was osmosis. Oh, I'm sorry, I missed this. Does it work when you skip the merge of the replicate changesets (ie with a plain planet dump). If this works, wouldn't it be a solution to apply the replicate changesets afterwards onto the database? Are you writing a log on what you were doing? This would be cool to reproduce these steps on the wikimedia toolservers. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Major improvements to MapOSMatic
Lennard l...@xs4all.nl writes: Matthias Julius wrote: Are higher resolution icons an option? Probably, but you'll need lots of different sizes for different sized areas. Really? The level of detail one wants to see is independent of the size of the area. If I want to print a 1:1 map all icons are going to be the same size. The size on the PDF is fixed anyway. The parking icon I have just looked at seems to be 16x16 pixels and because the strokes are not aligned to pixels it looks pretty blurry when one zooms to a level where they are about 5 mm tall due to anti-aliasing. It would look a lot better if they had 32x32 or even more pixels. The size on the PDF should not change, of course. Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Again: Importing planet file with osmosis into PostgreSQL
On Wed, Jan 6, 2010 at 9:48 PM, Peter Körner osm-li...@mazdermind.dewrote: Frank Bielig schrieb: Thank you for reply. But your import script uses osm2pgsql. This script expects the simple database schema for storing data. I really want to have an 1:1 mirror using the official schema. The only tool I found for that was osmosis. Oh, I'm sorry, I missed this. Does it work when you skip the merge of the replicate changesets (ie with a plain planet dump). If this works, wouldn't it be a solution to apply the replicate changesets afterwards onto the database? That will probably work. I believe that the reason for the failure is that replicate diffs can contain multiple changes per entity per change file. In other words, node 1234 may have a version 4 and 5 in the same change file. If you merge these change files into a planet file you could end up with duplicate entries for some entities. Many Osmosis tasks were never designed for full history change files. I *think* the --write-apidb-change task will work, but I haven't tried it. There is another task that is useful here, the --simplify-change task. It takes as input a full history change stream, and converts it into a delta style change stream with a maximum of one change per entity. If you are using the --apply-change task to patch a planet file you will need to feed the change stream through --simplify-change first. Again, first choice should be to import a planet and then apply replication files directly to the database, and only use --simplify-change if the --write-apidb-change task fails. Second choice is to fully patch a planet file first and use the --simply-change task in front of the --apply-change task. I haven't tested any of this out on full planet files though, only on small test files. Hope that helps, Brett PS. Frank, I may have misunderstood but you mentioned something about the simple schema. The simple schema is something else entirely. The apidb tasks support the rails port schema which is not the simple schema. The simple schema is an Osmosis specific PostGIS schema supported by the pgsql tasks. Simple refers to the fact that it has no history tables. To complete the picture, the osm2pgsql tool has its own separate PostGIS schema which has no Osmosis support. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Major improvements to MapOSMatic
Vector (SVG) symbols in mapnik are a work in progress, and not currently implemented in the main codebase. Are higher resolution icons an option? The majority of the icons used are available as SVG, see here: http://www.sjjb.co.uk/mapicons/SJJBMapIconsv0_03.tar.gz (if anyone can ever get the svg icons working in mapnik!) In the mean time larger png versions are available here: http://www.sjjb.co.uk/mapicons/SJJBMapIconsv0.03/recoloured/ If you need even bigger ones I'll happily render them, or there are scripts included with the SVG files to generate them. -- Brian ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
On Wed, Jan 6, 2010 at 12:40 PM, Brett Henderson br...@bretth.com wrote: On Mon, Dec 21, 2009 at 3:05 PM, Brett Henderson br...@bretth.com wrote: On Sat, Dec 19, 2009 at 3:06 PM, Brett Henderson br...@bretth.comwrote: Are there any objections to me disabling the old minute and hourly changesets and deleting their files from the planet server? http://planet.openstreetmap.org/minute/ http://planet.openstreetmap.org/minute-slow/ http://planet.openstreetmap.org/hourly/ They are replaced by the (hopefully) reliable transaction based replication changesets. http://planet.openstreetmap.org/minute-replicate/ http://planet.openstreetmap.org/hour-replicate/ I haven't heard any complaints about the replication changesets so I'm assuming they're working well. I'll leave the daily changesets running for now because they are running with a much longer delay and shouldn't miss data. I'm not aware of anybody still using the minute diffs. I'll disable them early in the new year. I won't delete the files straight away so they can be re-enabled if somebody suddenly screams. I hope to disable the hour diffs around the same time but will wait until Florian has moved off them first. The minute and minute-slow diffs are now disabled. I'll delete the following directories in a couple of days if nobody yells. http://planet.openstreetmap.org/minute/ http://planet.openstreetmap.org/minute-slow/ I'm keen to delete the hourly diffs as well. *Florian*, have you had a chance to move off these yet? http://planet.openstreetmap.org/hourly/ I haven't heard anything but I've disabled the hourly diffs as well. They can be re-enabled if necessary. As per the minute and minute-slow diffs, I'll delete these if nobody yells. I'll give it at least a few days though. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
2010/1/7 Brett Henderson br...@bretth.com: As per the minute and minute-slow diffs, I'll delete these if nobody yells. I'll give it at least a few days though. Not that it's going to be a problem for me, but I don't anyone will yell till the parent directories are deleted and people realise things have changed and they need to update their systems. At present just disabling new files from being generated isn't going to break anything, it will just stop updates, but that probably won't spit out errors. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
Yep, fair enough. How about I rename them so that things break ... On Thu, Jan 7, 2010 at 12:21 PM, John Smith deltafoxtrot...@gmail.comwrote: 2010/1/7 Brett Henderson br...@bretth.com: As per the minute and minute-slow diffs, I'll delete these if nobody yells. I'll give it at least a few days though. Not that it's going to be a problem for me, but I don't anyone will yell till the parent directories are deleted and people realise things have changed and they need to update their systems. At present just disabling new files from being generated isn't going to break anything, it will just stop updates, but that probably won't spit out errors. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
2010/1/7 Brett Henderson br...@bretth.com: Yep, fair enough. How about I rename them so that things break ... I doubt it matters either way, since people will need to alter their systems sooner rather than later, all I'm saying is there is no point just turning off the update it won't break anything. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
On Thu, Jan 7, 2010 at 1:04 PM, John Smith deltafoxtrot...@gmail.comwrote: 2010/1/7 Brett Henderson br...@bretth.com: Yep, fair enough. How about I rename them so that things break ... I doubt it matters either way, since people will need to alter their systems sooner rather than later, all I'm saying is there is no point just turning off the update it won't break anything. I've renamed the directories. At least this way I can re-enable them if necessary. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
