Re: [Potlatch-dev] Potlatch 2.3
On 26 November 2011 12:32, NopMap ekkeh...@gmx.de wrote: I played at migrating to P2.3 again and ran into a few problems. Hi Nop, First off, two apologies. P2.3 was released with a major upgrade to the flex libraries that pretty much broke i18n completely, although due to the seemingly never-ending saga of me being unable to reproduce problems meant that it wasn't explained clearly in the release announcement. http://lists.openstreetmap.org/pipermail/potlatch-dev/2011-September/001220.html Second apology is in not getting back to you at the weekend, since I knew I was going to be doing a lot of work around i18n and hoping to fix things. See the other thread for more on this. Now, to your points: Is there a full set or browsable instance of the locales somewhere? See http://random.dev.openstreetmap.org/potlatch2/ which is a continually updated build of potlatch2 with directory browsing enabled. I've also added a trac ticket http://trac.openstreetmap.org/ticket/4112 to provide bundled builds in order to cut down on these hassles. 2. setting the locale. How do you do that in P2.3? The parameter format has changed, it used to be fo.addVariable(locale, de_DE); but what is it now? Try again with 2.3-112 or later, it should be working, thanks to work by Hiroshi. 3. Config files. There was a hint earlier in this thread that the config file format has slightly changed. At first glance, my unchanged files appear to work. Can you give me more information where adjustments are necessary? I'm not sure which config files you are referring to, but perhaps Richard can describe more the subcategory panels work. 4. Authorization. P2.3 asked for a fresh authorization key. What changes cause it to do that? That request seems to pop up from time to time, I found that I had about 10 authrizations in my osm account. The question is also asked occasionally by users. That will happen when either the flash cookies are cleared, or the .swf is served from a different URL. Perhaps your users have browser privacy plugins that are clearing saved flash cookies? When i executed the authorization, I had firebug running as I used it to spy on P2. This caused Firefox to always crash after the authorization window opened. After disabling firebug it worked. Is this a known problem? Haven't heard of that before. Cheers, Andy ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] [OpenStreetMap] #3746: Vietnamese translation of Potlatch 2
#3746: Vietnamese translation of Potlatch 2 -+-- Reporter: Minh Nguyen | Owner: potlatch-dev@… Type: enhancement | Status: new Priority: major| Milestone: Component: potlatch2| Version: 2.0 Keywords: l10n | -+-- Comment(by Minh Nguyen): I’ve updated the translation. See [https://github.com/openstreetmap/potlatch2/pull/4 pull request 4] at !GitHub. -- Ticket URL: https://trac.openstreetmap.org/ticket/3746#comment:3 OpenStreetMap http://www.openstreetmap.org/ OpenStreetMap is a free editable map of the whole world ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] [OpenStreetMap] #3746: Vietnamese translation of Potlatch 2
#3746: Vietnamese translation of Potlatch 2 -+-- Reporter: Minh Nguyen | Owner: potlatch-dev@… Type: enhancement | Status: new Priority: major| Milestone: Component: potlatch2| Version: 2.0 Keywords: l10n | -+-- Comment(by Richard): Thanks! Could you send it to the project repository at http://github.com/systemed rather than the one for just the osm.org instance? (Last time I took a pull request over from git.openstreetmap.org the user information got lost somewhere along the way.) -- Ticket URL: https://trac.openstreetmap.org/ticket/3746#comment:4 OpenStreetMap http://www.openstreetmap.org/ OpenStreetMap is a free editable map of the whole world ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [Potlatch-dev] [OpenStreetMap] #3746: Vietnamese translation of Potlatch 2
#3746: Vietnamese translation of Potlatch 2 -+-- Reporter: Minh Nguyen | Owner: potlatch-dev@… Type: enhancement | Status: new Priority: major| Milestone: Component: potlatch2| Version: 2.0 Keywords: l10n | -+-- Comment(by Minh Nguyen): [https://github.com/systemed/potlatch2/pull/21 Done]. -- Ticket URL: https://trac.openstreetmap.org/ticket/3746#comment:5 OpenStreetMap http://www.openstreetmap.org/ OpenStreetMap is a free editable map of the whole world ___ Potlatch-dev mailing list Potlatch-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Frederik Ramm wrote: Hi, On 11/30/2011 07:25 AM, Ákos Maróy wrote: What I've tried so far is importing the current planet-XXX.osm.bz2 file into PostGIS via osm2pgsl, which I have used with the --slim option, as without it the memory load exceeded the 16GB memory I had in my system significantly (it was using about 26GB when I shut the process down). A planet import without --slim takes more than 48 GB of RAM. this way, it took about a week to import the current planet.osm file, on a Xeon 4 core system running 64 bit Linux. Have you got the latest osm2pgsql checked out? Kai has made some improvements in the last few weeks (albeit someone else just now reported that this made things slower for them - my results were rather good with the new code). If you do not need differential updates, newer versions of osm2pgsql also have a somewhat experimental --drop command line flag that avoids creating some indexes and cleans up temporary data after a --slim import, leaving you with about the same that you would have when you do an import without --slim. After this succeeded, I wanted to try to replicate this database, so I created a pg_dump using the -Fc switch This is a bad idea because a significant amount of osm2pgsql import time is spent building indexes, and these are not dumped, so after restoring the data the same time is spent again. The only efficient way to copy an imported planet between two systems is to copy the raw postgresql data files directly, and this only works reliably if postgres, postgis and geos have identical versions on both systems (and of course both systems have the same architecture). It can be slow but it is not always a bad idea. I have a very lean Linux virtual server with about 700 MB of memory and it is very slow to import even Finnish excerpt with osm2pgsql. In addition import tends to fail totally about every second time. However, the virtual box has no troubles if I run osm2pgsql at home, upload PG dump files into server and run restore there. Even running restore from my home computer with pgAdmin III through a 1 Mb line upwards is faster than making import with osm2pgsql on the Linux box. Ogr2ogr from the home PostGIS to remote PostGIS has also worked reliably for me. Finnish data may be heavy for osm2pgsql because there are rather many polygons (407625 vs. 452944 linear features) to construct, including big lake and land use relations. Perhaps native support for areas in OSM will make is faster. -Jukka Rahkonen- Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Incomplete diffs?
On 21 November 2011 06:06, Peter Körner osm-li...@mazdermind.de wrote: Am 05.11.2011 22:33, schrieb mar...@gmx.eu: Hi Erik, thanks for your help. The missing node seems to be available via minutely and hourly diff files but NOT via the daily file. Meanwhile I found an explanation in the Wiki: http://wiki.openstreetmap.org/wiki/Planet.osm/diffs I've made a number of updates to the table on that page to (hopefully) better explain the various extracts available. I've also just created a day-replicate job to complement the existing minute and hour jobs. Now is probably a good time to consider the daily diffs deprecated in favour of the day-replicate diffs. Brett ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [Potlatch-dev] Potlatch 2.3
Andy wrote: I'm not sure which config files you are referring to, but perhaps Richard can describe more the subcategory panels work. Sure. The change is that there's now an optional 'subcategory' attribute in input. So you can say: input type=freetext presence=always category=Naming subcategory=Little-used names name=International Ref key=int_ref description=International reference number/ If you do that, then the input will appear in a subgroup 'box', which opens in a disclosure triangle. You can see this on P2 on osm.org. This means that little-used tags can still be offered via the editing interface, but don't clutter up the main UI unless the user expressly chooses to see them. You absolutely don't have to use these, they're completely optional. Apart from that everything should just work. You'll see that the tab-bar doesn't scroll any more (because it was buggy and broke every five minutes), but instead, we use icons to save space. At present the icons are hard-coded but that's a todo for me some day! cheers Richard ___ Potlatch-dev mailing list potlatch-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/potlatch-dev
[OSM-dev] 3D OSM Dev weekend 03/2012
Hi friends, as to a lot of developments on 3D during the past months there were various requests on doing a meeting focused on 3D. So what tags are in the wild and how are they interpreted? What about shared services between all of the single 3d tools in the OSM ecosphere? http://forum.openstreetmap.org/viewtopic.php?id=14540 Next year in March we will do a weekend on the 3D aspects of OSM in Germany and we would be happy about anybody that is interested in the modelling of 3D tagged OSM objects and who has applications to work on this data. Please join the discussion on the forums and review/add topics of your interests. Maybe we can add a video conference to link everybody who can't be present at the meeting. bye Matthias (user:!i!) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On 01/-10/-28163 12:59 PM, Jukka Rahkonen wrote: Frederik Ramm wrote: [...] After this succeeded, I wanted to try to replicate this database, so I created a pg_dump using the -Fc switch This is a bad idea because a significant amount of osm2pgsql import time is spent building indexes, and these are not dumped, so after restoring the data the same time is spent again. The only efficient way to copy an imported planet between two systems is to copy the raw postgresql data files directly, and this only works reliably if postgres, postgis and geos have identical versions on both systems (and of course both systems have the same architecture). Yes, the indexing stage typically takes about half or more of the total time. On a planet import I just did, it took 14 hours for writing of data and another 41 hours for the indexing. I have not tried how long it would take to restore a dump, but I suspect in that case not much less. It can be slow but it is not always a bad idea. I have a very lean Linux virtual server with about 700 MB of memory and it is very slow to import even Finnish excerpt with osm2pgsql. In addition import tends to fail totally about every second time. However, the virtual box has no troubles if I run osm2pgsql at home, upload PG dump files into server and run restore there. Even running restore from my home computer with pgAdmin III through a 1 Mb line upwards is faster than making import with osm2pgsql on the Linux box. Ogr2ogr from the home PostGIS to remote PostGIS has also worked reliably for me. Can you describe a bit more of what your setup is, and where it fails? Which version of Postgresql, PostGis and Osm2pgsql? 32bit or 64bit? 700Mb of ram really is very low, but I would like osm2pgsql to work with little resources even if very slow. Finnish data may be heavy for osm2pgsql because there are rather many polygons (407625 vs. 452944 linear features) to construct, including big lake and land use relations. Perhaps native support for areas in OSM will make is faster. Not sure how problematic those polygons are, but it doesn't seem to bad to me. I have just tried a Finnish import and it only took 10 minutes for the import. 5 minutes for writing of data and 5 minutes for the indexing. OK, my laptop sounds like it is more powerful than your server with 2.4GHz dual-core and 4 Gb ram, but osm2pgsql only used about 150Mb of ram. Non slim mode took overall less than 4 minutes, but that did take a little under 2 Gb of Ram, so wouldn't be feasible on your server. Kai -Jukka Rahkonen- Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Kai Krueger wrote: On 01/-10/-28163 12:59 PM, Jukka Rahkonen wrote: It can be slow but it is not always a bad idea. I have a very lean Linux virtual server with about 700 MB of memory and it is very slow to import even Finnish excerpt with osm2pgsql. In addition import tends to fail totally about every second time. However, the virtual box has no troubles if I run osm2pgsql at home, upload PG dump files into server and run restore there. Even running restore from my home computer with pgAdmin III through a 1 Mb line upwards is faster than making import with osm2pgsql on the Linux box. Ogr2ogr from the home PostGIS to remote PostGIS has also worked reliably for me. Can you describe a bit more of what your setup is, and where it fails? Which version of Postgresql, PostGis and Osm2pgsql? 32bit or 64bit? 700Mb of ram really is very low, but I would like osm2pgsql to work with little resources even if very slow. For me it takes many hours with the Finnish dataset and if it fails it happens in some Going over pending ways phase. I will need to make some further tests some day so I can give you better information. I know that it is Ubuntu 10.04 and PostgreSQL 9.0 with PostGIS is running on the same machine. I believe it is 32-bit and osm2pgsql is half an year old. I have not used it many times because of failures. My Windows laptop in not much faster but it does not fail. Osm2pgsql version is the newest that exists for Windows, it is rather old (April 9, 2010). Your 20 minutes total time feels amazing for me. But my server is very basic and I do not wait very much for 27 euros per month. But when the data are in it works well as a WFS server. -Jukka- Kai ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Jukka Rahkonen-2 wrote [...] For me it takes many hours with the Finnish dataset and if it fails it happens in some Going over pending ways phase. I will need to make some further tests some day so I can give you better information. If it is at the very beginning of the Going over pending ways (before it prints out any other information) then that is probably during the sql querry loading all of the pending ways into memory. It was mentioned in another thread that this can be problematic on low memory systems. One could possibly recode it to use postgres cursors to not have to load the full result set into memory, but that would break the new parallelisation stage which relies on having the full result set in memory before forking the helper processes. I'll have to think about if there are solutions to this. Jukka Rahkonen-2 wrote I know that it is Ubuntu 10.04 and PostgreSQL 9.0 with PostGIS is running on the same machine. I believe it is 32-bit and osm2pgsql is half an year old. I have not used it many times because of failures. Half a year old osm2pgsql is probably before the introductions of my improvements which were in October. Osm2pgsql used to be incredibly inefficient with memory for its node cache on extracts, as it was optimised for full planet imports. So it both used a lot of ram and on memory constrained systems didn't get a good cache hit ratio which makes it very very slow to import. If you can it would therefore be good if you could try a new version of osm2pgsql. You might need to set the --cache-strategy to optimized or sparse, as I think it defaults to the old inefficient behaviour on 32 bit compiles. 64 bit compiles should use the optimized strategy as default. Jukka Rahkonen-2 wrote My Windows laptop in not much faster but it does not fail. Osm2pgsql version is the newest that exists for Windows, it is rather old (April 9, 2010). Ah Windows... Unfortunately the windows osm2pgsql is indeed ancient. Can anyone try and compile a recent osm2pgsql for windows? Or does anyone know how to compile it? I could set up a windows VM to try and build it, but currently I don't even know where to get a C compiler or the autoconf/autobuild tools for windows, so I am not sure how successful I would be to build it. But it would be very helpful to have a modern osm2pgsql for windows as that would allow more people to play with rendering their own tiles. Jukka Rahkonen-2 wrote Your 20 minutes total time feels amazing for me. But my server is very basic and I do not wait very much for 27 euros per month. But when the data are in it works well as a WFS server. Yes, I'd like the whole rendering stack to become more lightweight, at least for small extracts, so that more people can play with rendering their own tiles, either on their home laptop/desktop or on fairly cheap servers. The huge resources often required for working with osm data is imho a big barrier to more people using it and thus needs to be addressed. My recent improvements to osm2pgsql together with the easy to install ubuntu packages have hopefully helped somewhat in this respect, but there is still much more to do... Kai -- View this message in context: http://gis.638310.n2.nabble.com/speeding-up-loading-an-OSM-dump-into-PostGIS-tp7045762p7047301.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSMJS - parse argv
Hi, Against my better judgement I am wondering whether osmjs can pass argvals to the JS script. I need to process a number of OSM data files and I want the output files generated by OSMJS to have unique names based on the input file. Martijn -- martijn van exel geospatial omnivore 1109 1st ave #2 salt lake city, ut 84103 801-550-5815 http://oegeo.wordpress.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSMJS - parse argv
Hi! On Wed, Nov 30, 2011 at 09:03:45AM -0700, Martijn van Exel wrote: Against my better judgement I am wondering whether osmjs can pass argvals to the JS script. I need to process a number of OSM data files and I want the output files generated by OSMJS to have unique names based on the input file. All extra arguments from the command line are available in the global Javascript array named argv. Jochen -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSMJS - parse argv
On Wed, Nov 30, 2011 at 9:16 AM, Jochen Topf joc...@remote.org wrote: Hi! On Wed, Nov 30, 2011 at 09:03:45AM -0700, Martijn van Exel wrote: Against my better judgement I am wondering whether osmjs can pass argvals to the JS script. I need to process a number of OSM data files and I want the output files generated by OSMJS to have unique names based on the input file. All extra arguments from the command line are available in the global Javascript array named argv. Thanks, you just saved me some boring work. Martijn -- martijn van exel geospatial omnivore 1109 1st ave #2 salt lake city, ut 84103 801-550-5815 http://oegeo.wordpress.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Dear All, Thank your the detailed responses. Indeed, I'm using the pg_dump pg_restore method to transfer a database to a system running a different version of posgresql. my hope is that this is faster than loading everything from scratch using osm2pgsql. so it seems I'll just wait see :) Akos ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [osmosis-dev] some tests failed on Gentoo (may be because of wrong CLASSPATH)
Hi Dmytro, Testcase: testSimple took 0.005 sec Caused an ERROR org.openstreetmap.osmosis.testutil.TestDataUtilities.newFile()Ljava/io/File; java.lang.NoSuchMethodError: org.openstreetmap.osmosis.testutil.TestDataUtilities.newFile()Ljava/io/File; at org.openstreetmap.osmosis.testutil.TestDataUtilities.createDataFile(TestDataUtilities.java:54) at org.openstreetmap.osmosis.xml.v0_6.XmlChangeReaderWriterTest.testSimple(XmlChangeReaderWriterTest.java:36) hmm, that is interesting... In your original post, you said you were trying to build 0.39. However, org.openstreetmap.osmosis.testutil was introduced just over a month ago, well after 0.39 was released. May it be that $ git clone git://... osmosis $ cd osmosis git checkout 0.39 creates a mixup of old and new sources which - naturally - don't mix well? It's doing that here on my machine, somehow, and I had to be very careful in getting a 0.39 tree to get it to build _at all_. After I managed to get a clean 0.39, however, it built and passed all trees. Hope that helps Igor ___ osmosis-dev mailing list osmosis-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[OSM-dev] Change in wtfe.gryph.de Quick History Service API
Hi, the server at wtfe.gryph.de currently has information about the history of all objects. In the near future, I plan to change that and remove all information about users who have already agreed to the Contributor Terms, so that only problematic information remains. This frees up some resources on the machine which I will then use to treat harmless edits differently. At the moment, wtfe-based license change plugins/views will always flag an object as problematic even if the edit didn't change the object or didn't add information. For example, we have a number of near-vandalism edits by non-agreing user hasse_osm_korinthenkacker. If you look at this way http://www.openstreetmap.org/browse/way/24908221/history it was once touched by this user to remove a source=survey tag and is otherwise a perfectly normal OSM way - a way that will show up as problematic in current license change views, which is unnecessary. In the future, such edits will still be pointed out by wtfe.gryph.de but will be flagged as harmless so that mappers do not need to concern themselves with remapping such ways. The old wtfe.gryph.de interface, http://wtfe.gryph.de/api/0.6/userlist?... will be phased out, and replaced with a similar call http://wtfe.gryph.de/api/0.6/problems?... At the moment, both calls are available, and the new call does not yet flag harmless edits - all edits are still in the normal category. But I will make up some rules to detect harmless edits and flag them in the coming days. See details on http://wtfe.gryph.de/ Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Change in wtfe.gryph.de Quick History Service API
Hi, forgot one thing: On 11/30/11 18:36, Frederik Ramm wrote: In the future, such edits will still be pointed out by wtfe.gryph.de but will be flagged as harmless so that mappers do not need to concern themselves with remapping such ways. For anyone displaying problems on a map, I would suggest the following colour coding: gray or unmarked for severity=none yellow or unmarked for severity=harmless orange for severity=normal, version=other (indicates revert but no complete loss of object) red for severity=normal, version=first (indicates loss of object) Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slow slim import
Petr Morávek [Xificurk] napsal(a): Hello, I've just upgraded osm2pgsql to latest SVN version (previously I was running a version from August, not sure which revision exactly) and I'm experiencing extreme slow-down in the phase Going over pending ways. The processing is going at rate ~ 0.1k/s. With the old version the import of a relatively small extract took approximately 1.5 hour, now I'm at 3 hours and the import is still far from completed. I have postgresql 9.1 and my import command looks like this: osm2pgsql --slim -d osm -U osm -p osm --hstore -C 1500 -S ./import.style -r pbf ./data.osm.pbf As far as I can tell, the bottle-neck is IO due to UPDATE command. I have also tried running the import with options '--cache-strategy sparse' and/or '--disable-parallel-indexing', but as far as I can tell, it has no effect on this phase. Any ideas where the problem could be or how to debug this? Best regards, Petr Morávek aka Xificurk An update: I've narrowed it down - the problem appeared in rev26893 (Parralellize pending ways / relations). I've tested multiple revisions with a really small extract: 13MB pbf (1450k nodes, 178k ways, 2250 rels), I used the command stated above. rev26892: import time ~1min rev26893: import time ~22min (the phase going over pending ways went at rate 0.11k/s) I've got installed: postgresql-9.1.1 postgis-1.5.3 geos-3.2.2 The weird thing is that such a small extract should fit into RAM, so I have no idea why is the processing so slow. Petr Morávek aka Xificurk attachment: xificurk.vcf signature.asc Description: OpenPGP digital signature ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On Miércoles, 30 de Noviembre de 2011 07:25:25 Ákos Maróy escribió: I wonder what ways are there to speed up importing an OSM planet file into a PostGIS database? What I've tried so far is importing the current planet-XXX.osm.bz2 file into PostGIS via osm2pgsl Imposm, man, imposm. Takes 48 hours to import all the data I need from a full planet in a decent server. -- -- Iván Sánchez Ortega i...@sanchezortega.es i...@geonerd.org For office use only. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slow slim import
Hello, Hi, sorry, no solutions ;-( but I ran into the exact same trouble rev26892: import time ~1min rev26893: import time ~22min (the phase going over pending ways went at rate 0.11k/s) Nice catch. revision 26892 ran that phase for the same extract at 6.36k/s instead of 0.14k/s I've got installed: postgresql-9.1.1 postgis-1.5.3 geos-3.2.2 Same here, not helping to find if it's postgresql related However, interestingly, I have access to another server with the same postgresql/postgis version which does not exibit the problem (I copied the osm2pgsql binary to be sure) I'll check if I can find the config diff to explain that But thanks for finding the unaffected osm2pgsql version ;-) -- sly -- View this message in context: http://gis.638310.n2.nabble.com/osm2pgsql-slow-slim-import-tp7044819p7049116.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql slow slim import
sylvain letuffe wrote Nice catch. revision 26892 ran that phase for the same extract at 6.36k/s instead of 0.14k/s 6.4k/s is much more what I would expect from such a small extract. So yes something is wrong there, but I haven't seen that behavior before and I can't currently reproduce it. Can you try if using the switch --number-processes=1? That should disable any parallelisation and make it more or less the same as before. (Although there is a slight difference in transaction handling) Otherwise, can you check if you are getting 100% hit ratio from the node cache? sylvain letuffe wrote I've got installed: postgresql-9.1.1 postgis-1.5.3 geos-3.2.2 Same here, not helping to find if it's postgresql related However, interestingly, I have access to another server with the same postgresql/postgis version which does not exibit the problem (I copied the osm2pgsql binary to be sure) I'll check if I can find the config diff to explain that But thanks for finding the unaffected osm2pgsql version ;-) -- sly -- View this message in context: http://gis.638310.n2.nabble.com/osm2pgsql-slow-slim-import-tp7044819p7049219.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql update
Frederik Ramm wrote I found out that the culprit is in the multipolygon code, where after finding out that an one-way outer ring is tagged the same as the multipolgon relation itself, a delete_way_from_output is issued, presumably to remove that already-generated ring. This leads to a DELETE from table where osm_id=id which requires a table scan because of lack of primary keys. I have now disabled this for --slim --drop mode (the change will not affect normal --slim mode), but have to investigate further - this will likely create some extra areas for outer rings, but since it doesn't have these indexes, non-slim mode should exhibit the same behaviour. Is anyone aware of multipolygon handling not working right when not using --slim? We might have to (re)introduce the primary key for osm_id at least on the polygon table to allow this deletion of duplicate areas. I think this might be fine to do and in fact it might be superfluous in the normal slim mode import as well, although it is needed during diff processing. During the way processing stage of the import, osm2pgsql only writes those ways to the output tables that, according to the style file, are not polygons. Those ways that are polygons are only written to the slim mode tables and marked as pending to be dealt with during the going over pending ways phase, precisely because they might have to be deleted again should they be part of a multi-polygon. So the DELETE from table where osm_id=id should always turn up blank during a fresh import. (if I am not mistaken) Kai -- View this message in context: http://gis.638310.n2.nabble.com/osm2pgsql-update-tp7027563p7049376.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Really? What is a decent server ? Yves -- Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté. Iván Sánchez Ortega i...@sanchezortega.es a écrit : On Miércoles, 30 de Noviembre de 2011 07:25:25 Ákos Maróy escribió: I wonder what ways are there to speed up importing an OSM planet file into a PostGIS database? What I've tried so far is importing the current planet-XXX.osm.bz2 file into PostGIS via osm2pgsl Imposm, man, imposm. Takes 48 hours to import all the data I need from a full planet in a decent server. -- _ Iván Sánchez Ortega i...@sanchezortega.es i...@geonerd.org For office use only. _ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
