Re: [OSM-dev] 120111 Planet Node Order?
Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. So add some swap and try again :-) Simon Am 15.01.2012 02:57, schrieb Lynn W. Deffenbaugh (Mr): I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/15/2012 5:09 AM, Simon Poole wrote: current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Any pointers on the osm2pgsql command line differences for .pbf vs .bz2-compressed OSM data? With the call for people to quit accessing the OSM tile servers, this really does need to be better, or more currently, documented. I managed to get one running, but apparently there's better ways to do it now. To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s THAT is the kind of data I was looking for and haven't noticed. So your Ways are about 10% of the rate of the Nodes, and Relations are about twice as fast as the Ways. and there are very few Relations by comparison. There is definitely something different in my nodes going at 45K/s and my ways going at 0.12k/s. I've since aborted that run and restarted with some VM changes, but it looks like I've got a ways to go yet. Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. I'm also downloading the previous week's planet file just to see if my configuration goes faster on the Ways without the error, assuming I don't get the Node order error on the previous week's planet file. So add some swap and try again :-) I'll give that a go when the current run gets to the Ways and I see what the rate turns out to be. I'm at 705000k Nodes right now, so not too much longer. But I'm only getting 30.9k/s rate instead of my previous 45 so I think some of my change in the VM had less-than-desirable effects. Lynn (D) Simon Am 15.01.2012 02:57, schrieb Lynn W. Deffenbaugh (Mr): I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
Hi Simon, On 15.01.2012 11:09, Simon Poole wrote: To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Can you add your specs to the benchmarks page? http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks Your machine look similar to the Hetzner EX4 listed there but your import was seven times faster. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
The machine -is- a Hetzner EX-4. The one in use for cleanmap has some additional HW, but the planet import can be done on a stock machine in 33 hours. Simon PS: there is no big secret, the large gain is from -C 12000 Stephan Knauss o...@stephans-server.de schrieb: Hi Simon, On 15.01.2012 11:09, Simon Poole wrote: To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Can you add your specs to the benchmarks page? http://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks Your machine look similar to the Hetzner EX4 listed there but your import was seven times faster. Stephan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit Kaiten Mail gesendet.___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 15 January 2012 11:26, Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to wrote: On 1/15/2012 5:09 AM, Simon Poole wrote: To give you current numbers, I just did (10 days ago) an import on a i7 2600, 16GB box that took 30 hours with the initial import phase running at Processing: Node(1322468k 340.2k/s) Way(120291k 35.64k/s) Relation(1243830 58.30/s) parse time: 28598s Node stats: total(1322468982), max(1576326287) in 3887s Way stats: total(120291564), max(144049709) in 3375s Relation stats: total(1243833), max(1951174) in 21335s THAT is the kind of data I was looking for and haven't noticed. So your Ways are about 10% of the rate of the Nodes, and Relations are about twice as fast as the Ways. and there are very few Relations by comparison. There is definitely something different in my nodes going at 45K/s and my ways going at 0.12k/s. I've since aborted that run and restarted with some VM changes, but it looks like I've got a ways to go yet. The smaller cache will affect the ways import rate much more than nodes so this might be the result. Because the box has limited memory I did an earlier attempt with -C 8000 because I knew that it would swap a lot with -C 12000. However that ran at roughly 4k/s ways and after a couple of hours I aborted it. The good news is that you -can- import on a machine with less memory (good luck keeping up with the updates on a VM though), for example lonvia has imported on a machine with 6GB total (but with -C 12000). The error message you got is weird and may point to an issue with the export process, but it just states that the node cache is going to be less efficient space wise, so IMHO you can simply ignore that. If Lynn is running a 32-bit system or a 32-bit version of osm2pgsql, it might be impossible to allocate more than there is physical memory in the system if it's close to 2-3GB. Cheers ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page (http://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On Sun, Jan 15, 2012 at 4:24 PM, Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to wrote: On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page ( http://wiki.openstreetmap.**org/wiki/Planethttp://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. http://planet.openstreetmap.org/pbf/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
http://planet.openstreetmap.org/pbf/ Simon PS: could you fix the wiki, as it really doesn't seem to point out where the planet pbf files are. Am 15.01.2012 23:24, schrieb Lynn W. Deffenbaugh (Mr): On 1/15/2012 5:09 AM, Simon Poole wrote: Lynn, current versions of osm2pgsql can import .pbf, so you are on the right track there. Richards text is good, but it was written when planet files were a third of the current size and is just a bit dated. If osm2pgsql can't hold the nodes in it's cache it will have to retrieve them from the database and that is in my and other peoples experience at least an order of magnitude slower. Ok, any hints on where I can find a planet-level .pbf file? The only files I can find when linking from the Planet page (http://wiki.openstreetmap.org/wiki/Planet) download section are for the more traditional bz2 files (planet-date-osm.bz2). I found local area extracts in pbf format, but no planet pbf. I'm still trying to grok the options and appreciate the continuing assistance. Lynn (D) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] 120111 Planet Node Order?
Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I got everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load and going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) - http://aprsisce.wikidot.com/ - A user of OSM data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
You do have -C 12000 set? Haven't seen the error message before, but if you set the node cache smaller than necessary to store -all- nodes importing will be very slow (naturally if you don't have enough memory the machine will swap, but it sill still be faster). Simon Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr): Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I got everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load and going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) - http://aprsisce.wikidot.com/ - A user of OSM data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
Actually, the instructions I was following called for -C 2048 (http://weait.com/content/build-your-own-openstreetmap-server), but when I did that I got the following error (running in a 4GB Ubuntu 10.04 VM): Out of memory for dense node cache, reduce --cache size I had to reduce to -C 1800 to make it even run. Nodes imported nice and fast (43.6k/s), but I received the following error: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency which to my novice reading implies that there was an issue in the Planet file dealing with node ordering. And the Way import is running MUCH slower than I would have thought given what I've read. 0.12k/s is WAY less than 43.6k/s. I know ways are slower, but I figured maybe 10% of the speed worst case, not this much slower. Lynn (D) PS. I haven't seen a -C 12000 recommended anywhere. And interestingly with -C 2048 complaining , -C 1800 is still only using 2GB of the 3.7GB available according to Ubuntu's System Monitor. On 1/14/2012 6:10 PM, Simon Poole wrote: You do have -C 12000 set? Haven't seen the error message before, but if you set the node cache smaller than necessary to store -all- nodes importing will be very slow (naturally if you don't have enough memory the machine will swap, but it sill still be faster). Simon Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr): Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I got everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load and going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) - http://aprsisce.wikidot.com/ - A user of OSM data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
Believe me, if you can't allocate enough memory (you should be running a 64bit OS and have at least enough swap allocated if you don't have sufficient memory) for the cache for the import in question, it is going to be very very very very slow. Don't be confused by the Florida or whatever import running fast, that is peanuts in comparison to importing a full planet (1.4 billion nodes or so). Simon Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to schrieb: Actually, the instructions I was following called for -C 2048 (http://weait.com/content/build-your-own-openstreetmap-server), but when I did that I got the following error (running in a 4GB Ubuntu 10.04 VM): Out of memory for dense node cache, reduce --cache size I had to reduce to -C 1800 to make it even run. Nodes imported nice and fast (43.6k/s), but I received the following error: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency which to my novice reading implies that there was an issue in the Planet file dealing with node ordering. And the Way import is running MUCH slower than I would have thought given what I've read. 0.12k/s is WAY less than 43.6k/s. I know ways are slower, but I figured maybe 10% of the speed worst case, not this much slower. Lynn (D) PS. I haven't seen a -C 12000 recommended anywhere. And interestingly with -C 2048 complaining , -C 1800 is still only using 2GB of the 3.7GB available according to Ubuntu's System Monitor. On 1/14/2012 6:10 PM, Simon Poole wrote: You do have -C 12000 set? Haven't seen the error message before, but if you set the node cache smaller than necessary to store -all- nodes importing will be very slow (naturally if you don't have enough memory the machine will swap, but it sill still be faster). Simon Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr): Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I got everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load and going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) - http://aprsisce.wikidot.com/ - A user of OSM data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Sent from my Galaxy Tab with Kaiten Mail.___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail.___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
I'm a rank amateur at this, can you provide a link on how to use (or what to use instead of) osm2pgsql to import from a pbf instead of planet-120111.osm.bz2? .pbfs are not mentioned at http://weait.com/content/build-your-own-openstreetmap-server which is the best reference I've found on getting a tile server running for the planet. Lynn (D) On 1/14/2012 8:06 PM, Simon Poole wrote: Further tip: use the .pbf Files. It won't help a lot with your current issue, but is quite a bit faster. Simon -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 120111 Planet Node Order?
On 1/14/2012 7:56 PM, Simon Poole wrote: Believe me, if you can't allocate enough memory (you should be running a 64bit OS and have at least enough swap allocated if you don't have sufficient memory) for the cache for the import in question, it is going to be very very very very slow. With System Monitor reporting only 53% memory in use, I don't think memory is the reason for the slow Way import. The 1.329 billion nodes imported reasonably consistently and memory use hasn't changed with the transition from Nodes to Ways. I really think it has more to do with the out of order node error from the planet file, but google hasn't helped me find any reference to what that really means. Don't be confused by the Florida or whatever import running fast, that is peanuts in comparison to importing a full planet (1.4 billion nodes or so). Understood. I did the Florida import just to make sure the tool-chain worked. Didn't want to run the whole planet just to test it. Lynn (D) Simon Lynn W. Deffenbaugh (Mr) ldeff...@homeside.to schrieb: Actually, the instructions I was following called for -C 2048 (http://weait.com/content/build-your-own-openstreetmap-server), but when I did that I got the following error (running in a 4GB Ubuntu 10.04 VM): Out of memory for dense node cache, reduce --cache size I had to reduce to -C 1800 to make it even run. Nodes imported nice and fast (43.6k/s), but I received the following error: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency which to my novice reading implies that there was an issue in the Planet file dealing with node ordering. And the Way import is running MUCH slower than I would have thought given what I've read. 0.12k/s is WAY less than 43.6k/s. I know ways are slower, but I figured maybe 10% of the speed worst case, not this much slower. Lynn (D) PS. I haven't seen a -C 12000 recommended anywhere. And interestingly with -C 2048 complaining , -C 1800 is still only using 2GB of the 3.7GB available according to Ubuntu's System Monitor. On 1/14/2012 6:10 PM, Simon Poole wrote: You do have -C 12000 set? Haven't seen the error message before, but if you set the node cache smaller than necessary to store -all- nodes importing will be very slow (naturally if you don't have enough memory the machine will swap, but it sill still be faster). Simon Am 14.01.2012 23:16, schrieb Lynn W. Deffenbaugh (Mr): Greetings, I've been lurking in the group for a long time now and finally got motivated to try putting together my own Tile server. I go! t everything working, the whole way through mod-tile using a Florida extract as my data to keep things moving along. It was good. Then I downloaded planet-120111.osm.bz2 and started importing it yesterday. It completed the nodes and is now working on the Ways, but I received on error message: Out of order node 244067335 (33792779,7) - this will impact the cache efficiency And the Ways are processing much more slowly than I would have expected, namely: Processing: Node(1329673k 43.6k/s) Way (3282k 0.12k/s) Relation(0 0.00/s) Has anyone else imported this particular planet file and is the slow Way loading to be expected after that error? Would I be better off in scrapping this slow load a! nd going back another week to download the Planet and then apply the diffs to bring it up to date? Lynn (D) -http://aprsisce.wikidot.com/ - A user of OSM data dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Sent from my Galaxy Tab with Kaiten Mail. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
