Re: [OSM-talk] Contours server (was: Re: ski pistes)
Just a quick follow-up with some numbers for disk space usage for those interested. I had a go at importing 10 metre contour lines for the whole of Eurasia into PostGIS - latitudes of 0 - 46 degrees North required about 110 gig of disk space for the Postgres table and amounted to around 105 million contour lines. (I stopped it at this point before I ran out of space :) - the SRTM3 data set extends up to 60 degrees North). 0-46N across Eurasia amounts to 3244 1x1 degree tiles. So this averages you around 35MB of disk space to import a 1x1 degree tile into PostGIS (obviously dependent on the terrain the tile covers), giving rough estimated numbers of: * Africa: 111 GB (3250 tiles) * Australia: 36 GB (1060 tiles) * Eurasia:202 GB (5902 tiles) * Islands:5 GB(141 tiles) * N America: 82 GB (2412 tiles) * S America: 62 GB (1807 tiles) * WHOLE WORLD:498 GB (14572 tiles) So with half a terabyte of disk you can import the whole lot... There is also the higher resolution SRTM1 data set covering North America - I'm not clear on how using those data would affect these numbers - probably not much since you'll probably have roughly the same number of contour lines, they will just be positioned more accurately. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server
On Mon, Mar 17, 2008 at 11:05 PM, Jon Burgess [EMAIL PROTECTED] wrote: That is correct. When the --slim option was working the entire import was IO bound and took several times longer than the RAM approach. You could try ionice but I don't think that helps control swap IO. I looked at the code the other day and it seemed rather inefficient. Fixing it will be a PITA though... Would be very nice though, I'm think of looking into it when I have time... Have a nice day, -- Martijn van Oosterhout [EMAIL PROTECTED] http://svana.org/kleptog/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server
On Tue, 18 Mar 2008, Martijn van Oosterhout wrote: I looked at the code the other day and it seemed rather inefficient. Fixing it will be a PITA though... Would be very nice though, I'm think of looking into it when I have time... I was pondering rewriting it from scratch with the aim to allow imports of the diff datasets from the start. It shouldn't be too hard (making it efficient is probably the hardest bit, but becomes less important if you're importing diffs most of the time rather than the whole planet). However, it will have to wait until I have time. :-/ - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server (was: Re: ski pistes)
Steve Hill wrote: Contours layer presented by openpistemap is simply great. Does it exist a server publishing only this layer? There is still the relief layer available, using addresses like http://srtm.in-ulm.de/layer/relief/z8/row89/8_134-89.jpg The tutorial on how to use these is here: http://www.maps-for-free.com/ Sebastian ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
[OSM-talk] Contours server (was: Re: ski pistes)
Contours layer presented by openpistemap is simply great. Does it exist a server publishing only this layer? I'm working on Viking, a desktop GPS data editor capable of rendering data on top of image downloaded on line. Having a contours layer would be usefull to prepare hicking. Thanks in advance. 2008/3/17, Andy Allan [EMAIL PROTECTED]: On Sun, Mar 16, 2008 at 10:46 PM, Rodrigo Moya [EMAIL PROTECTED] wrote: Also, where does openpistemap get the level curves from? (like in http://openpistemap.org/?lat=42.7674lon=-0.3991zoom=13layers=B000 ) Both openpistemap and the cycle map use contours from derived from a NASA dataset called SRTM - see http://wiki.openstreetmap.org/index.php/Contours for details of how we do it. -- Guilhem BONNEFILLE -=- #UIN: 15146515 JID: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED] -=- mailto:[EMAIL PROTECTED] -=- http://nathguil.free.fr/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
[OSM-talk] Contours server (was: Re: ski pistes)
Contours layer presented by openpistemap is simply great. Does it exist a server publishing only this layer? I'm not sure how that would work - you really need the contours data set to be the bottom layer, with the normal layers on top of that (is it possible to ask OpenLayers to render the OSM Mapnik tiles on top of another layer? I suspect not since the tiles aren't transparent...) OpenPisteMap only has contours data for specific areas because the data set is pretty enormous (I did try converting the whole SRTM3 data set to shapefiles - I gave up by the time it had done about 3/4 of Eurasia and sucked up over 100 gig of disk space. I haven't tried pulling the whole data set into PostGIS yet - I wonder if that is any more efficient). Another consideration is that the contours tiles have to be tailored to the map type a bit - for example, the cycle map renders the contours closer together (i.e. it introduces each set of contour lines at lower zoom levels than the piste map). The mountainous terrain of the pistes results in unreadably close contour lines if you just try to apply the cycle map styles. I'm working on Viking, a desktop GPS data editor capable of rendering data on top of image downloaded on line. Having a contours layer would be usefull to prepare hicking. For that kind of application, it might be better to implement an API to retrieve the contour shapes from an online database and render them on-the-fly in the application. That would allow the user to tune the settings to suit the terrain. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server (was: Re: ski pistes)
On Mon, Mar 17, 2008 at 4:36 PM, Steve Hill [EMAIL PROTECTED] wrote: Contours layer presented by openpistemap is simply great. Does it exist a server publishing only this layer? I'm not sure how that would work - you really need the contours data set to be the bottom layer, with the normal layers on top of that (is it possible to ask OpenLayers to render the OSM Mapnik tiles on top of another layer? I suspect not since the tiles aren't transparent...) OpenPisteMap only has contours data for specific areas because the data set is pretty enormous (I did try converting the whole SRTM3 data set to shapefiles - I gave up by the time it had done about 3/4 of Eurasia and sucked up over 100 gig of disk space. I haven't tried pulling the whole data set into PostGIS yet - I wonder if that is any more efficient). Another consideration is that the contours tiles have to be tailored to the map type a bit - for example, the cycle map renders the contours closer together (i.e. it introduces each set of contour lines at lower zoom levels than the piste map). The mountainous terrain of the pistes results in unreadably close contour lines if you just try to apply the cycle map styles. There's also the amount of time it would take to render the tiles. It takes over 15 hours to render the contours used on the cycle map, and all things considered that covers a very small % of the planet surface at any kind of decent zoom level. It's a slow enough process that render on demand doesn't work so well either. So any such server would need a few TB of disk space or be a come back in 24 hours after request job, or both. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server (was: Re: ski pistes)
On Mon, 17 Mar 2008, Dave Stubbs wrote: There's also the amount of time it would take to render the tiles. It takes over 15 hours to render the contours used on the cycle map, and all things considered that covers a very small % of the planet surface at any kind of decent zoom level. It's a slow enough process that render on demand doesn't work so well either. So any such server would need a few TB of disk space or be a come back in 24 hours after request job, or both. I've been quite successful with doing render on demand - tiles which have never been rendered are pushed into a render queue and the HTTP connection is left idling. If the tile is rendered within 30 seconds the user gets it sent over the HTTP connection as soon as possible, otherwise they get a 404 and the tile remains in the queue until it is rendered. The render-server process is currently set to be able to render up to 4 tiles in parallel and also copes well with requests for tiles that are already queued or being rendered. That said, it is running on a pretty meaty server (an 8 core 1.86GHz Xeon E5320), and it isn't very busy (serving 15k - 36k tiles per day over the past 5 days). I suspect that storing the contours in PostGIS helps quite a bit too. My main performance issues revolve around importing the planet OSM file (osm2pgsql uses up crazy amounts of RAM (or rather, swap, in my case) - there is the -s option, but that is currently broken...). What I'm really after is a way to import the daily diffs into the existing PostGIS DB. - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server (was: Re: ski pistes)
On Mon, Mar 17, 2008 at 5:43 PM, Steve Hill [EMAIL PROTECTED] wrote: On Mon, 17 Mar 2008, Dave Stubbs wrote: There's also the amount of time it would take to render the tiles. It takes over 15 hours to render the contours used on the cycle map, and all things considered that covers a very small % of the planet surface at any kind of decent zoom level. It's a slow enough process that render on demand doesn't work so well either. So any such server would need a few TB of disk space or be a come back in 24 hours after request job, or both. I've been quite successful with doing render on demand - tiles which have never been rendered are pushed into a render queue and the HTTP connection is left idling. If the tile is rendered within 30 seconds the user gets it sent over the HTTP connection as soon as possible, otherwise they get a 404 and the tile remains in the queue until it is rendered. The render-server process is currently set to be able to render up to 4 tiles in parallel and also copes well with requests for tiles that are already queued or being rendered. That said, it is running on a pretty meaty server (an 8 core 1.86GHz Xeon E5320), and it isn't very busy (serving 15k - 36k tiles per day over the past 5 days). I suspect that storing the contours in PostGIS helps quite a bit too. Yeah, on the cycle map it manages 12 tiles/sec with 4 cores. I was mainly comparing with standard OSM tiles which I can get nearly 200 tiles/sec. I'm assuming you can get about 20-30 tiles/sec, probably higher with your wider spaced contours. So it'll probably work as long as no one starts trying to scrape the tiles... My main performance issues revolve around importing the planet OSM file (osm2pgsql uses up crazy amounts of RAM (or rather, swap, in my case) - there is the -s option, but that is currently broken...). What I'm really after is a way to import the daily diffs into the existing PostGIS DB. That would hopefully speed up the weekly import I do quite a lot. Not a very trivial change though. I wouldn't recommend the -m option even if it did work... it's a LOT slower -- it's faster to just let it use swap, but it might be worth it if you could then apply updates. Personally I now have 8GB RAM so don't care so much as long as the memory requirements stay below 4GB (it's only on a 32bit OS). ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server
Dave Stubbs wrote: it's faster to just let it use swap, I I/O load hits my server pretty hard. Trying to do anything else while that's happening is quite painful. :-/ Doing it without the heavy I/O load is preferable, even if it takes a couple of days - it can just trundle away in the background. Personally I now have 8GB RAM so don't care so much as long as the memory requirements stay below 4GB (it's only on a 32bit OS). Physically I have enough memory, but it is allocated to other virtual machines. One option might be for me to juggle the memory allocations between the VMs when doing an import. -- - Steve xmpp:[EMAIL PROTECTED] sip:[EMAIL PROTECTED] http://www.nexusuk.org/ Servatis a periculum, servatis a maleficum - Whisper, Evanescence ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server
On Mon, Mar 17, 2008 at 8:00 PM, Steve Hill [EMAIL PROTECTED] wrote: Dave Stubbs wrote: it's faster to just let it use swap, I I/O load hits my server pretty hard. Trying to do anything else while that's happening is quite painful. :-/ Doing it without the heavy I/O load is preferable, even if it takes a couple of days - it can just trundle away in the background. Yeah, but the postgres middle option just puts the nodes in a postgres table and then does selects to find them. If you don't have the RAM to cache the DB table you probably will have a similar i/o load just over a longer period... you might have a better chance of deprioritising and containing that though. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Contours server
On Mon, 2008-03-17 at 20:28 +, Dave Stubbs wrote: On Mon, Mar 17, 2008 at 8:00 PM, Steve Hill [EMAIL PROTECTED] wrote: Dave Stubbs wrote: it's faster to just let it use swap, I I/O load hits my server pretty hard. Trying to do anything else while that's happening is quite painful. :-/ Doing it without the heavy I/O load is preferable, even if it takes a couple of days - it can just trundle away in the background. Yeah, but the postgres middle option just puts the nodes in a postgres table and then does selects to find them. If you don't have the RAM to cache the DB table you probably will have a similar i/o load just over a longer period... That is correct. When the --slim option was working the entire import was IO bound and took several times longer than the RAM approach. You could try ionice but I don't think that helps control swap IO. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
