Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
I suspect I didn't with tilecache (not sure if it's possible), but I do now with mod_tile. Martijn van Oosterhout wrote: A bit late and maybe a stupid question but: do you have metatiling turned on. Rendering is going to really suck without it. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
On Mon, Jun 16, 2008 at 8:20 AM, Brett Henderson [EMAIL PROTECTED] wrote: I suspect I didn't with tilecache (not sure if it's possible), but I do now with mod_tile. It's definitly possible, otherwise it would swiftly go awol like you saw. The config file has quite a few comments, i'm surprised you missed it. Have a nce day, -- Martijn van Oosterhout [EMAIL PROTECTED] http://svana.org/kleptog/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
A bit late and maybe a stupid question but: do you have metatiling turned on. Rendering is going to really suck without it. On Mon, Jun 9, 2008 at 8:08 AM, Brett Henderson [EMAIL PROTECTED] wrote: My tilecache.cfg file contains the following entry (I assume I should re-enable tms_type but have been playing with different settings): [osm] type=Mapnik mapfile=/home/tilecache/mapnik/osm.xml spherical_mercator=true #tms_type=google That significantly reduces the number if tiles it tries to render smultaneously Have a nice day, -- Martijn van Oosterhout [EMAIL PROTECTED] http://svana.org/kleptog/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
Yes, it was always designed for use with openlayers tiles Thanks to Sebastian, who fixed the slippy.html today to display pyrender tiles (typically pyrender is run as its own application, and it starts serving tiles on http://localhost:1280/layer/z/x/y.png ) http://svn.openstreetmap.org/applications/rendering/pyrender/ There's actually a screenshot of Sebastian's new rendering rules, looking quite nice given the minimal amount of data its allowed to use: http://img175.imageshack.us/my.php?image=snapshot20080612145459xz2.png note: the SVN copy of pyroute will query an OSM API server for data, which sounds quite useful for your Myanmar setup where you have a local OSM server, but not really appropriate for general use where it's rude to do bulk-downloads. There's a copy of the tile data module which doesn't use the API, it assumes a server optimised for handing-out data in packages of one map-tile per download: http://svn.openstreetmap.org/applications/routing/pyroutelib2/tiledata.py but that relies on my [[Tile data server]] which doesn't yet have a server admin to keep it all up-to-date... Note: the pyrender graphics are *nothing like* what you see in the main data layers - currently they're barely more than a debug output of the map. So don't use it when you want to impress anyone with map quality. However, it does share your 'quick setup' idea of not needing to do stupid amounts of processing in advance just to serve map tile images from an OSM API On Wed, Jun 11, 2008 at 2:29 AM, Brett Henderson [EMAIL PROTECTED] wrote: OJ W wrote: It's only outline code now (i.e. no rendering rules), but pyrender shares those objectives (render on demand, get small amounts of data as required) http://svn.openstreetmap.org/applications/rendering/pyrender/ Thanks for the info. Is this something that could be configured to generate tiles for an OpenLayers enabled web site? I have mod_tile up and running now so will go with that approach for now. One thing that might trigger a switch to another rendering mechanism is resource consumption, although mod_tile doesn't seem too heavy. Brett ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
In message [EMAIL PROTECTED] Jon Burgess [EMAIL PROTECTED] wrote: The recent Mapnik builds try to mmap() the shapefiles while rendering[1]. Since the largest shapefile is a few hundred MB this could easily cause some issues if the file was mmap'd multiple times in a single process. This might be a particular problem if you use a 32-bit system in the multi-threaded Apache worker mode. Um... no. At least not if it does the sensible thing and maps it for shared read. Then it will only exist once in memory not matter how many times a process maps it, or indeed how many processes map it in. In fact that is probably the ideal way to access a file like that as you effectively have one disk backed cache of the file that is shared by all users of it. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
Hi, Um... no. At least not if it does the sensible thing and maps it for shared read. I don't think it is even possible to mmap somthing for private read. The boost lib only ever sets the SHARED flag when it wants write access: void* data = ::mmap( hint, pimpl_-size_, readonly ? PROT_READ : (PROT_READ |PROT_WRITE), readonly ? MAP_PRIVATE : MAP_SHARED, pimpl_-handle_, p.offset ); I guess that a shared mode is implicitly used by mmap if you specify read access only; anything else would not make sense (and would mean that the boost guys got it wrong, which is not impossible but rare). Bye Frederik ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
On Tue, 2008-06-10 at 10:40 +0200, Frederik Ramm wrote: Hi, Um... no. At least not if it does the sensible thing and maps it for shared read. I don't think it is even possible to mmap somthing for private read. The boost lib only ever sets the SHARED flag when it wants write access: void* data = ::mmap( hint, pimpl_-size_, readonly ? PROT_READ : (PROT_READ |PROT_WRITE), readonly ? MAP_PRIVATE : MAP_SHARED, pimpl_-handle_, p.offset ); I guess that a shared mode is implicitly used by mmap if you specify read access only; anything else would not make sense (and would mean that the boost guys got it wrong, which is not impossible but rare). Those lines above imply that readonly mappings are flagged MAP_PRIVATE which looks a little odd. The the strace output below seems to confirm these these flags: 23807 open(/home/jburgess/osm/data/world_boundaries/processed_p.shp, O_RDONLY) = 18 ** 23807 mmap(NULL, 327324396, PROT_READ, MAP_PRIVATE, 18, 0) = 0x7f46842d9000 23807 open(/home/jburgess/osm/data/world_boundaries/processed_p.dbf, O_RDONLY) = 19 23807 mmap(NULL, 9758339, PROT_READ, MAP_PRIVATE, 19, 0) = 0x7f469a477000 23807 open(/home/jburgess/osm/data/world_boundaries/processed_p.index, O_RDONLY unfinished ... 23808 open(/home/jburgess/osm/data/world_boundaries/processed_p.shp, O_RDONLY unfinished ... 23807 ... open resumed ) = 20 23808 ... open resumed ) = 21 23807 mmap(NULL, 2268216, PROT_READ, MAP_PRIVATE, 20, 0) = 0x7f46840af000 ** 23808 mmap(NULL, 327324396, PROT_READ, MAP_PRIVATE, 21, 0) = 0x7f4670885000 The two lines ** are memory maps of the same 300MB shapefiles though the fd's are different. This is being done concurrently by 2 threads and the resulting address mappings returned are different. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
Jon Burgess wrote: On Tue, 2008-06-10 at 10:40 +0200, Frederik Ramm wrote: Um... no. At least not if it does the sensible thing and maps it for shared read. I don't think it is even possible to mmap somthing for private read. The boost lib only ever sets the SHARED flag when it wants write access: void* data = ::mmap( hint, pimpl_-size_, readonly ? PROT_READ : (PROT_READ |PROT_WRITE), readonly ? MAP_PRIVATE : MAP_SHARED, pimpl_-handle_, p.offset ); I guess that a shared mode is implicitly used by mmap if you specify read access only; anything else would not make sense (and would mean that the boost guys got it wrong, which is not impossible but rare). Those lines above imply that readonly mappings are flagged MAP_PRIVATE which looks a little odd. The the strace output below seems to confirm these these flags: That does indeed look like a very odd thing to do... The two lines ** are memory maps of the same 300MB shapefiles though the fd's are different. This is being done concurrently by 2 threads and the resulting address mappings returned are different. The actual physical memory will still be shared of course, but you will be exhausting your address space by having the physical memory mapped multiple times. That's especially true on 32 bit systems where address space is more limited. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
On Tue, 2008-06-10 at 08:46 +0100, Tom Hughes wrote:
In message [EMAIL PROTECTED]
Jon Burgess [EMAIL PROTECTED] wrote:
The recent Mapnik builds try to mmap() the shapefiles while
rendering[1]. Since the largest shapefile is a few hundred MB this could
easily cause some issues if the file was mmap'd multiple times in a
single process. This might be a particular problem if you use a 32-bit
system in the multi-threaded Apache worker mode.
Um... no. At least not if it does the sensible thing and maps it
for shared read. Then it will only exist once in memory not matter
how many times a process maps it, or indeed how many processes map
it in.
Yes it'll be once in physical memory. My other email in this thread
shows that that each thread maps it to a different virtual address and
32 bit processes can be quite limited in virtual address space.
The attached test program can only map the 313MB shapefile 8 times
before it fails when run on a plain 32 bit machine:
asus:/tmp$ gcc -o mtest mtest.c
asus:/tmp$ ./mtest
File length: 328204288
Mapping: 0 = 0xa46a3000
Mapping: 1 = 0x90da3000
Mapping: 2 = 0x7d4a3000
Mapping: 3 = 0x69ba3000
Mapping: 4 = 0x562a3000
Mapping: 5 = 0x429a3000
Mapping: 6 = 0x2f0a3000
Mapping: 7 = 0x1b7a3000
Mapping: 8 = Mapping failed at 8
asus:/tmp$ uname -a
Linux asus 2.6.23.12-52.fc7 #1 SMP Tue Dec 18 21:18:02 EST 2007 i686
i686 i386 GNU/Linux
In fact that is probably the ideal way to access a file like that
as you effectively have one disk backed cache of the file that is
shared by all users of it.
Yes, I agree with the mmap() approach in principle. It is much faster
than the old one which did thousands of seeks when reading the files.
Jon
#include stdio.h
#include sys/mman.h
#include sys/types.h
#include sys/stat.h
#include fcntl.h
#include stdlib.h
#include unistd.h
#define SHP /home/jburgess/osm/data/world_boundaries/processed_p.shp
int main(void)
{
void *m[100];
int fd = open(SHP, O_RDONLY);
int i;
size_t len;
if (fd 0) {
perror(SHP);
exit(-1);
}
len = lseek(fd, 0, SEEK_END);
printf(File length: %zd\n, len);
for (i=0; i100; i++) {
printf(Mapping: %d = , i);
m[i] = mmap(NULL, len, PROT_READ, MAP_PRIVATE, fd, 0);
if (m[i] == MAP_FAILED) {
printf(Mapping failed at %d\n, i);
break;
} else
printf(%p\n, m[i]);
}
for(;i=0; i--)
munmap(m[i], len);
close(fd);
return 0;
}
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
OJ W wrote: It's only outline code now (i.e. no rendering rules), but pyrender shares those objectives (render on demand, get small amounts of data as required) http://svn.openstreetmap.org/applications/rendering/pyrender/ Thanks for the info. Is this something that could be configured to generate tiles for an OpenLayers enabled web site? I have mod_tile up and running now so will go with that approach for now. One thing that might trigger a switch to another rendering mechanism is resource consumption, although mod_tile doesn't seem too heavy. Brett ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
Hi All,
Just to give some background on what I'm trying to do here. I'm helping
a team in Myanmar setup a software package called Sahana which is an
open source disaster response package. It will be used to help in the
response to the recent Cyclone.
http://www.sahana.lk/
It has a map component based on OpenLayers allowing it to display maps
of affected areas. They wish to use OSM data to provide this because it
provides them with the best chance of producing decent maps of the area.
A constraint they face is extremely poor and intermittent network
connectivity. I can limit the bandwidth required by using osmosis to
download incremental OSM data updates into a local OSM file and
regularly re-import it into a mapnik postgis database.
They also wish to setup users as OSM editors to directly edit OSM data
in the region, at this point we will attempt to use JOSM's offline
editing. In future we might look into a full replicated OSM instance
with a replication mechanism back to the main OSM database but this is
complicated and lower priority right now.
The Sahana guys have already been able to get Sahana configured to
retrieve tiles from OSM tile servers so the client side is working. The
next step is to render these tiles locally without requiring an Internet
connection. I don't want to pre-render tiles because this constrains
the turnaround time on making new data available for display, on-demand
rendering is far preferable because I'm expecting the load to be
relatively low and disk space will be significantly reduced this way.
I am trying to use tilecache to generate mapnik tiles but have run into
the issue described in my previous email. It is probably something
screwy with the way I've configured things but I've been unable to
diagnose the problem.
The tilecache and mapnik software is all installed under a
/home/tilecache directory tree. I want to keep everything in one place
if possible to make it easier to transfer between machines. All
directories and files are owned by root (will change this to a more
limited user at some point) except the cache directory which is owned by
apache.
* /home/tilecache/app contains the tilecache application
* /home/tilecache/mapnik contains the mapnik scripts from the osm svn
* /home/tilecache/cache contains the tilecache cache files
* /home/tilecache/world_boundaries contains the mapnik world boundary data
My tilecache.cfg file contains the following entry (I assume I should
re-enable tms_type but have been playing with different settings):
[osm]
type=Mapnik
mapfile=/home/tilecache/mapnik/osm.xml
spherical_mercator=true
#tms_type=google
I have an apache configuration section that looks like this:
Alias /tilecache /home/tilecache/app
Directory /home/tilecache/app
AddHandler python-program .py
PythonHandler TileCache.Service
PythonOption TileCacheConfig /home/tilecache/app/tilecache.cfg
PythonPath ['/home/tilecache/app'] + sys.path
PythonDebug On
/Directory
I create an OpenLayers layer with a code snippet like this:
layer = new OpenLayers.Layer.TMS( OSM,
tilecache.py/, {layername: 'osm', type: 'png'} );
You can test the site here:
http://www.bretth.com/tilecache/
Some tiles are working, some are failing. Sometimes retrying fixes
broken tiles, sometimes not.
I am running out of ideas. If anybody has any suggestions or can help
in any way please let me know.
Regards,
Brett
Brett Henderson wrote:
Hi All,
I'm trying to setup a server rendering mapnik tiles on demand using
tilecache.
It seems to be working intermittently but most requests result in the
following response being returned to the browser when requesting new
tiles.
An error occurred: failed mapping file: Cannot allocate memory
File /home/tilecache/app/TileCache/Service.py, line 221, in
modPythonHandler
host )
File /home/tilecache/app/TileCache/Service.py, line 205, in
dispatchRequest
return self.renderTile(tile, params.has_key('FORCE'))
File /home/tilecache/app/TileCache/Service.py, line 138, in renderTile
data = layer.render(tile)
File /home/tilecache/app/TileCache/Layer.py, line 411, in render
return self.renderTile(tile)
File /home/tilecache/app/TileCache/Layers/Mapnik.py, line 38, in
renderTile
mapnik.load_map(m,self.mapfile)
If I refresh the tile it eventually gets generated but sometimes takes
several tries. The apache access_log shows the request as a 500. The
apache error_log doesn't contain anything. I have the PythonDebug
directive set to On.
Any ideas what is causing this?
Regards,
Brett
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
In message [EMAIL PROTECTED]
Brett Henderson [EMAIL PROTECTED] wrote:
I am trying to use tilecache to generate mapnik tiles but have run into
the issue described in my previous email. It is probably something
screwy with the way I've configured things but I've been unable to
diagnose the problem.
I may be being dense here, but this is openstreetmap-talk (not
even dev) not tilecache-talk... Is there not somewhere better you
should be going to with this where there will be experts in tilecache
that can help you?
It seems to be working intermittently but most requests result in the
following response being returned to the browser when requesting new
tiles.
An error occurred: failed mapping file: Cannot allocate memory
File /home/tilecache/app/TileCache/Service.py, line 221, in
modPythonHandler
host )
File /home/tilecache/app/TileCache/Service.py, line 205, in
dispatchRequest
return self.renderTile(tile, params.has_key('FORCE'))
File /home/tilecache/app/TileCache/Service.py, line 138, in renderTile
data = layer.render(tile)
File /home/tilecache/app/TileCache/Layer.py, line 411, in render
return self.renderTile(tile)
File /home/tilecache/app/TileCache/Layers/Mapnik.py, line 38, in
renderTile
mapnik.load_map(m,self.mapfile)
Well the obvious answer is that something in your apache configuration
is limiting the amount of memory that the daemons can use.
Unless of course you really are managing to exhaust your physical
memory and swap - how big do your apache processes get?
Tom
--
Tom Hughes ([EMAIL PROTECTED])
http://www.compton.nu/
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
On Sun, Jun 8, 2008 at 11:08 PM, Brett Henderson [EMAIL PROTECTED] wrote: Hi All, Just to give some background on what I'm trying to do here. I'm helping a team in Myanmar setup a software package called Sahana which is an open source disaster response package. It will be used to help in the response to the recent Cyclone. http://www.sahana.lk/ Brett, Kudos for giving your time to this humanitarian effort and demonstrating the power of OSM at the same time! Karl ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
It's only outline code now (i.e. no rendering rules), but pyrender shares those objectives (render on demand, get small amounts of data as required) http://svn.openstreetmap.org/applications/rendering/pyrender/ Various versions of that can go to an OSM API for data, or use a [[Tile Data Server]]. e.g. file here: http://svn.openstreetmap.org/applications/routing/pyroutelib2/tiledata.py On Mon, Jun 9, 2008 at 7:08 AM, Brett Henderson [EMAIL PROTECTED] wrote: The next step is to render these tiles locally without requiring an Internet connection. I don't want to pre-render tiles because this constrains the turnaround time on making new data available for display, on-demand rendering is far preferable because I'm expecting the load to be relatively low and disk space will be significantly reduced this way. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
Re: [OSM-talk] Mapnik Tilecache Memory Error (Myanmar Cyclone Relief)
On Mon, 2008-06-09 at 18:48 +1000, Brett Henderson wrote: Tom Hughes wrote: I may be being dense here, but this is openstreetmap-talk (not even dev) not tilecache-talk... Is there not somewhere better you should be going to with this where there will be experts in tilecache that can help you? I've sent an email to the tilecache mailing list, just thought there might be some folks around here familiar with this stuff. As for osm-dev versus osm-talk, I wasn't sure which was appropriate in this case ... Well the obvious answer is that something in your apache configuration is limiting the amount of memory that the daemons can use. Sounds reasonable, I'm not too familiar with the innards of apache. It's a default fedora 8 install, I'll poke around and see what I can find. Unless of course you really are managing to exhaust your physical memory and swap - how big do your apache processes get? The machine has 2GB of RAM and isn't overly stressed. The processes are all using around 115MB virtual memory, 24MB resident memory and 13MB shared memory. My previous experience with tilecache suggested that it did not have any mechanisms to limit the number of tiles that could be rendered concurrently which could lead to lots of processes trying to render at once. The recent Mapnik builds try to mmap() the shapefiles while rendering[1]. Since the largest shapefile is a few hundred MB this could easily cause some issues if the file was mmap'd multiple times in a single process. This might be a particular problem if you use a 32-bit system in the multi-threaded Apache worker mode. Perhaps you should try drastically cutting down the number of Apache server processes and see if this resolves the issue. Alternatively you might want to look at mod_tile[2]. One of its design goals was to achieve a well defined behaviour under load. It runs well on the main tile.openstreetmap.org server. [1] http://trac.mapnik.org/changeset/628 [2] http://wiki.openstreetmap.org/index.php/Mod_tile Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
