Hi Thomas So size looks really good. Recreated one of our caches which has minimal 'blank tiles', and av. tile size is reasonably heavy ~ 24K
File Based: 65G Previous BDB (empty-tiles branch) 142G BDB (empty-tiles branch) with PAGESIZE 1*1024 62G However I have run into an issue with apache. Getting seg fault messages in apache error log, e.g. [notice] child pid 5555 exit signal Segmentation fault (11) Which requires restart of apache to make things work again. Was trying to work out the steps required to reproduce, restarting apache each time and I ran into a different error, I think the one you have mentioned before, relating to multiple restarts of apache unable to allocate memory for mutex; resize mutex region [Thu May 31 17:13:32 2012] [error] [client <IP>] failed to aquire connection to bdb backend: bdb cache failure 1 for db->open: Cannot allocate memory, referer: http:/server/bdb_test.html Running a db_recover on it fixes this, as you mentioned. I need to head home now, but happy to debug tomorrow, if you run me through the steps. Cheers Andrew -----Original Message----- From: thomas bonfort [mailto:[email protected]] Sent: Tuesday, 29 May 2012 11:07 p.m. To: Ludovic Gnemmi; Andrew Cowie Cc: [email protected] Subject: Re: [mapserver-users] MapCache - A way to reduce BerkeleyDB Cache size ? Salut Ludovic, hi Andrew, I've investigated a bit more on the disk space issue with berkeley db backends, and here are some of my findings. To summarize, the space occupancy/efficiency is closely tied to the size of the images stored in the cache, and the configured pagesize of the bdb backend. For a cache that contains many empty tiles, i.e. that are just a few hundred to a couple thousand bytes, here are the space occupancies (the disk cache uses 4k blocks). This is for reference only, usual caches will not resemble this one. Also note that the disk cache was not configured to symlink blank tiles, and that the bdb cache backend has some code in it to reduce the tile data stored on empty tiles. disk: 197M bdb pagesize=64k: 106M bdb pagesize=4k: 96M bdb pagesize=1k: 73M sqlite: 103M mbtiles (sqlite with blank tile detection): 62M So these numbers actually aren't so bad for bdb in all configurations, but in this case the 4k filesytem blocksize is really inefficient. These are the occupancies for another tileset, where the tiles are much heavier (tiles around 40-45k). There are also quite a few empty tiles. disk:153M bdb pagesize=64k: 209M bdb pagesize=4k: 148M bdb pagesize=1k: 144M sqlite: 140M mbtiles (sqlite with blank tile detection): 137M I had initally stayed away from using small pagesizes in the bdb backend because of some stability issues, but in the current tests I have not run into any issues. From these tests, I think the natural way forward is to change the default pagesize, and maybe make it configurable in mapcache.xml as an advanced configuration option. If you are able to confirm my findings on your instances, that would be great. To set the pagesize you'll have to recompile/reinstall, after having changed lib/cache_bdb.c by replacing #define PAGESIZE 64*1024 by #define PAGESIZE 1*1024 near line 50. regards, thomas On Wed, May 23, 2012 at 3:55 PM, Ludovic Gnemmi <[email protected]> wrote: > Hi list, > I would like to experiment Berkeley DB cache type but i'm surprised of > the resulting db size compared to sqlite cache type. Here is my > results for about 13000 tiles: > > Total tiles size = 44 Mb > Sqlite = 48 Mb > Berkeley DB = 61 Mb > Disk = 66 Mb (with 4kb filesystem block size) > > I thought i could reduce storage capacity with Berkeley DB cache type > as with Sqlite but it seems there is not much difference with disk cache type. > Is there a way to reduce Berkeley DB cache size ? > > Thanks for your help > > Ludovic > > -- > View this message in context: > http://osgeo-org.1560.n6.nabble.com/MapCache-A-way-to-reduce-BerkeleyD > B-Cache-size-tp4976403.html Sent from the Mapserver - User mailing > list archive at Nabble.com. > _______________________________________________ > mapserver-users mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/mapserver-users Please consider the environment before printing this email Warning: This electronic message together with any attachments is confidential. If you receive it in error: (i) you must not read, use, disclose, copy or retain it; (ii) please contact the sender immediately by reply email and then delete the emails. The views expressed in this email may not be those of Landcare Research New Zealand Limited. http://www.landcareresearch.co.nz _______________________________________________ mapserver-users mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/mapserver-users
