Reading the docs more carefully, it sounds like SQLite databases may have concurrency problems even with only mapcache_seed accessing them, given a sufficiently high number of threads. I'm guessing that 8 is not "a high number", and that a few extra from MapCache itself isn't either. Does anybody know differently?
On 10/28/15 20:08, Paul Madden wrote: > Travis, > > Thank you -- I hadn't realized before what you could do with the various meta > caches. I'll definitely be experimenting with a front-end memcache at some > point. > > But, I can't yet see how I can overcome the sqlite3 issue with these methods. > It seems like the following would *almost* work: > > <cache name="multitier" type="multitier"> > <cache>db1</cache> > <cache>db2</cache> > </cache> > > where both db1 and db2 are sqlite3 caches, and db1 is being actively seeded > by mapcache_seed. At first I thought: MapCache will check db1 to see if the > requested tile has already been seeded and, if not, will move on to db2, > which will make the appropriate WMS requests to MapServer. Only mapcache_seed > will write to db1, and only MapCache will write to db2: No concurrency issue. > But I see this in the docs: "If a given tile isn’t found in the first child > cache, it will be read from the second child cache AND copied into the first > child cache for subsequent accesses." Given the "AND" part, it seems that > both mapcache_seed and MapCache will write concurrently to db1. If this is > right, does such a configuration actually avoid the sqlite3 concurrency > issue? Is there a better configuration that I've failed to imagine? > > On a related note: If *only* mapcache_seed writes to a sqlite3, but runs in > multi-threaded mode, is it sqlite3-safe? > > (The questions are for anyone kind enough to answer...) > > paul > > On 10/28/15 17:42, Travis Kirstine wrote: >> Paul, >> >> I would have a look at some of the new tiered caching options available in >> 1.4. SQLite can have issues locking under load >> >> On Wednesday, October 28, 2015, Paul Madden <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thomas, >> >> Enabling metatiling indeed led to a significant speedup. I can also >> verify that <threaded_fetching> works for me (I can see the additional >> threads running), but it did not lead to a noticeable speedup, perhaps due >> to a bottleneck where my MapServer talks to a remote database. I also >> increased my tile size from 256x256 to 1024x1024, which seems fine for my >> application and which obviously reduces the number of tiles helpfully. >> >> If I may ask one more question: Are you aware of any problems running >> mapcache_seed concurrently with users accessing MapCache via the web >> application, if one uses the sqlite3 cache? I saw a thread about potential >> conflicts, but I think that might have been specific to disk caches. >> >> Thanks again, >> paul >> >> On 10/28/15 9:59 AM, thomas bonfort wrote: >> >> Out of curiosity, can I assume that my other idea, about >> proxying some zoom >> levels directly to WMS and caching others, is unrealistic? >> >> not unrealistic, but not implemented >> >> -- >> thomas >> >> paul >> >> >> On 10/28/15 9:36 AM, thomas bonfort wrote: >> >> Paul, >> >> Are you using metatiling? That would substantially reduce >> the number >> of requests mapcache does to mapserver. >> >> You also have the >> <mapcache> >> ... >> <threaded_fetching>true</threaded_fetching> >> </mapcache> >> option, that lets mapcache launch mapserver requests in >> parallel >> instead of sequentially (YMMV, this option isn't very >> tested) >> >> -- >> thomas >> >> On 28 October 2015 at 16:20, Paul Madden >> <[email protected]> wrote: >> >> I am working with a web application that makes WMS >> queries to MapCache. >> The application supports zoom levels 0 through 13, and >> I have used >> mapcache_seed to seed the cache through level 10. The >> deeper levels take so >> long to seed that I am considering stopping at level 10 >> and letting MapCache >> / MapServer produce deeper levels on demand, as users >> visit them. The >> problem I'm facing is that, as far as I can tell, >> MapCache requests tiles >> from MapServer serially, then stitches together the >> required image from >> these tiles, when it receives a request for an uncached >> level. In one test, >> 34 separate tiles were requested from MapServer, one at >> a time, and it took >> ~30 seconds to render the final image. >> >> I'm wondering whether there is a way to configure >> MapCache to >> cache/provide a defined set of zoom levels, and to >> proxy requests at other >> zoom levels directly to MapServer. I tried setting >> minzoom and maxzoom on my >> <grid> in mapcache.xml to "0 10". I tried adding a >> proxy for my cached >> layer, hoping that MapCache would redirect requests for >> unsupported zoom >> levels to MapServer (I have proxies in place for other, >> completely uncached >> layers, and those work fine). I tried copying/renaming >> the >> GoogleMapsCompatible grid definition into my >> mapcache.xml and removing from >> <resolutions> zoom levels I don't want MapCache to >> cache. So far, these >> attempts apparently just prevent MapCache from serving >> the deeper layers at >> all. >> >> Any thoughts on how I might address this issue? It >> might also help if >> MapCache would make concurrent requests for tiles at >> uncached levels, but I >> have no idea if this is feasible. And it would be nice >> to be able to let >> MapServer handle some zoom levels directly in any case. >> >> Thanks in advance for any ideas. >> >> Paul Madden >> National Snow and Ice Data Center >> _______________________________________________ >> mapserver-users mailing list >> [email protected] >> http://lists.osgeo.org/mailman/listinfo/mapserver-users >> >> >> _______________________________________________ >> mapserver-users mailing list >> [email protected] >> http://lists.osgeo.org/mailman/listinfo/mapserver-users >> >> >> _______________________________________________ >> mapserver-users mailing list >> [email protected] >> http://lists.osgeo.org/mailman/listinfo/mapserver-users >> _______________________________________________ mapserver-users mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/mapserver-users
