Re: [mapserver-users] MapCache: cache some zoom levels, proxy others?

[thomas bonfort]

Paul,

Are you using metatiling? That would substantially reduce the number
of requests mapcache does to mapserver.

You also have the

...
   true

option, that lets mapcache launch mapserver requests in parallel
instead of sequentially (YMMV, this option isn't very tested)

--
thomas

On 28 October 2015 at 16:20, Paul Madden  wrote:
> I am working with a web application that makes WMS queries to MapCache. The 
> application supports zoom levels 0 through 13, and I have used mapcache_seed 
> to seed the cache through level 10. The deeper levels take so long to seed 
> that I am considering stopping at level 10 and letting MapCache / MapServer 
> produce deeper levels on demand, as users visit them. The problem I'm facing 
> is that, as far as I can tell, MapCache requests tiles from MapServer 
> serially, then stitches together the required image from these tiles, when it 
> receives a request for an uncached level. In one test, 34 separate tiles were 
> requested from MapServer, one at a time, and it took ~30 seconds to render 
> the final image.
>
> I'm wondering whether there is a way to configure MapCache to cache/provide a 
> defined set of zoom levels, and to proxy requests at other zoom levels 
> directly to MapServer. I tried setting minzoom and maxzoom on my  in 
> mapcache.xml to "0 10". I tried adding a proxy for my cached layer, hoping 
> that MapCache would redirect requests for unsupported zoom levels to 
> MapServer (I have proxies in place for other, completely uncached layers, and 
> those work fine). I tried copying/renaming the GoogleMapsCompatible grid 
> definition into my mapcache.xml and removing from  zoom levels I 
> don't want MapCache to cache. So far, these attempts apparently just prevent 
> MapCache from serving the deeper layers at all.
>
> Any thoughts on how I might address this issue? It might also help if 
> MapCache would make concurrent requests for tiles at uncached levels, but I 
> have no idea if this is feasible. And it would be nice to be able to let 
> MapServer handle some zoom levels directly in any case.
>
> Thanks in advance for any ideas.
>
> Paul Madden
> National Snow and Ice Data Center
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache: cache some zoom levels, proxy others?

[Paul Madden]

Hi Thomas,

Thank you for reminding me about metatiling. I was initially using the 
MapServer v6.4 / MapCache v1.2 available with the version of Ubuntu I 
need to use, and disabled metatiling because of a (maybe) bug whereby, 
at some shallow zoom levels, some parts of my layer would fail to appear 
or display. I found that, by varying the  values, I could push 
the problem to different zoom levels, but I could not seem to eliminate 
it. When I turned metatiling off, this problem disappeared but I then 
saw, probably unsurprisingly, issues with truncated and overlapping 
labels at deep zoom levels. I then built the v7.0 / v1.4 releases from 
source and the problem disappeared. However, Apache would segfault in 
the MapCache module when I tried to use the "sqlite3" cache type ("disk" 
worked fine, but I have more tiles than inodes, so I cannot use it). 
Finally, I built the head of your git master and then "sqlite3" worked 
fine, too. (I can get you a stacktrace of the v1.4 segfault if you're 
interested -- but you seem already to have fixed whatever the underlying 
issue was).


Sorry for the long backstory -- maybe some of it will be useful to someone.

I will try using metatiling again, and will test  as 
well. Thanks!


Out of curiosity, can I assume that my other idea, about proxying some 
zoom levels directly to WMS and caching others, is unrealistic?


paul

On 10/28/15 9:36 AM, thomas bonfort wrote:

Paul,

Are you using metatiling? That would substantially reduce the number
of requests mapcache does to mapserver.

You also have the

...
true

option, that lets mapcache launch mapserver requests in parallel
instead of sequentially (YMMV, this option isn't very tested)

--
thomas

On 28 October 2015 at 16:20, Paul Madden  wrote:

I am working with a web application that makes WMS queries to MapCache. The 
application supports zoom levels 0 through 13, and I have used mapcache_seed to 
seed the cache through level 10. The deeper levels take so long to seed that I 
am considering stopping at level 10 and letting MapCache / MapServer produce 
deeper levels on demand, as users visit them. The problem I'm facing is that, 
as far as I can tell, MapCache requests tiles from MapServer serially, then 
stitches together the required image from these tiles, when it receives a 
request for an uncached level. In one test, 34 separate tiles were requested 
from MapServer, one at a time, and it took ~30 seconds to render the final 
image.

I'm wondering whether there is a way to configure MapCache to cache/provide a defined set of zoom 
levels, and to proxy requests at other zoom levels directly to MapServer. I tried setting minzoom and 
maxzoom on my  in mapcache.xml to "0 10". I tried adding a proxy for my cached 
layer, hoping that MapCache would redirect requests for unsupported zoom levels to MapServer (I have 
proxies in place for other, completely uncached layers, and those work fine). I tried 
copying/renaming the GoogleMapsCompatible grid definition into my mapcache.xml and removing from 
 zoom levels I don't want MapCache to cache. So far, these attempts apparently 
just prevent MapCache from serving the deeper layers at all.

Any thoughts on how I might address this issue? It might also help if MapCache 
would make concurrent requests for tiles at uncached levels, but I have no idea 
if this is feasible. And it would be nice to be able to let MapServer handle 
some zoom levels directly in any case.

Thanks in advance for any ideas.

Paul Madden
National Snow and Ice Data Center
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache: cache some zoom levels, proxy others?

[thomas bonfort]

> Out of curiosity, can I assume that my other idea, about proxying some zoom
> levels directly to WMS and caching others, is unrealistic?

not unrealistic, but not implemented

--
thomas

>
> paul
>
>
> On 10/28/15 9:36 AM, thomas bonfort wrote:
>>
>> Paul,
>>
>> Are you using metatiling? That would substantially reduce the number
>> of requests mapcache does to mapserver.
>>
>> You also have the
>> 
>> ...
>> true
>> 
>> option, that lets mapcache launch mapserver requests in parallel
>> instead of sequentially (YMMV, this option isn't very tested)
>>
>> --
>> thomas
>>
>> On 28 October 2015 at 16:20, Paul Madden  wrote:
>>>
>>> I am working with a web application that makes WMS queries to MapCache.
>>> The application supports zoom levels 0 through 13, and I have used
>>> mapcache_seed to seed the cache through level 10. The deeper levels take so
>>> long to seed that I am considering stopping at level 10 and letting MapCache
>>> / MapServer produce deeper levels on demand, as users visit them. The
>>> problem I'm facing is that, as far as I can tell, MapCache requests tiles
>>> from MapServer serially, then stitches together the required image from
>>> these tiles, when it receives a request for an uncached level. In one test,
>>> 34 separate tiles were requested from MapServer, one at a time, and it took
>>> ~30 seconds to render the final image.
>>>
>>> I'm wondering whether there is a way to configure MapCache to
>>> cache/provide a defined set of zoom levels, and to proxy requests at other
>>> zoom levels directly to MapServer. I tried setting minzoom and maxzoom on my
>>>  in mapcache.xml to "0 10". I tried adding a proxy for my cached
>>> layer, hoping that MapCache would redirect requests for unsupported zoom
>>> levels to MapServer (I have proxies in place for other, completely uncached
>>> layers, and those work fine). I tried copying/renaming the
>>> GoogleMapsCompatible grid definition into my mapcache.xml and removing from
>>>  zoom levels I don't want MapCache to cache. So far, these
>>> attempts apparently just prevent MapCache from serving the deeper layers at
>>> all.
>>>
>>> Any thoughts on how I might address this issue? It might also help if
>>> MapCache would make concurrent requests for tiles at uncached levels, but I
>>> have no idea if this is feasible. And it would be nice to be able to let
>>> MapServer handle some zoom levels directly in any case.
>>>
>>> Thanks in advance for any ideas.
>>>
>>> Paul Madden
>>> National Snow and Ice Data Center
>>> ___
>>> mapserver-users mailing list
>>> mapserver-users@lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>
> ___
> mapserver-users mailing list
> mapserver-users@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] MapCache: cache some zoom levels, proxy others?

[Paul Madden]

Reading the docs more carefully, it sounds like SQLite databases may have 
concurrency problems even with only mapcache_seed accessing them, given a 
sufficiently high number of threads. I'm guessing that 8 is not "a high 
number", and that a few extra from MapCache itself isn't either. Does anybody 
know differently?

On 10/28/15 20:08, Paul Madden wrote:
> Travis,
> 
> Thank you -- I hadn't realized before what you could do with the various meta 
> caches. I'll definitely be experimenting with a front-end memcache at some 
> point.
> 
> But, I can't yet see how I can overcome the sqlite3 issue with these methods. 
> It seems like the following would *almost* work:
> 
> 
> db1
> db2
> 
> 
> where both db1 and db2 are sqlite3 caches, and db1 is being actively seeded 
> by mapcache_seed. At first I thought: MapCache will check db1 to see if the 
> requested tile has already been seeded and, if not, will move on to db2, 
> which will make the appropriate WMS requests to MapServer. Only mapcache_seed 
> will write to db1, and only MapCache will write to db2: No concurrency issue. 
> But I see this in the docs: "If a given tile isn’t found in the first child 
> cache, it will be read from the second child cache AND copied into the first 
> child cache for subsequent accesses." Given the "AND" part, it seems that 
> both mapcache_seed and MapCache will write concurrently to db1. If this is 
> right, does such a configuration actually avoid the sqlite3 concurrency 
> issue? Is there a better configuration that I've failed to imagine?
> 
> On a related note: If *only* mapcache_seed writes to a sqlite3, but runs in 
> multi-threaded mode, is it sqlite3-safe?
> 
> (The questions are for anyone kind enough to answer...)
> 
> paul
> 
> On 10/28/15 17:42, Travis Kirstine wrote:
>> Paul,
>>
>> I would have a look at some of the new tiered caching options available in 
>> 1.4.  SQLite can have issues locking under load
>>
>> On Wednesday, October 28, 2015, Paul Madden > > wrote:
>>
>>  Thomas,
>>
>>  Enabling metatiling indeed led to a significant speedup. I can also 
>> verify that  works for me (I can see the additional 
>> threads running), but it did not lead to a noticeable speedup, perhaps due 
>> to a bottleneck where my MapServer talks to a remote database. I also 
>> increased my tile size from 256x256 to 1024x1024, which seems fine for my 
>> application and which obviously reduces the number of tiles helpfully.
>>
>>  If I may ask one more question: Are you aware of any problems running 
>> mapcache_seed concurrently with users accessing MapCache via the web 
>> application, if one uses the sqlite3 cache? I saw a thread about potential 
>> conflicts, but I think that might have been specific to disk caches.
>>
>>  Thanks again,
>>  paul
>>
>>  On 10/28/15 9:59 AM, thomas bonfort wrote:
>>
>>  Out of curiosity, can I assume that my other idea, about 
>> proxying some zoom
>>  levels directly to WMS and caching others, is unrealistic?
>>
>>  not unrealistic, but not implemented
>>
>>  --
>>  thomas
>>
>>  paul
>>
>>
>>  On 10/28/15 9:36 AM, thomas bonfort wrote:
>>
>>  Paul,
>>
>>  Are you using metatiling? That would substantially reduce 
>> the number
>>  of requests mapcache does to mapserver.
>>
>>  You also have the
>>  
>>  ...
>>true
>>  
>>  option, that lets mapcache launch mapserver requests in 
>> parallel
>>  instead of sequentially (YMMV, this option isn't very 
>> tested)
>>
>>  --
>>  thomas
>>
>>  On 28 October 2015 at 16:20, Paul Madden 
>>  wrote:
>>
>>  I am working with a web application that makes WMS 
>> queries to MapCache.
>>  The application supports zoom levels 0 through 13, and 
>> I have used
>>  mapcache_seed to seed the cache through level 10. The 
>> deeper levels take so
>>  long to seed that I am considering stopping at level 10 
>> and letting MapCache
>>  / MapServer produce deeper levels on demand, as users 
>> visit them. The
>>  problem I'm facing is that, as far as I can tell, 
>> MapCache requests tiles
>>  from MapServer serially, then stitches together the 
>> required image from
>>  these tiles, when it receives a request for an uncached 
>> level. In one test,
>>  34 separate tiles were requested from MapServer, one at 
>> a time, and it took
>>  ~30 seconds to render the final image.
>>
>>  I'm wondering whether there is a way to configure 
>> MapCache to
>>  

Re: [mapserver-users] MapCache: cache some zoom levels, proxy others?

[Paul Madden]

Thomas,

Enabling metatiling indeed led to a significant speedup. I can also 
verify that  works for me (I can see the additional 
threads running), but it did not lead to a noticeable speedup, perhaps 
due to a bottleneck where my MapServer talks to a remote database. I 
also increased my tile size from 256x256 to 1024x1024, which seems fine 
for my application and which obviously reduces the number of tiles 
helpfully.


If I may ask one more question: Are you aware of any problems running 
mapcache_seed concurrently with users accessing MapCache via the web 
application, if one uses the sqlite3 cache? I saw a thread about 
potential conflicts, but I think that might have been specific to disk 
caches.


Thanks again,
paul

On 10/28/15 9:59 AM, thomas bonfort wrote:

Out of curiosity, can I assume that my other idea, about proxying some zoom
levels directly to WMS and caching others, is unrealistic?

not unrealistic, but not implemented

--
thomas


paul


On 10/28/15 9:36 AM, thomas bonfort wrote:

Paul,

Are you using metatiling? That would substantially reduce the number
of requests mapcache does to mapserver.

You also have the

...
 true

option, that lets mapcache launch mapserver requests in parallel
instead of sequentially (YMMV, this option isn't very tested)

--
thomas

On 28 October 2015 at 16:20, Paul Madden  wrote:

I am working with a web application that makes WMS queries to MapCache.
The application supports zoom levels 0 through 13, and I have used
mapcache_seed to seed the cache through level 10. The deeper levels take so
long to seed that I am considering stopping at level 10 and letting MapCache
/ MapServer produce deeper levels on demand, as users visit them. The
problem I'm facing is that, as far as I can tell, MapCache requests tiles
from MapServer serially, then stitches together the required image from
these tiles, when it receives a request for an uncached level. In one test,
34 separate tiles were requested from MapServer, one at a time, and it took
~30 seconds to render the final image.

I'm wondering whether there is a way to configure MapCache to
cache/provide a defined set of zoom levels, and to proxy requests at other
zoom levels directly to MapServer. I tried setting minzoom and maxzoom on my
 in mapcache.xml to "0 10". I tried adding a proxy for my cached
layer, hoping that MapCache would redirect requests for unsupported zoom
levels to MapServer (I have proxies in place for other, completely uncached
layers, and those work fine). I tried copying/renaming the
GoogleMapsCompatible grid definition into my mapcache.xml and removing from
 zoom levels I don't want MapCache to cache. So far, these
attempts apparently just prevent MapCache from serving the deeper layers at
all.

Any thoughts on how I might address this issue? It might also help if
MapCache would make concurrent requests for tiles at uncached levels, but I
have no idea if this is feasible. And it would be nice to be able to let
MapServer handle some zoom levels directly in any case.

Thanks in advance for any ideas.

Paul Madden
National Snow and Ice Data Center
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users


___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users