Hi,
Thanks for the good info Peter !! Looking at the text just below these rows: https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65 I see that there is an option that is not documented: CLOSE CONNECTION=NORMAL Shouldn't that option be described in the manual page for LAYER, even if it is the default behavior ? /Lars Schylberg -----Originalmeddelande----- Från: "Peter Schmitt" <[email protected]> Till: [email protected] Datum: 2017-10-27 20:58 Ämne: Re: [mapserver-users] Understanding MapServer Memory Usage with WMS raster layers I think I understand what's going on now and want to share my findings for future Google searchers wondering the same thing. I see unbounded memory growth with many raster layers having a single image because the default behavior of raster layers lacking a Tile Index is DEFER. https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L838-L839 With DEFER, the Connection Pool will keep the rasters open until mapserv closes. https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mappool.c#L62-L65 My setup (https://github.com/pedros007/debian-mapserver Docker container with Supervisord running 8 mapserv processes routed via nginx fcgi) ensures mapserver never exits. So with my Mapfile of many large single-file raster layers results in many large rasters which are opened indefinitely by 8 separate mapserv processes. My typical Mapfile setup has many raster layers each with a tile index of many rasters. In this configuration, I don't see unbounded memory growth. Looks like the default behavior with a tile index is to close each raster in each MapServer request! https://github.com/mapserver/mapserver/blob/93e3eb28b49b508dd610af2fde118f04e2177f13/mapraster.c#L845 In short, I can prevent unbounded memory growth by setting CLOSE_CONNECTION=ALWAYS on single-image raster layers. I suppose I figured that out before writing my original question to the mailing list, but at least I now know why this solves my problem. Sorry for adding noise to the list! On Wed, Oct 25, 2017 at 11:42 AM, Peter Schmitt <[email protected]> wrote: Hi, I am using MapServer as a WMS endpoint serving rasters. My configuration has Supervisord running 8 mappserv processes which are listening on a socket and dispatched via Nginx FastCGI. This all runs in the following Docker image: https://github.com/pedros007/debian-mapserver I noticed my memory utilization monotonically increases over time only drops when there's a log in dmesg on the host Linux OS (... /not/ in the OS of the Docker container) like the following: [508072.641250] Memory cgroup out of memory: Kill process 21147 (mapserv) score 143 or sacrifice child [508072.646072] Killed process 21147 (mapserv) total-vm:672384kB, anon-rss:449172kB, file-rss:0kB, shmem-rss:0kB I can dramatically lower my memory footprint by adding this directive to every raster LAYER as suggested by https://github.com/mapserver/mapserver/issues/4389 PROCESSING "CLOSE_CONNECTION=ALWAYS" With all of that, I have a few questions: 1. With the default CLOSE_CONNECTION=DEFER on a raster layer, are raster file handles ever closed? 2. What does the config variable GDAL_CACHEMAX do in relation to CLOSE_CONNECTION = DEFER or ALWAYS? I assume with processing directive CLOSE_CONNECTION=ALWAYS, GDAL_CACHEMAX does nothing for me. 3. My Mapfile actually reads rasters at /vsicurl/ URLs. I assume that VSI_CACHE=TRUE and VSI_CACHE_SIZE are a LRU cache which is in addition to the GDAL cache and any OS/host level caching of open file handles. Is the same data cached separately in up to 3 places (GDAL cache, Host cache and VSI cache)? Thanks! Pete -- Pete _______________________________________________ mapserver-users mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/mapserver-users --
_______________________________________________ mapserver-users mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/mapserver-users
