I solved my problem... I found a way to reduce memory utilization and allocations when using MapServer with GDAL >= 2.3.0:
CONFIG "GDAL_HTTP_MULTIRANGE" "SERIAL" With this setting, the memory footprint matches that of GDAL <= 2.2.4. With the default (YES) has mapserv allocates much more memory. Running mapserv with supervisor & configured over FCGI results in a much higher memory footprint over time. My best guess is that with the default parallel mode has many HTTP connections kept alive and each connection consumes some non-trivial amount of memory. With CONFIG "CPL_CURL_VERBOSE" "ON", I see 9 HTTP connections kept around only with the default GDAL_HTTP_MULTIRANGE=ON setting: * Curl_http_done: called premature == 0 * Connection #17 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #11 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #13 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #14 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #15 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #19 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #16 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #18 to host s3.amazonaws.com left intact * Curl_http_done: called premature == 0 * Connection #12 to host s3.amazonaws.com left intact Side-note: the comments in cpl_vsil_curl.cpp for GDAL_HTTP_MULTIRANGE do not match the ConfigOptions documentation https://github.com/OSGeo/gdal/blob/1d799cf5d8c399d9ac3feb7ef9f05dc6e3297a82/gdal/port/cpl_vsil_curl.cpp#L2691-L2696 https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_HTTP_MULTIRANGE The valid settings are actually SINGLE_GET, SERIAL or _anything_ else for "parallel" mode It's great to see all the vsicurl improvements to GDAL which MapServer can leverage! Time for me to experiment with the other gdal-2.3 config options for libcurl >= 7.33 built with nghttp2. Cheers, Pete On Fri, Mar 29, 2019 at 3:02 PM Peter Schmitt <pschm...@gmail.com> wrote: > Hi, > > I run MapServer in a Docker container. I noticed its memory utilization > climbed much higher than expected. This seems to happen only under the > following conditions: > > * Files are accessed network-based filesystems (I tried both /vsicurl/ and > /vsis3/) > * Using gdal > 2.2.4 (tested with 2.3.0, 2.3.2 & 2.4.1. I see the same > behavior with both MapServer-7.0.7 and MapServer-7.2.2. > > I had a hard time isolating this to GDAL and could not reproduce with > similar gdal_translate commands. So I put together a little example to try > and demonstrate the problem. > https://github.com/pedros007/gdal_mapserver_memory > > Does anyone have any ideas as to what might be going on? I tried various > cache settings, which do not seem to reduce memory utilization: > > https://github.com/pedros007/gdal_mapserver_memory/blob/master/mapfiles/indonesian_redcross.map#L15-L18 > > Thanks, > Pete > -- Pete
_______________________________________________ mapserver-users mailing list mapserver-users@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/mapserver-users