The smaller caches might indeed explain the difference. Also the ceiling is indeed observed.
Do you have any idea why setting `gdal.SetCacheMax(0)` does not clear the memory? Setting it to 0 should explicitly clear the cache according to https://github.com/OSGeo/gdal/blob/master/gdal/gcore/gdalrasterblock.cpp#L167 but `GetCacheUsed` already reports 0 right after the python call finishes. Some more extracts of our logfile: ``` Setting GDAL CacheMax to 31111.349609375 MB … 95 9702.1 MiB 9593.9 MiB gdal.Warp(temp.name, input_path, **kwargs) 97 40839.7 MiB 31137.6 MiB gdal.Translate(output_path, temp.name, creationOptions=copts) ``` We set the CacheMax using the python bindings and we can see the Translate call reaches the maximum amount. Afterwards gdal reports `gdal cache: 0.0/31111.349609375` (Used/max) but the memory usage stays the same. To explain why this is a problem for us. Afterwards we run gdal2tiles with GDAL_CACHEMAX set to 50% of the free RAM which is very low because the cache from the calls before is seen as not-free or is not freed. Thanks already for helping me look into this. Evert On 26 Dec 2019, at 16:07, Even Rouault <[email protected]<mailto:[email protected]>> wrote: On jeudi 26 décembre 2019 14:30:35 CET Evert Etienne (SITEMARK) wrote: Using a bigger file (8GB) and a machine with 64GB Ram we can see the increase being higher. For both gdal.Warp and gdal.Translate GDAL may use up to GDAL_CACHEMAX memory, plus some other smaller caches. I'm not sure how memory_profile monitors memory consumption, but it is possible that the apparent virtual memory is not returned to the OS. But if you run those commands several times in the same procss, you should hopefully observe some ceiling in memory consumption. Even -- Spatialys - Geospatial professional services https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=01%7C01%7Cevert.etienne%40sitemark.com%7Cb0c5f46254fc4bad08ac08d78a15516c%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=St%2FX27v2p7nYw2IP2RwKMnH3ghGSJFj8Jfld5%2F%2B%2BYNY%3D&reserved=0
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
