Angus,

there has been a recent extra fix that landed in GDAL 3.6.2 that might possibly help: https://github.com/OSGeo/gdal/pull/6926

Even

Le 25/01/2023 à 01:36, Angus Dickey a écrit :
Hi all,

I am running into an issue where GDAL is overestimating the amount of physical memory it has leading to it locking up the OS by taking 100% of the memory. Here is an example program that illustrates the issue:

#include <stdio.h>
#include "gdal.h"

int main(void) {
   printf("GDAL version is %s\n", GDALVersionInfo("RELEASE_NAME"));
   printf("GDAL thinks is has %lld bytes of physical memory\n", CPLGetPhysicalRAM());    printf("GDAL thinks it has %lld bytes of usable physical memory\n", CPLGetUsablePhysicalRAM());
   return 0;
}

When this is compiled with GDAL 3.5.1 on Ubuntu 22.04 we get:

$ ./get_gdal_memory
GDAL version is 3.5.1
GDAL thinks is has 811526475776 bytes of physical memory
GDAL thinks it has 811526475776 bytes of usable physical memory

Which is not consistent with the actual available memory:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           2.0Gi       148Mi       1.2Gi       0.0Ki 639Mi       1.8Gi
Swap:          256Mi          0B       256Mi

So GDAL thinks it has 755GB of memory but it only has 2GB, this causes issues with the raster read cache and maybe elsewhere. I suspect this is happening because it is running in a Linux container <https://linuxcontainers.org/> and GDAL is getting the total physical memory of the host, not the container. The strange thing is Linux containers use cgroups for memory restrictions and the API docs mention it was fixed in GDAL 2.4.0 <https://gdal.org/api/cpl.html#_CPPv417CPLGetPhysicalRAMv> but I am still seeing the issue in 3.5.1.

Any help or insight would be appreciated; I am happy to provide any additional information or testing.

Thanks,

Angus

_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Reply via email to