I actually have no idea what the extra code in your examples is doing, can you explain?
For the behavior: it is the same as psutil (makes sense since memory_profiler uses psutil by default). You can see it reach max cache by logging psutil values in the Translate callback. I also saw the same behavior by looking at the memory usage using top. Is there another way you know that works that I can try to see if I can replicate the memory usage values? Thank you > On 27 Dec 2019, at 21:45, Even Rouault <[email protected]> wrote: > > I question the reliability of memory_profiler > > See > > Line # Mem usage Increment Line Contents > ================================================ > 27 112.125 MiB 112.125 MiB @profile > 28 def test2(): > 29 112.676 MiB 0.551 MiB gdal.SetCacheMax(1000 * 1024 * 1024) > 30 1101.855 MiB 989.180 MiB gdal.Translate('/tmp/out.tif', > 'byte.tif', options = '-co TILED=YES -outsize 100000 10000') > 31 1101.855 MiB 0.000 MiB dummy = 1 > > vs > > Line # Mem usage Increment Line Contents > ================================================ > 28 109.336 MiB 109.336 MiB @profile > 29 def test2(): > 30 109.336 MiB 0.000 MiB gdal.SetCacheMax(1000 * 1024 * 1024) > 31 1098.309 MiB 988.973 MiB gdal.Translate('/tmp/out.tif', > 'byte.tif', options = '-co TILED=YES -outsize 100000 10000') > 32 1098.309 MiB 0.000 MiB h = ctypes.cdll.LoadLibrary(None) > 33 1098.309 MiB 0.000 MiB h.malloc.argtypes = [ctypes.c_size_t] > 34 1098.309 MiB 0.000 MiB h.malloc.restype = ctypes.c_void_p > 35 1098.309 MiB 0.000 MiB h.free.argtypes = [ctypes.c_void_p] > 36 1098.309 MiB 0.000 MiB h.free.restype = None > 37 1098.309 MiB 0.000 MiB size = 1024 * 1024 > 38 1098.309 MiB 0.000 MiB x = h.malloc(size) > 39 117.422 MiB 0.000 MiB h.free(x) > 40 117.422 MiB 0.000 MiB dummy = 1 > > vs > > Line # Mem usage Increment Line Contents > ================================================ > 28 111.305 MiB 111.305 MiB @profile > 29 def test2(): > 30 111.852 MiB 0.547 MiB gdal.SetCacheMax(1000 * 1024 * 1024) > 31 188.504 MiB 76.652 MiB gdal.Translate('/tmp/out.tif', > 'byte.tif', options = '-co TILED=YES -outsize 100000 10000') > 32 188.504 MiB 0.000 MiB h = ctypes.cdll.LoadLibrary(None) > 33 188.504 MiB 0.000 MiB h.free.argtypes = [ctypes.c_void_p] > 34 188.504 MiB 0.000 MiB h.free.restype = None > 35 188.504 MiB 0.000 MiB h.free(None) > 36 188.504 MiB 0.000 MiB dummy = 1 > > vs > > Line # Mem usage Increment Line Contents > ================================================ > 28 112.164 MiB 112.164 MiB @profile > 29 def test2(): > 30 112.164 MiB 0.000 MiB gdal.SetCacheMax(1000 * 1024 * 1024) > 31 112.770 MiB 0.605 MiB h = ctypes.cdll.LoadLibrary(None) > 32 112.770 MiB 0.000 MiB h.free.argtypes = [ctypes.c_void_p] > 33 112.770 MiB 0.000 MiB h.free.restype = None > 34 1101.918 MiB 989.148 MiB gdal.Translate('/tmp/out.tif', > 'byte.tif', options = '-co TILED=YES -outsize 100000 10000') > 35 1101.918 MiB 0.000 MiB h.free(None) > 36 1101.918 MiB 0.000 MiB dummy = 1 > > That doesn't make much sense to me. > > Even > > -- > Spatialys - Geospatial professional services > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=01%7C01%7Cevert.etienne%40sitemark.com%7C39f1fc8e39b14a21169d08d78b0da8ab%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=Mg1ztmCAuQnlKkFq3Ny00bE%2BfmzljlV0cWVRrYyU8PQ%3D&reserved=0 _______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
