The issue is also visible on process level. Do you know if this cache exists in process level and if so, if free can show this as well?
On Dec 27, 2019 20:08, Andrew C Aitchison <[email protected]> wrote: On Thu, 26 Dec 2019, Evert Etienne (SITEMARK) wrote: > I am measuring memory usage using psutil: Process: > `psutil.Process().memory_info().rss` and > `psutil.Process().memory_percent()` General: > `psutil.virtual_memory().available` > > I do tests both on MacOS and linux. > > Any idea about how to investigate the file cache or how to use > psutil to check the difference between this file-system cache and > other memory usage? I'm afraid I don't know enough python to help. On Linux (not sure about MacOS) the command "free" shows this filesystem cache as "buff/cache". > By the way, instead of using a real tempfile but removing it and > using `input + â??.tmpâ??` as output file yields the same behavior. > > When calling the gdal calls in a Process, the issues do not appear (3 > different approaches) > > ``` > 74 109.5 MiB 0.0 MiB ctx = > multiprocessing.get_context('spawn') > 75 109.5 MiB 0.0 MiB p = > ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), > kwargs=kwargs) > 76 109.7 MiB 0.1 MiB p.start() > 77 105.5 MiB 0.0 MiB p.join() > 78 105.5 MiB 0.0 MiB ctx = > multiprocessing.get_context('fork') > 79 105.5 MiB 0.0 MiB p = > ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path), > kwargs=kwargs) > 80 105.5 MiB 0.0 MiB p.start() > 81 105.5 MiB 0.0 MiB p.join() > 82 3750.6 MiB 3645.0 MiB gdalwarp_wrapper(output_path, > input_path, **kwargs) > ``` > > Thanks! > > On 26 Dec 2019, at 20:21, Andrew C Aitchison > <[email protected]<mailto:[email protected]>> wrote: > > How are you measuring free RAM at this point ? > > The GDAL cache will be included in the the memory use reported by > the python interpreter, but at least on Linux the system command > "free" also reports the memory used by the file-system cache, which > will contain (the read parts of) the file, until the system decides > to use that RAM for something else. Your tempfile is likely being > stored in the filesystem cache.
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
