On Thu, 26 Dec 2019, Evert Etienne (SITEMARK) wrote:
I am measuring memory usage using psutil: Process:
`psutil.Process().memory_info().rss` and
`psutil.Process().memory_percent()` General:
`psutil.virtual_memory().available`
I do tests both on MacOS and linux.
Any idea about how to investigate the file cache or how to use
psutil to check the difference between this file-system cache and
other memory usage?
I'm afraid I don't know enough python to help.
On Linux (not sure about MacOS) the command "free" shows this filesystem
cache as "buff/cache".
By the way, instead of using a real tempfile but removing it and
using `input + â??.tmpâ??` as output file yields the same behavior.
When calling the gdal calls in a Process, the issues do not appear (3 different
approaches)
```
74 109.5 MiB 0.0 MiB ctx =
multiprocessing.get_context('spawn')
75 109.5 MiB 0.0 MiB p =
ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path),
kwargs=kwargs)
76 109.7 MiB 0.1 MiB p.start()
77 105.5 MiB 0.0 MiB p.join()
78 105.5 MiB 0.0 MiB ctx =
multiprocessing.get_context('fork')
79 105.5 MiB 0.0 MiB p =
ctx.Process(target=gdalwarp_wrapper, args=(output_path, input_path),
kwargs=kwargs)
80 105.5 MiB 0.0 MiB p.start()
81 105.5 MiB 0.0 MiB p.join()
82 3750.6 MiB 3645.0 MiB gdalwarp_wrapper(output_path,
input_path, **kwargs)
```
Thanks!
On 26 Dec 2019, at 20:21, Andrew C Aitchison
<[email protected]<mailto:[email protected]>> wrote:
How are you measuring free RAM at this point ?
The GDAL cache will be included in the the memory use reported by
the python interpreter, but at least on Linux the system command
"free" also reports the memory used by the file-system cache, which
will contain (the read parts of) the file, until the system decides
to use that RAM for something else. Your tempfile is likely being
stored in the filesystem cache.
_______________________________________________
gdal-dev mailing list
[email protected]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
