Tim Peters <[email protected]> added the comment:
There's no evidence of a Python issue here, so I recommend closing this. It's
not the Python bug tracker's job to try to make sense of platform-specific
reporting tools, which, as already explained, can display exceedingly confusing
numbers. We (the Python project) didn't write them, have no control over what
they display, and have no special insight into them.
Here's a similar simple program that creates a 4 GiB block of shared memory,
and passes it to 16 processes. I'm running this on Windows 10, on a machine
with 4 physical cores and 16 GiB of RAM. If the memory weren't actually being
shared, Windows would have killed the job (there's no "over allocation" allowed
at all on Windows), because there's nowhere near 4 * 17 = 68 GiB of RAM
available.
Instead, the program brings the machine to its knees, but because it created 16
processes each of which is 100% CPU- and memory-bound. The Windows task manager
detailed view shows less than a MiB of non-shared memory in use by each of the
worker processes, with the 4 GiB in the "memory in use that can be shared with
other processes" column for each worker process. I'm running lots of other
stuff too, and Windows still says there's over 5 GiB of my 16 GiB of RAM free
for use.
Windows reporting has its own quirks. For example, the shared memory block
doesn't show up at all in the workers at first. Instead the amount reported
steadily increases, as the write loop in each worker process forces the OS to
materialize shared pages in the worker's virtual address space (that is,
Windows reports pages actually in use by a process, not virtual address space
reservations).
from multiprocessing import Process, shared_memory
SHM_SIZE = 2 ** 32 # 4 GiB
def f(name):
shm = shared_memory.SharedMemory(name=name)
print(f"shm of size {shm.size:,}")
buf = shm.buf
for i in range(shm.size):
buf[i] = 123
shm.close()
def main():
shm = shared_memory.SharedMemory(create=True, size=SHM_SIZE, name='shm')
ps = []
for i in range(16):
p = Process(target=f, args=('shm',))
p.start()
ps.append(p)
for p in ps:
p.join()
shm.close()
shm.unlink()
if __name__ == '__main__':
main()
----------
nosy: +tim.peters
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue41587>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
