Tim Peters <t...@python.org> added the comment:
There's no evidence of a Python issue here, so I recommend closing this. It's not the Python bug tracker's job to try to make sense of platform-specific reporting tools, which, as already explained, can display exceedingly confusing numbers. We (the Python project) didn't write them, have no control over what they display, and have no special insight into them. Here's a similar simple program that creates a 4 GiB block of shared memory, and passes it to 16 processes. I'm running this on Windows 10, on a machine with 4 physical cores and 16 GiB of RAM. If the memory weren't actually being shared, Windows would have killed the job (there's no "over allocation" allowed at all on Windows), because there's nowhere near 4 * 17 = 68 GiB of RAM available. Instead, the program brings the machine to its knees, but because it created 16 processes each of which is 100% CPU- and memory-bound. The Windows task manager detailed view shows less than a MiB of non-shared memory in use by each of the worker processes, with the 4 GiB in the "memory in use that can be shared with other processes" column for each worker process. I'm running lots of other stuff too, and Windows still says there's over 5 GiB of my 16 GiB of RAM free for use. Windows reporting has its own quirks. For example, the shared memory block doesn't show up at all in the workers at first. Instead the amount reported steadily increases, as the write loop in each worker process forces the OS to materialize shared pages in the worker's virtual address space (that is, Windows reports pages actually in use by a process, not virtual address space reservations). from multiprocessing import Process, shared_memory SHM_SIZE = 2 ** 32 # 4 GiB def f(name): shm = shared_memory.SharedMemory(name=name) print(f"shm of size {shm.size:,}") buf = shm.buf for i in range(shm.size): buf[i] = 123 shm.close() def main(): shm = shared_memory.SharedMemory(create=True, size=SHM_SIZE, name='shm') ps = [] for i in range(16): p = Process(target=f, args=('shm',)) p.start() ps.append(p) for p in ps: p.join() shm.close() shm.unlink() if __name__ == '__main__': main() ---------- nosy: +tim.peters _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue41587> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com