Kim Hansen wrote: >>> I tried adding the /3GB switch to boot.ini as you suggested: >>> multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP >>> Professional" /noexecute=optin /fastdetect /3GB >>> and rebooted the system. >>> >>> Unfortunately that did not change anything for me. I still hit a hard >>> deck around 1.9 GB. Strange. >>> >>> >> The 3Gb thing only works for application specifically compiled for it: >> >> http://blogs.msdn.com/oldnewthing/archive/2004/08/12/213468.aspx >> >> I somewhat doubt python is built with this, but you could check this in >> python sources to be sure, >> >> cheers, >> >> David >> > Ahh, that explains it. Thank you for that enlightening link. Anyway > would it not be worth mentioning in the memmap documentation that > there is this 32 bit limitation, or is it so straightforwardly obvious > (it was not for me) that his is the case? >
Well, the questions has popped up a few times already, so I guess this is not so obvious :) 32 bits architecture fundamentally means that a pointer is 32 bits, so you can only address 2^32 different memory locations. The 2Gb instead of 4Gb is a consequence on how windows and linux kernels work. You can mmap a file which is bigger than 4Gb (as you can allocate more than 4Gb, at least in theory, on a 32 bits system), but you cannot 'see' more than 4Gb at the same time because the pointer is too small. Raymond Chen gives an example on windows: http://blogs.msdn.com/oldnewthing/archive/2004/08/10/211890.aspx I don't know if it is possible to do so in python, though. > The reason it isn't obvious for me is because I can read and > manipulate files >200 GB in Python with no problems (yes I process > that large files), so I thought why should it not be capable of > handling quite large memmaps as well... > Handling large files is no problem on 32 bits: it is just a matter of API (and kernel/fs support). You move the file location using a 64 bits integer and so on. Handling more than 4 Gb of memory at the same time is much more difficult. To address more than 4Gb, you would need a segmented architecture in your memory handling (with a first address for a segment, and a second address for the location within one segment). cheers, David _______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
