Hi,
A look at memory usage in the posted test cases indicates that address
space fragmentation is a major factor at least in my setup
Python 3.4.1 |Anaconda 2.1.0 (32-bit)| (default, Jun 11 2014, 17:29:32)
[MSC v.1600 32 bit (Intel)] running on 64 bit Windows 7.
Such fragmention is described in, e.g.,
http://www.mgroeber.de/misc/windows_heap.html
The statistics in the following listing are from VMMap.exe from Microsoft
Sysinternals.
Spyder started
- 1734 MB free address space in 91 blocks, largest block 883 MB
In [1]: from numpy import *
- 1741 MB free address space in 86 blocks, largest block 883 MB
In [2]: a_flt = ones((7000,7000), dtype=float64)+4
- 1358 MB free address space in 87 blocks, largest block 382 MB
In [3]: b_flt = ones((7000,7000), dtype=float64)+1
- Raises MemoryError
- 1358 MB free address space in 87 blocks, largest block 382 MB
- The next largest free blocks: 207 MB, 173 MB, 117 MB, 115 MB, ...
In [4]: b_flt = ones((7000,7000), dtype=float64)
- 975 MB free address space in 86 blocks, largest block 207 MB
In [5]: b_flt += 1
- 975 MB free address space in 86 blocks, largest block 207 MB
The numbers vary slightly in consecutive runs. The next case:
Spyder started
- 1741 MB free address space in 81 blocks, largest block 880 MB
In [1]: import random
- 1741 MB free address space in 81 blocks, largest block 880 MB
In [2]: import struct
- 1741 MB free address space in 81 blocks, largest block 880 MB
In [3]: intlist = [int(100*random.random()) for _ in range(33000000)]
- 1612 MB free address space in 81 blocks, largest block 880 MB
In [4]: buf = struct.pack('%sh' %len(intlist), *intlist)
- 1450 MB free address space in 81 blocks, largest block 784 MB
- The next largest free blocks: 173 MB, 117 MB, 78 MB, ...
VMMap.exe (View --> Fragmentation View) shows that DLL's are loaded in a
pretty fragmented pattern. After running the second case there are 1171
separate blocks belonging to executable images. The most fragmented module
is PyQt4\QtGui.pyd: 69 separate blocks.
The main difference I see between running in Spyder and otherwise is the
number of DLL's loaded. I don't know if anything can be done about that,
except for loading fewer modules (and of course, switching to 64 bits
and/or to Linux). A similar issue has been discussed in the Python bug
tracker http://bugs.python.org/issue19246. It was closed with a comment:
"So the issue comes the Windows heap allocator. I don't see any obvious
improvment that Python can do to improve the memory usage. I close the issue."
Cheers,
Olli
--
You received this message because you are subscribed to the Google Groups
"spyder" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/spyderlib.
For more options, visit https://groups.google.com/d/optout.
