On Feb 22, 10:53 am, a bunch of folks wrote: > Memory is basically free.
This is true if you are simply scanning a file into memory. However, I'm storing the contents in some in-memory data structures and doing some data manipulation. This is my speculation: Several small objects per scanned line get allocated, and then unreferenced. If the heap is relatively small, GC has to do some work in order to make space for subsequent scan results. At some point, it realises it cannot keep up and has to extend the heap. At this point, VM and physical memory is committed, since it needs to be used. And this keeps going on. At some point, GC will take a good deal of time to compact the heap, since I and loading in so much data and creating a lot of smaller objects. If I could have a heap that is larger and does not need to be dynamically extended, then the Python GC could work more efficiently. Interesting discussion. Cheers, Andy -- -- http://mail.python.org/mailman/listinfo/python-list
