Re: Python Memory Manager

2008-02-20 Thread MartinRinehart
Jeff Schwab wrote: What's the Intel architecture? Do you mean the x86_64 architecture that was actually developed by AMD, or x86 for x some number, or do you actually mean IA64? I mean chips of the family that goes back to the 8086 and 8088 chips, chips that support the REPZ prefix to the

Re: Python Memory Manager

2008-02-20 Thread Paul Rubin
[EMAIL PROTECTED] writes: I mean chips of the family that goes back to the 8086 and 8088 chips, chips that support the REPZ prefix to the MOVSW instruction. repz movsw is a pretty lame way to copy data on current x86's. Use XMM instead. -- http://mail.python.org/mailman/listinfo/python-list

Re: Python Memory Manager

2008-02-20 Thread MartinRinehart
Paul Rubin wrote: repz movsw is a pretty lame way to copy data on current x86's. Use XMM instead. Thank you, Paul. I'm pretty sure you meant MMX, Multi-Media eXtensions. Paul's just told me to upgrade my 32-bit thinking to use newer 64-bit registers, even on a 32-bit cpu. Please divide my

Re: Python Memory Manager

2008-02-20 Thread MartinRinehart
Steve Holden wrote: You have a strange idea of nearly free ... Extending an integer array from 100 to 150 items is a pretty puny operation when you compare it with the amount of data that might need to be moved during a compactifying garbage collection of a 20MB Python program image. 20

Re: Python Memory Manager

2008-02-20 Thread Paul Rubin
[EMAIL PROTECTED] writes: Use XMM instead. Thank you, Paul. I'm pretty sure you meant MMX, Multi-Media eXtensions. MMX is the obsolete 64 bit predecessor to XMM. XMM encompasses 128 bit wide MMX-like integer instructions and several generations of SSE floating point ops. Main thing about

Re: Python Memory Manager

2008-02-20 Thread Ross Ridge
[EMAIL PROTECTED] wrote: 20 MBs = 5 M 32-bit words = 1.25 millis to move half of them on a 2GHz machine. Don't know how much a milli costs where you live. A 2GHz machine doesn't have 20Mb of 2GHz memory. You made the mistake of measuring the speed of processor's cache, rather than the RAM.

Re: Python Memory Manager

2008-02-18 Thread MartinRinehart
Paul Rubin wrote: The problem here is with a high allocation rate, you have to GC a lot more often, which typically involves copying live data. This is last century's issue. Copying data, RAM to RAM, is nearly free using the Intel architecture. This short article,

Re: Python Memory Manager

2008-02-18 Thread Steve Holden
[EMAIL PROTECTED] wrote: Paul Rubin wrote: The problem here is with a high allocation rate, you have to GC a lot more often, which typically involves copying live data. This is last century's issue. Copying data, RAM to RAM, is nearly free using the Intel architecture. This short

Re: Python Memory Manager

2008-02-18 Thread rbossy
Quoting Steve Holden [EMAIL PROTECTED]: [...] Not only that, but all pointers to an object have to be updated when it is relocated. Any problem in computer science can be solved by another level of indirection -- David John Wheeler ;-) RB --

Re: Python Memory Manager

2008-02-18 Thread Jeff Schwab
[EMAIL PROTECTED] wrote: Paul Rubin wrote: The problem here is with a high allocation rate, you have to GC a lot more often, which typically involves copying live data. This is last century's issue. Copying data, RAM to RAM, is nearly free using the Intel architecture. What's the Intel

Python Memory Manager

2008-02-17 Thread Pie Squared
I've been looking at the Python source code recently, more specifically trying to figure out how it's garbage collector works. I've gathered that it uses refcounting as well as some cycle-detection algorithms, but I haven't been able to figure out some other things. Does Python actually have a

Re: Python Memory Manager

2008-02-17 Thread Paul Rubin
Pie Squared [EMAIL PROTECTED] writes: Also, if it does, how does it deal with memory segmentation? This question bothers me because I've been trying to implement a moving garbage collector, and am not sure how to deal with updating all program pointers to objects on the heap, and thought

Re: Python Memory Manager

2008-02-17 Thread Steve Holden
Paul Rubin wrote: Pie Squared [EMAIL PROTECTED] writes: Also, if it does, how does it deal with memory segmentation? This question bothers me because I've been trying to implement a moving garbage collector, and am not sure how to deal with updating all program pointers to objects on the

Re: Python Memory Manager

2008-02-17 Thread Pie Squared
On Feb 17, 1:57 pm, Paul Rubin http://[EMAIL PROTECTED] wrote: Pie Squared [EMAIL PROTECTED] writes: Also, if it does, how does it deal with memory segmentation? This question bothers me because I've been trying to implement a moving garbage collector, and am not sure how to deal with

Re: Python Memory Manager

2008-02-17 Thread MartinRinehart
I researched this for some Java I wrote. Try to avoid shuffling physical memory - you'll write a lot less code and it will be faster, too. Use an allocated list and an available list. Keep them in address order. Inserting (moving list elements from insertion point to end) and deleting

Re: Python Memory Manager

2008-02-17 Thread Christian Heimes
Pie Squared wrote: I've been looking at the Python source code recently, more specifically trying to figure out how it's garbage collector works. I've gathered that it uses refcounting as well as some cycle-detection algorithms, but I haven't been able to figure out some other things.

Re: Python Memory Manager

2008-02-17 Thread Pie Squared
On Feb 17, 3:05 pm, [EMAIL PROTECTED] wrote: I researched this for some Java I wrote. Try to avoid shuffling physical memory - you'll write a lot less code and it will be faster, too. Use an allocated list and an available list. Keep them in address order. Inserting (moving list elements

Re: Python Memory Manager

2008-02-17 Thread Paul Rubin
Pie Squared [EMAIL PROTECTED] writes: It seems to me that another, perhaps better strategy, would be to allocate a large heap space, then store a pointer to the base of the heap, the current heap size, and the beginning of the free memory. When you need to 'allocate' more room, just return a

Re: Python Memory Manager

2008-02-17 Thread thebjorn
On Feb 17, 10:01 pm, Pie Squared [EMAIL PROTECTED] wrote: [...] It seems to me that another, perhaps better strategy, would be to allocate a large heap space, then store a pointer to the base of the heap, the current heap size, and the beginning of the free memory. When you need to 'allocate'

Re: Python Memory Manager

2008-02-17 Thread Martin v. Löwis
Also, if it does, how does it deal with memory segmentation? This question bothers me because I've been trying to implement a moving garbage collector, and am not sure how to deal with updating all program pointers to objects on the heap, and thought perhaps an answer to this question would

Re: Python Memory Manager

2008-02-17 Thread Paul Rubin
Martin v. Löwis [EMAIL PROTECTED] writes: That's not exactly true, i.e. it isn't mark-and-sweep, but some similar scheme that allows incremental collection without write barriers. This particular scheme heavily relies on refcounting itself (specifically, an object is garbage in a certain

Re: Python Memory Manager

2008-02-17 Thread greg
Paul Rubin wrote: As I understand it, Python primarily uses reference counting, with a mark and sweep scheme for cycle breaking tacked on as an afterthought. It's not mark-and-sweep, it's a cycle detector. It goes through all allocated objects of certain types, and all objects reachable from

Re: Python Memory Manager

2008-02-17 Thread greg
Christian Heimes wrote: In release builds PyObject_HEAD only contains the ref count and a link to the object type. In Py_DEBUG builds it also contains a double linked list of all allocated objects to debug reference counting bugs. There's also a doubly-linked list used by the cycle detector,