Vladimir Marangozov wrote:
May I chime in? :-)
Please do!
Gents, the current implementation of Python's memory management
is really fine and most problems it used to have in the past
have been fixed in recent years (at least the ones I know of).
Probably the only one left, indeed, is the
On Fri, Feb 22, 2008, Andrew MacIntyre wrote:
Vladimir Marangozov wrote:
And, of course, if the int/float freelist scheme was a real issue
we would have probably heard of it by now in a very sound way.
Not much noise has been made here, but I've come across 2 separate
complaints in
And, of course, if the int/float freelist scheme was a real issue
we would have probably heard of it by now in a very sound way.
As Aahz says: people run into this problem frequently, and then report it.
Regards,
Martin
___
Python-Dev mailing list
Andrew MacIntyre wrote:
M.-A. Lemburg wrote:
If we're down to voting, here's my vote:
+1 on removing the freelists from ints and floats, but not the
small int sharing optimization
+1 on focusing on improving pymalloc to handle int and float
object allocations even better
-1 on
On Wed, Feb 20, 2008, Andrew MacIntyre wrote:
I've now written up my testing and attached the write-up to issue 2039
(http://bugs.python.org/issue2039).
Nice work! Too often we (generic we) don't try to re-simplify code.
--
Aahz ([EMAIL PROTECTED]) *
Christian Heimes [EMAIL PROTECTED] wrote:
+1 on focusing on improving pymalloc to handle int and float
object allocations even better
I wonder if the int and float types could use a faster internal
pymalloc interface. I can't remember the details but I seem to
recall that pymalloc must jump
On 2008-02-13 08:02, Andrew MacIntyre wrote:
Christian Heimes wrote:
Andrew MacIntyre wrote:
I tried a LIFO stack implementation (though I won't claim to have done it
well), and found it slightly slower than no freelist at all. The
advantage of such an approach is that the known size of the
Christian Heimes wrote:
Andrew MacIntyre wrote:
I tested the updated patch you added to issue 2039. With the int
freelist set to 500 and the float freelist set to 100, its about the same
as the no-freelist version for my tests, but PyBench shows the simple
float arithmetic to be about 10%
On 2008-02-13 12:56, Andrew MacIntyre wrote:
I'm not that interested in debating the detail of exactly how big the
prospective LIFO freelists are - I just want to see the situation
resolved with maximum utilisation of memory for minimum performance
penalty. To that end, +1 from me for
M.-A. Lemburg wrote:
On 2008-02-13 12:56, Andrew MacIntyre wrote:
I'm not that interested in debating the detail of exactly how big the
prospective LIFO freelists are - I just want to see the situation
resolved with maximum utilisation of memory for minimum performance
penalty. To that end,
M.-A. Lemburg wrote:
+1 on removing the freelists from ints and floats, but not the
small int sharing optimization
+1 on focusing on improving pymalloc to handle int and float
object allocations even better
-1 on changing the freelist implementations to use pymalloc for
Christian Heimes wrote:
By the way objects are always aligned at 8 byte address boundaries. A 12
or 4 bytes object occupies 16 bytes.
Maybe pymalloc should be enhanced so it can cope with
certain odd-sized objects, such as 12 bytes?
--
Greg
___
Christian Heimes wrote:
By the way objects are always aligned at 8 byte address boundaries. A 12
or 4 bytes object occupies 16 bytes.
No, with PyMalloc a 4 byte object occupies 8 bytes (see the comments at
the top of Objects/obmalloc.c).
Cheers,
Andrew.
--
Andrew MacIntyre wrote:
By the way objects are always aligned at 8 byte address boundaries. A 12
or 4 bytes object occupies 16 bytes.
No, with PyMalloc a 4 byte object occupies 8 bytes (see the comments at
the top of Objects/obmalloc.c).
I know. It's a typo. It should read 12 or 14 byte
Andrew MacIntyre wrote:
I seem to recall Tim Peters paying a lot of attention to cache effects
when he went over the PyMalloc code before the 2.3 release, which would
contribute to its performance.
Well, I guess we have to ask Uncle Tim on this matter and ask for his
opinion. I've no
Andrew MacIntyre wrote:
I tried a LIFO stack implementation (though I won't claim to have done it
well), and found it slightly slower than no freelist at all. The
advantage of such an approach is that the known size of the stack makes
deallocating excess objects easy (and thus no need for
I've done my own profiling with a different script. I was able to verify
Andrews results. The pymalloc approach makes int creation slightly
slower and has almost no effect on floats.
The creation of 1000 times a list of 1000 ints (range(1000)) is about
20msec slower. It almost doubles the time
Well, yes, it doesn't run out of memory, but if pymalloc needs
to allocate lots of objects of the same size, then performance
degrades due to the management overhead involved for checking
the free pools as well as creating new arenas as needed.
To reduce this overhead, it may be a good idea
M.-A. Lemburg wrote:
On 2008-02-07 14:09, Andrew MacIntyre wrote:
Probably in response to the same stimulus as Christian it occurred to me
that the freelist approach had been adopted long before PyMalloc was
enabled as standard (in 2.3), and that much of the performance gains
between 2.2 and
On 2008-02-08 08:21, Martin v. Löwis wrote:
One of the hopes of having a custom allocator for Python was to be
able to get rid off all free lists. For some reason that never happened.
Not sure why. People were probably too busy with adding new
features to the language at the time ;-)
Andrew MacIntyre wrote:
For the int case, that patch is slower than no free list at all. For the
float case its very slightly faster (55.3s vs 55.5s) than no free list at
all. Slower than the trunk code for both cases. Did you run the test
scripts on your own machine?
I've used a simple
Tim Peters wrote:
pymalloc ensures 8-byte alignment. This is one plausible reason to
keep the current int free list: an int object struct holds 3 4-byte
members on most boxes (type pointer, refcount, and the int's value),
and the int freelist code uses exactly 12 bytes for each on most
Neal Norwitz wrote:
It's not just size. Architectures may require data aligned on 4, 8,
or 16 addresses for optimal performance depending on data type. IIRC,
malloc aligns by 8 (not sure if that was a particular arch or very
common). I don't know if pymalloc handles alignment.
Yes,
On 2008-02-08 19:28, Christian Heimes wrote:
In addition to the pure performance aspect, there is the issue of memory
utilisation. The current trunk code running the int test case in my
original post peaks at 151MB according to top on my FreeBSD box, dropping
back to about 62MB after the dict
M.-A. Lemburg wrote:
As you can see, Integers and floats fall into the same pymalloc size
class. What's strange in Andrew's result is that both integers
and floats use the same free list technique and fall into the same
pymalloc size class, yet the results are different.
My take is not that
Tim Peters wrote:
pymalloc ensures 8-byte alignment. This is one plausible reason to
keep the current int free list: an int object struct holds 3 4-byte
members on most boxes (type pointer, refcount, and the int's value),
and the int freelist code uses exactly 12 bytes for each on most
Martin v. Löwis wrote:
You mean, as the digits? You would have to rewrite the entire long
datatype, which is a tedious exercise. Plus, I believe you would have
to make some operations 64-bit on all systems: when you multiply
digits today, the product will be 32-bit. If you extend the digits,
[Neal Norwitz]
It's not just size. Architectures may require data aligned on 4, 8,
or 16 addresses for optimal performance depending on data type. IIRC,
malloc aligns by 8 (not sure if that was a particular arch or very
common).
Just very common. Because malloc has no idea what the pointer
Christian Heimes wrote:
Andrew MacIntyre wrote:
For the int case, that patch is slower than no free list at all. For the
float case its very slightly faster (55.3s vs 55.5s) than no free list at
all. Slower than the trunk code for both cases. Did you run the test
scripts on your own
On 2008-02-07 14:09, Andrew MacIntyre wrote:
Probably in response to the same stimulus as Christian it occurred to me
that the freelist approach had been adopted long before PyMalloc was
enabled as standard (in 2.3), and that much of the performance gains
between 2.2 and 2.3 were in fact due
On Feb 7, 2008 5:09 AM, Andrew MacIntyre [EMAIL PROTECTED] wrote:
So I've been testing with the freelists ripped out and ints and floats
reverted to fairly standard PyObject_New allocation (retaining the small
int interning) and thus relying on PyMalloc to do any compaction.
The results have
One of the hopes of having a custom allocator for Python was to be
able to get rid off all free lists. For some reason that never happened.
Not sure why. People were probably too busy with adding new
features to the language at the time ;-)
Probably not. It's more that the free lists still
32 matches
Mail list logo
