On 2008-02-08 12:23, M.-A. Lemburg wrote:
> On 2008-02-08 08:21, Martin v. Löwis wrote:
>>> One of the hopes of having a custom allocator for Python was to be
>>> able to get rid off all free lists. For some reason that never happened.
>>> Not sure why. People were probably too busy with adding new
>>> features to the language at the time ;-)
>> Probably not. It's more that the free lists still outperformed pymalloc.
>>
>>> Something you could try to make PyMalloc perform better for the builtin
>>> types is to check the actual size of the allocated PyObjects and then
>>> make sure that PyMalloc uses arenas large enough to hold a good quantity
>>> of them, e.g. it's possible that the float types fall into the same
>>> arena as some other type and thus don't have enough "room" to use
>>> as free list.
>> I don't think any improvements can be gained here. PyMalloc carves
>> out pools of 4096 bytes from an arena when it runs out of blocks
>> for a certain size class, and then keeps a linked list of pools of
>> the same size class. So when many float objects get allocated,
>> you'll have a lot of pools of the float type's size class.
>> IOW, PyMalloc has always enough room.
>
> Well, yes, it doesn't run out of memory, but if pymalloc needs
> to allocate lots of objects of the same size, then performance
> degrades due to the management overhead involved for checking
> the free pools as well as creating new arenas as needed.
>
> To reduce this overhead, it may be a good idea to preallocate
> pools for common sizes and make sure they don't drop under a
> certain threshold.
>
> Here's a list of a few object sizes in bytes for Python 2.5
> on an AMD64 machine:
>
>>>> import mx.Tools
>>>> mx.Tools.sizeof(int(0))
> 24
>>>> mx.Tools.sizeof(float(0))
> 24
>
> 8-bit strings are var objects:
>
>>>> mx.Tools.sizeof(str(''))
> 40
>>>> mx.Tools.sizeof(str('a'))
> 41
>
> Unicode objects use an external buffer:
>
>>>> mx.Tools.sizeof(unicode(''))
> 48
>>>> mx.Tools.sizeof(unicode('a'))
> 48
>
> Lists do as well:
>
>>>> mx.Tools.sizeof(list())
> 40
>>>> mx.Tools.sizeof(list([1,2,3]))
> 40
>
> Tuples are var objects:
>
>>>> mx.Tools.sizeof(tuple())
> 24
>>>> mx.Tools.sizeof(tuple([1,2,3]))
> 48
>
> Old style classes:
>
>>>> class C: pass
> ...
>>>> mx.Tools.sizeof(C)
> 64
>
> New style classes are a lot heavier:
>
>>>> class D(object): pass
> ...
>>>> mx.Tools.sizeof(D)
> 848
>>>> mx.Tools.sizeof(type(2))
> 848
>
>
> As you can see, Integers and floats fall into the same pymalloc size
> class. What's strange in Andrew's result is that both integers
> and floats use the same free list technique and fall into the same
> pymalloc size class, yet the results are different.
>
> The only difference that's apparent is that small integers are
> shared, so depending on the data set used for the test, fewer
> calls to pymalloc or the free list are made.
Here's the same on a 32-bit machine. Notice the differences:
>>> import mx.Tools
>>> mx.Tools.sizeof(int(0))
12
>>> mx.Tools.sizeof(float(0))
16
>>> mx.Tools.sizeof(str(''))
24
>>> mx.Tools.sizeof(str('a'))
25
>>> mx.Tools.sizeof(unicode(''))
24
>>> mx.Tools.sizeof(unicode('a'))
24
>>> mx.Tools.sizeof(list())
20
>>> mx.Tools.sizeof(list([1,2,3]))
20
>>> mx.Tools.sizeof(tuple())
12
>>> mx.Tools.sizeof(tuple([1,2,3]))
24
>>> class C: pass
>>> mx.Tools.sizeof(C)
32
>>> class D(object): pass
>>> mx.Tools.sizeof(D)
424
>>> mx.Tools.sizeof(type(2))
424
Floats use 4 bytes more than integers. However, they still fall
into the same pymalloc size class (16 bytes this time around).
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 08 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
