On Fri, Jun 20, 2008 at 10:25 AM, Antoine Pitrou <[EMAIL PROTECTED]>
wrote:
>
> Kevin Jacobs <jacobs <at> bioinformed.com> <bioinformed <at> gmail.com>
> writes:
> >
> > +1 on a C API for enabling and disabling GC. I have several instances
> where
> I create a large number of objects non-cyclic objects where I see huge GC
> overhead (30+ seconds with gc enabled, 0.15 seconds when disabled).
>
> Could you try to post a stripped-down, self-contained example of such
> behaviour?
$ python -m timeit 'zip(*[range(1000000)]*5)'
10 loops, best of 3: 496 msec per loop
$ python -m timeit -s 'import gc; gc.enable()' 'zip(*[range(1000000)]*5)'
10 loops, best of 3: 2.93 sec per loop
Note that timeit cheats and disables GC by default.
Attached is a less stripped down script to demonstrate the super-linear
behavior for somewhat naively coded transpose operators. The output is
listed below:
FUNCTION ROWS COLUMNS GC ENABLED GC DISABLED
-------------- ----------- ---------- ------------ ------------
transpose_comp 5 0 0.0000 0.0000
transpose_comp 5 250000 0.5535 0.3537
transpose_comp 5 500000 1.4359 0.6868
transpose_comp 5 750000 2.7148 1.0760
transpose_comp 5 1000000 3.8070 1.3936
transpose_comp 5 1250000 5.5184 1.7617
transpose_comp 5 1500000 7.8828 2.1308
transpose_comp 5 1750000 9.3279 2.5364
transpose_comp 5 2000000 11.8248 2.7399
transpose_comp 5 2250000 14.7436 3.1585
transpose_comp 5 2500000 18.4452 3.5818
transpose_comp 5 2750000 21.4856 3.8988
transpose_comp 5 3000000 24.4110 4.3148
transpose_zip 5 0 0.0000 0.0000
transpose_zip 5 250000 0.2537 0.0658
transpose_zip 5 500000 0.8380 0.1324
transpose_zip 5 750000 1.7507 0.1989
transpose_zip 5 1000000 2.6169 0.2648
transpose_zip 5 1250000 4.0760 0.3317
transpose_zip 5 1500000 5.8852 0.4145
transpose_zip 5 1750000 7.3925 0.5161
transpose_zip 5 2000000 10.0755 0.6708
transpose_zip 5 2250000 14.2698 0.7760
transpose_zip 5 2500000 16.7291 0.9022
transpose_zip 5 2750000 20.3833 1.0179
transpose_zip 5 3000000 24.5515 1.0971
Hope this helps,
-Kevin
import gc
import time
def transpose_comp(rows):
return [ [ row[i] for row in rows ] for i in xrange(len(rows[0])) ]
def transpose_zip(rows):
return zip(*rows)
def bench(func, rows, cols):
gc.enable()
gc.collect()
data = [ range(cols) ]*rows
t0 = time.time()
func(data)
t1 = time.time()
gc.disable()
func(data)
t2 = time.time()
gc.enable()
return t1-t0,t2-t1
print 'FUNCTION ROWS COLUMNS GC ENABLED GC DISABLED '
print '-------------- ----------- ---------- ------------ ------------'
rows = 5
for func in [transpose_comp,transpose_zip]:
for cols in range(0,3000001,250000):
enabled,disabled = bench(func, rows, cols)
print '%-15s %10d %10d %12.4f %12.4f' % \
(func.func_name,rows,cols,enabled,disabled)
_______________________________________________
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
