Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14
minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
I currently have no idea why that is (well, it's likely because we
compile
more tests now, but Vitja's branch ran the tests in ~30 minutes). It
would
be great if someone could find the time to analyse this problem. The
current run time makes it basically impossible to keep these tests
enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
tests take 30-45 minutes, which is very long, but not completely out of
bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for
this. The Python process that runs the pyregr suite ends up with about 50GB
of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the
generator support. That's where I noticed the instant jump in the runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
Does your branch leak memory when you run a generator? Using a debug build
of CPython, running a Cython generator for the first time persistently
increases the reference counts for me (using the generators.pyx test):
Python 2.7.1+ (2.7:c821d3d335e8, Apr 22 2011, 18:37:12)
[GCC 4.4.3] on linux2
>>> import generators as G
[17021 refs]
>>> import gc
[17462 refs]
>>> gc.collect()
0
[17465 refs]
>>> gc.collect()
0
[17465 refs]
>>> list(G.with_outer(1,2,3,4)())
[1, 2, 3, 4]
[17474 refs]
>>> gc.collect()
0
[17470 refs]
It seems like this leaked five references. And it seems to be related to
the generator being inside of a closure itself:
>>> list(G.test_first_assignment())
[5, 10, (5, 10)]
[17475 refs]
>>> gc.collect()
0
[17470 refs]
back to the last value here, but:
>>> list(G.generator_nonlocal()(5))
[2, 3, 4, 5, 6]
[17481 refs]
>>> gc.collect()
0
[17476 refs]
Another six references leaked. And it's only the first time the generator
is run, running it a second time doesn't change anything:
>>> list(G.generator_nonlocal()(5))
[2, 3, 4, 5, 6]
[17481 refs]
>>> gc.collect()
0
[17476 refs]
Stefan
_______________________________________________
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
