Vitja Makarov, 25.04.2011 11:04:
2011/4/25 Stefan Behnel<[email protected]>:
Vitja Makarov, 25.04.2011 08:19:
2011/4/25 Stefan Behnel:
Stefan Behnel, 07.04.2011 13:52:
Stefan Behnel, 07.04.2011 13:46:
I just noticed that the CPython pyregr tests have jumped up from ~14
minutes for a run to ~4 hours when we added generator support.
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
I currently have no idea why that is (well, it's likely because we
compile
more tests now, but Vitja's branch ran the tests in ~30 minutes). It
would
be great if someone could find the time to analyse this problem. The
current run time makes it basically impossible to keep these tests
enabled.
Ok, it looks like this is mostly an issue with the Py2.6 tests. The
Py2.7
tests take 30-45 minutes, which is very long, but not completely out of
bounds. I've disabled the Py2.6 pyregr tests for now.
There seems to be a huge memory leak which almost certainly accounts for
this. The Python process that runs the pyregr suite ends up with about
50GB
of memory at the end, also in the latest Py3k builds.
I have no idea where it may be, but it started to show when we merged the
generator support. That's where I noticed the instant jump in the
runtime.
That's very strange for my branch it takes about 30 minutes that is ok.
There's also a second path that's worth investigating. As part of the merge,
there was another change that came in: the CythonPyregrTestCase
implementation. This means that the regression tests are now being run
differently than before. The massive memory consumption may simply be due to
the mass of unit tests being loaded into memory.
def run_test():
..................................
try:
module = __import__(self.module)
if hasattr(module, 'test_main'):
module.test_main()
except (unittest.SkipTest, support.ResourceDenied):
result.addSkip(self, 'ok')
It seems that all the modules stay loaded so may be they should be
unloaded with del sys.modules[module_name]?
(Binary) module unloading isn't really supported in CPython. There's PEP
3121 that has the potential to change it, but it's not completely
implemented, neither in CPython nor in Cython. A major problem is that
unloading a module deletes its globals but not necessarily the code that
uses them. For example, instances of types defined in the module can still
be alive at that point.
The way runtests.py deals with this is forking before loading a module.
However, this does not currently work with the "xmlrunner" which we use on
Hudson, so we let all tests run in a single process there.
Stefan
_______________________________________________
cython-devel mailing list
[email protected]
http://mail.python.org/mailman/listinfo/cython-devel
