2011/4/25 Stefan Behnel <[email protected]>: > Vitja Makarov, 25.04.2011 11:04: >> >> 2011/4/25 Stefan Behnel<[email protected]>: >>> >>> Vitja Makarov, 25.04.2011 08:19: >>>> >>>> 2011/4/25 Stefan Behnel: >>>>> >>>>> Stefan Behnel, 07.04.2011 13:52: >>>>>> >>>>>> Stefan Behnel, 07.04.2011 13:46: >>>>>>> >>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14 >>>>>>> minutes for a run to ~4 hours when we added generator support. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend >>>>>>> >>>>>>> I currently have no idea why that is (well, it's likely because we >>>>>>> compile >>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It >>>>>>> would >>>>>>> be great if someone could find the time to analyse this problem. The >>>>>>> current run time makes it basically impossible to keep these tests >>>>>>> enabled. >>>>>> >>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The >>>>>> Py2.7 >>>>>> tests take 30-45 minutes, which is very long, but not completely out >>>>>> of >>>>>> bounds. I've disabled the Py2.6 pyregr tests for now. >>>>> >>>>> There seems to be a huge memory leak which almost certainly accounts >>>>> for >>>>> this. The Python process that runs the pyregr suite ends up with about >>>>> 50GB >>>>> of memory at the end, also in the latest Py3k builds. >>>>> >>>>> I have no idea where it may be, but it started to show when we merged >>>>> the >>>>> generator support. That's where I noticed the instant jump in the >>>>> runtime. >>>> >>>> That's very strange for my branch it takes about 30 minutes that is ok. >>> >>> There's also a second path that's worth investigating. As part of the >>> merge, >>> there was another change that came in: the CythonPyregrTestCase >>> implementation. This means that the regression tests are now being run >>> differently than before. The massive memory consumption may simply be due >>> to >>> the mass of unit tests being loaded into memory. >> >> def run_test(): >> .................................. >> try: >> module = __import__(self.module) >> if hasattr(module, 'test_main'): >> module.test_main() >> except (unittest.SkipTest, support.ResourceDenied): >> result.addSkip(self, 'ok') >> >> >> It seems that all the modules stay loaded so may be they should be >> unloaded with del sys.modules[module_name]? > > (Binary) module unloading isn't really supported in CPython. There's PEP > 3121 that has the potential to change it, but it's not completely > implemented, neither in CPython nor in Cython. A major problem is that > unloading a module deletes its globals but not necessarily the code that > uses them. For example, instances of types defined in the module can still > be alive at that point. > > The way runtests.py deals with this is forking before loading a module. > However, this does not currently work with the "xmlrunner" which we use on > Hudson, so we let all tests run in a single process there. >
Btw when running plain python code with generators total ref counter doesn't get back to initial value. I tried to trace scope and generator destructors and they are run as expected. So I'm not sure about leaks in generators. -- vitja. _______________________________________________ cython-devel mailing list [email protected] http://mail.python.org/mailman/listinfo/cython-devel
