Re: Refactoring of the test suite
Mike Looijmans wrote: Oh and if we are refactoring the tests, I want a "make tests" rule. I'm tired of doing: ./configure; make; sudo make install; make tests; DOH! cd test; python test.py. :) Make that "make check" (like autotools), to not confuse old-skool autoconfers like myself. That works for me. Jim
Re: Refactoring of the test suite
Oh and if we are refactoring the tests, I want a "make tests" rule. I'm tired of doing: ./configure; make; sudo make install; make tests; DOH! cd test; python test.py. :) Make that "make check" (like autotools), to not confuse old-skool autoconfers like myself.
Re: Refactoring of the test suite
My only comment is that, especially if tests are being split into separate files, that the access/error log files be distinct for each test with a name incorporating the name of the test. I always found it a pain to try and dig through a big log of multiple tests trying to work out which was the output from the one that was failing. Graham On 11/02/2006, at 2:23 AM, Jim Gallacher wrote: Nicolas Lehuen wrote: Hi, There is something I'd like to do for the 3.3 version : it is to refactor the test suite. It's more a chore than real development, but the current test suite is slowly becoming big and quite difficult to maintain. Also, I think the size and complexity may be intimidating to new contributors. I don't mind so much now that I'm familiar with it but when you first dive in it looks like a lot of code. We want the tests themselves to be as robust as possible and if refactoring helps I'm all for it. We've had a few problems where the a unit test was incorrect and as a result was missing mod_python problems. What I'd like to do is simply split the test runner and the published tests in various parts, matching the different functionalities that are tested. What would be great, then, is to introduce platform-specific tests, of even MPM-specific tests, so that we could exercise some parts of the code that are only available with a threaded MPM (for example, the MemorySession implementation, or the publisher reload test which I erroneously implemented with a threaded MPM in mind). +1 One thing that I'd like to settle with you is whether you are OK to split the big PerRequestTestCase class into multiple classes, to ease maintenance and configuration switches. This has the downside that Apache server is likely to be stopped and restarted a few more times during the tests, so the tests will take more time to run. On the other hand, I'd like to introduce a way to select the tests to run when launching the test suite, so that we won't have to wait for the whole test suite to pass while debugging a given problem. I think this is a good idea, and I've been thinking along the same lines in light of the changes that are necessary to support apache 2.2 (ie, different test configurations required for 2.0 and 2.2). I'm not too concerned about taking more time for the tests to complete. Anything we can do to make the tests more reliable and easier to write will be worth the extra testing time and the initial refactoring effort. Ah, and one thing I'd definitely get rid of - usage of backquotes like in `rsp`. I really don't like this magic quoting thingy, it reminds me all too much of PHP :). Well, we can't have that now can we? :) I don't know if this would be practical, but one idea I have is to put each test in it's own file in a test_modules direcotry. This directory would be scanned on test startup and the tests added dynamically. That way if someone contributes some new code, they can also provide the corresponding unit test without worrying about editing the unwieldy test.py and htdocs/tests.py files. All they need to do is create test_modules/mytest.py and htdocs/mytest.py files which get dropped into the test framework with no additional configuration required. Oh and if we are refactoring the tests, I want a "make tests" rule. I'm tired of doing: ./configure; make; sudo make install; make tests; DOH! cd test; python test.py. :) Jim
Re: Refactoring of the test suite
Nicolas Lehuen wrote: Hi, There is something I'd like to do for the 3.3 version : it is to refactor the test suite. It's more a chore than real development, but the current test suite is slowly becoming big and quite difficult to maintain. Also, I think the size and complexity may be intimidating to new contributors. I don't mind so much now that I'm familiar with it but when you first dive in it looks like a lot of code. We want the tests themselves to be as robust as possible and if refactoring helps I'm all for it. We've had a few problems where the a unit test was incorrect and as a result was missing mod_python problems. What I'd like to do is simply split the test runner and the published tests in various parts, matching the different functionalities that are tested. What would be great, then, is to introduce platform-specific tests, of even MPM-specific tests, so that we could exercise some parts of the code that are only available with a threaded MPM (for example, the MemorySession implementation, or the publisher reload test which I erroneously implemented with a threaded MPM in mind). +1 One thing that I'd like to settle with you is whether you are OK to split the big PerRequestTestCase class into multiple classes, to ease maintenance and configuration switches. This has the downside that Apache server is likely to be stopped and restarted a few more times during the tests, so the tests will take more time to run. On the other hand, I'd like to introduce a way to select the tests to run when launching the test suite, so that we won't have to wait for the whole test suite to pass while debugging a given problem. I think this is a good idea, and I've been thinking along the same lines in light of the changes that are necessary to support apache 2.2 (ie, different test configurations required for 2.0 and 2.2). I'm not too concerned about taking more time for the tests to complete. Anything we can do to make the tests more reliable and easier to write will be worth the extra testing time and the initial refactoring effort. Ah, and one thing I'd definitely get rid of - usage of backquotes like in `rsp`. I really don't like this magic quoting thingy, it reminds me all too much of PHP :). Well, we can't have that now can we? :) I don't know if this would be practical, but one idea I have is to put each test in it's own file in a test_modules direcotry. This directory would be scanned on test startup and the tests added dynamically. That way if someone contributes some new code, they can also provide the corresponding unit test without worrying about editing the unwieldy test.py and htdocs/tests.py files. All they need to do is create test_modules/mytest.py and htdocs/mytest.py files which get dropped into the test framework with no additional configuration required. Oh and if we are refactoring the tests, I want a "make tests" rule. I'm tired of doing: ./configure; make; sudo make install; make tests; DOH! cd test; python test.py. :) Jim
Refactoring of the test suite
Hi, There is something I'd like to do for the 3.3 version : it is to refactor the test suite. It's more a chore than real development, but the current test suite is slowly becoming big and quite difficult to maintain. What I'd like to do is simply split the test runner and the published tests in various parts, matching the different functionalities that are tested. What would be great, then, is to introduce platform-specific tests, of even MPM-specific tests, so that we could exercise some parts of the code that are only available with a threaded MPM (for example, the MemorySession implementation, or the publisher reload test which I erroneously implemented with a threaded MPM in mind). One thing that I'd like to settle with you is whether you are OK to split the big PerRequestTestCase class into multiple classes, to ease maintenance and configuration switches. This has the downside that Apache server is likely to be stopped and restarted a few more times during the tests, so the tests will take more time to run. On the other hand, I'd like to introduce a way to select the tests to run when launching the test suite, so that we won't have to wait for the whole test suite to pass while debugging a given problem. Ah, and one thing I'd definitely get rid of - usage of backquotes like in `rsp`. I really don't like this magic quoting thingy, it reminds me all too much of PHP :). Like Graham says : "comments ?" Regards, Nicolas