Hi, it's quite rare that I have a bit of time for httpd nowadays. But I want to comment on a mail that Jacob Champion wrote on -security that contains some valid points about the lack of our test framework. I am posting this to -dev with his permission.
On Wednesday, 21 December 2016 08:55:30 CET Jacob Champion wrote: > - Our APIs are really complex, and we don't really have unit tests for > them. Nor are the internal APIs documented as well as the external APIs > are. We had a few false starts for security fixes this release that were > later shown to break something else, and I think that's related. Yes, httpd lacks unit tests. One problem is that many APIs depend on very complex structs like request_rec, conn_rec, server_conf, etc. In order to write unit tests for such APIs, one would need to write quite a bit of infrastructure to set these things up. I think it would be worth the effort, but it's not a small task. As there does not seem to be anybody with enough spare time to do it, one could possibly ask someone (CII?) for funding. A possible approach would be to compile the unit tests in the server and execute them on startup if a special define is given (like the various DUMP_* defines). Not sure how to get access to all the static helper function for unit tests, though. Unless one would somehow include the tests in the same .c file. > == Tests and Test Culture == > > Tests for security fixes are especially important, to prevent > regressions from committers who weren't part of the embargoed > conversations. But our current test battery isn't deep enough to match > the complexity of the server, and many fixes for complex bugs (whether > security-related or not) are going in without test cases. > > Compared to many other test suites I have used, our tests: > - are difficult for newcomers to install, run, write, and/or extend > - are not committed in the same breath as the bugfixes or features they > test, since they exist in a separate project root > - are not low-level enough to test our C API directly, at least as far > as I can tell. (I.e. if your code can't be easily called from a module, > you're out of luck.) If the test suite would be easier to run, maybe more people would submit tests. Is there a reason why the test suite is in a separate repository? Would it help if it was moved into the normal httpd repo? Would it make sense to include it in the release tarballs, possibly including the necessary non- standard perl modules? And include it in the makefiles in a way that a user can install a set of standard perl modules (from a distribution or cpan) and then call "make test" to start it. What is in the test/ dir in the httpd repo right now seems mostly useless and could probably be removed. Another idea to make writing tests more attractive could be to somehow include it in the backporting policy. For example, if there is a test for a new feature (positive and error handling) or a bug fix, we could require only two +1s for a backport. > For a guy like me who prefers doing his own work test-driven, it's just > kinda painful. > > Furthermore, the Apache::Test project appears to be a separate thing > entirely...? I don't know *how* separate it is, but I consider that a > red flag. To write code quickly, the way you need to when faced with > immediate security problems, you need 100% control of your test suite -- > including the ability to write embargoed tests. > > To give you a concrete example: I know that one of the httpd tests I > wrote before being invited to this project was left uncommitted, because > it would have required changes to Apache::Test, and that just wasn't > worth the effort. I have never tried to change Apache::Test. But in any case, understanding it enough to fix or extend it is a significant hurdle. Another thing that is missing: A buildbot that builds current trunk (and possibly 2.x branches) and runs the test suite and alerts the dev list of regressions. I guess this "just" needs a volunteer to set it up and document it and the ASF would provide the infrastructure. Cheers, Stefan
