On Mon, Apr 23, 2018 at 11:17 AM, Christophe Jaillet <christophe.jail...@wanadoo.fr> wrote: > Le 23/04/2018 à 16:00, Jim Jagielski a écrit : >> >> It seems that, IMO, if there was not so much concern about "regressions" >> in releases, this whole revisit-versioning debate would not have come up. >> This implies, to me at least, that the root cause (as I've said before) >> appears to be one related to QA and testing more than anything. Unless we >> address this, then nothing else really matters. >> >> We have a test framework. The questions are: >> >> 1. Are we using it? >> 2. Are we using it sufficiently well? >> 3. If not, what can we do to improve that? >> 4. Can we supplement/replace it w/ other frameworks? >> >> It does seem to me that each time we patch something, there should be a >> test added or extended which covers that bug. We have gotten lax in that. >> Same for features. And the more substantial the change (ie, the more core >> code it touches, or the more it refactors something), the more we should >> envision what tests can be in place which ensure nothing breaks. >> >> In other words: nothing backported unless it also involves some changes to >> the Perl test framework or some pretty convincing reasons why it's not >> required. >> > > Hi, > +1000 on my side for more tests. > > But, IMHO, the perl framework is complex to understand for most of us. > > Last week I tried to play with it. I tried to update proxy_balancer.t > because only lbmethod=byrequests is tested. > The current test itself is really simple. It just checks if the module > didn't crashed (i.e we receive 200). > I tried to extend it for the other lbmethod available. This looked as an > easy task. But figuring the relation between: > <VirtualHost proxy_http_bal1> > and > BalancerMember http://@SERVERNAME@:@PROXY_HTTP_BAL1_PORT@ > still remains a mystery to me. > > > The ./test framework could be useful as well. > At least it is written in C, so the entry ticket should be cheaper for most > of us. > But every thing can't be done with it, I guess. > Maybe, we should at least have some unit testing for each ap_ function? The > behavior of these function should not change as it can be used by 3rd party > modules.
I agree that having a quick way to make function level tests would be very helpful. It's something largely missing from httpd. (APR has more) Even in making mod_log_json, testing it via the test framework is complicated, as its not a module that changes the output of an HTTP Request, vs I could make a few quick C-based tests that make sure things are being serialized correctly very easily. > The more tests, the better, but I believe that most regressions come from > interaction between all what is possible with httpd. > A test-suite is only a test-suite. Everything can't be tested. > > > IMHO, as a minimum, all CVE should have their dedicated test which > explicitly fails with version n, and succeeds with version n+1. > It would help to make sure than known security issues don't come back. > > > > Another question with the perl framework. > Is there a way to send "invalid" data/request with it? > All, I see is some GET(...). I guess that it sends well formed date. > Checking the behavior when invalid queries are received would be great. > Some kind of RFC compliancy check. > > > just my 2c, > CJ
