Quoting Nick Kew <[EMAIL PROTECTED]>: > > > > Personally, I dislike using a random number to simulate a failure rate. > > OK so far. Perhaps the fact that my degree was maths and my first job > was in stochastic simulation makes me happier than some with them:-) > > > The > > purpose of the test suite isn't to emulate the real world where you need a > > failure rate. > > I think we'd need something more sophisticated to emulate the real world. > The difficult case is when all resources on a reslist fail at once.
I don't want to emulate the real world. I want to watch the code in a controlled environment. If we can get full coverage of all edge cases in the test suite in a controlled environment, then we can be positive that we can handle anything the real world throws at the code. In reality, we will never get 100% coverage, but we can come close. > > > All you need is one resource that has failed, so that you can > > test reslist_invalidate. > > I don't think that's right. My test generates instances where the number > of resources drops below the minimum, and we see new resources being > correctly generated. A single resource failure won't catch that. True, but those are two different test cases. You are using a random number generator to get test coverage. This means that each test run is actually running a slightly different test. The current test loops through a set number of reslist entries, and generates a random number to determine if the resource should be considered valid or not. What happens if none of the random numbers trigger the invalid case? You haven't actually tested the invalidate code path. I realize that this example is incredibly unlikely, but it isn't impossible, and it is just my reasoning taken to extremes. A less extreme example has the same problem, but it isn't as pronounced. In unit testing code, you don't want use a random number generator to simulate failure (unless you are testing the random number generator). Instead, you want a controlled environment where you can re-run the tests over and over again and you are sure to get the same test every time. What happens with this test, if it fails when 100% of the resource are invalid? The current test could trigger that failuer on one run, but not the next. Tracking that down would be almost impossible. A better way to write this test is to have a function that loops through the reslist invalidating a specified number of resources. Then you can call that function three or four or five different ways. Once where it shouldn't invalidate any resources. Once where it should invalidate only one resource. Once where it invalidates enough resources that exactly min resources remain. Once where it invalidates enough so that min - 1 resources remain. Etcetera. That way, you are gauranteed to exercise all of the error paths on each run of the test suite. Using a random number generator to exercise error paths means that you _will_ miss some cases that are important to catch. Now, notice that neither the APR nor the APR-util test suites are good in this respect. We do a horrible job at exercising our code properly, which is why some of our APIs aren't actually portable in APR. This is pretty much my fault. When I created APR initially, I wrote the original test suite, and it worked well enough for my needs, because I just needed to prove that the code worked. I hadn't been converted to the need for unit tests at the time. I wasn't trying to fully exercise the code, I was just trying to get enough success that it made sense to pass the API off to David Reid so that he could port to BeOS. This meant that new APIs were added regularly without any tests at all, so that we are now playing catch-up, and failing at it. One of the problems is that writing test cases isn't exactly glamorous work, the other is that most of the people on this list are software developers, not QA people. QA people are a special breed, and having a couple on this list would make our code better. In general, software developers (myself included) aren't good at testing code, because we like to see things work not fail. > There is another issue where the whole thing fails - e.g. talking to > a backend that goes down. We need a testcase to ensure it will fail > cleanly rather than go into an infinite loop. If you'd like me to > attack that one, let me know. Yeah, that would be great. Ryan
