Andrew McIntyre wrote:
On May 10, 2005, at 11:00 AM, David Van Couvering wrote:
Again, I'm not sure why there would be "lots of emails" for a nightly build. Also, if we email failures, that would be great, as I know I personally won't be checking the test web site on a regular basis.
Let's get hypothetical and say 2 derby-dev'rs decide to run the tests nightly against their own builds (on whatever platform/jdk revision they're interested in) and send the output to the list. Now, make that 5. or 10. Imagine if, by accident, someone checks in a change that happens to breaks all of the tests, and we have 5 people sending their test result failures to the list. Suddenly, 300 people have 100 megabytes of test diffs in their inbox. Not fun.
Oh, OK, I wasn't thinking about it that way. I was thinking there is one official nightly run of regression tests, and a report is sent out by email. This report would include all covered platforms. I wasn't thinking of this as a distributed job.
But you're probably right, it makes more sense to distribute the work, and have separate derby-devrs to choose the platforms they are interested in, and have a common web site where the results are posted...
But, what if instead derby-dev'r X running tests on platform Y and JDK Z posts to the list with "there is a serious failure going on, I think it's due to {whatever} and here's a link to the failure information: .... " - that's a lot more useful then automatic notification, even in the case of failure. And, with a page on the website that can collect the locations of where specific platform/jvm tests are running, at least all of that useful test information is available without derby-dev receiving a lot of email that is only of interest to a small subsection of people. If there are serious issues on a specific platform/jvm, then the people responsible for those tests can provide a more detailed description and bug report to derby-dev than if a failure notification is mailed to the list. Well, that's my opinion, anyway
Well, it seems to me you just have a cycle of "pull, build, full regresssions, post results." If we have a powerful enough machine, it shouldn't take that long, and we'd have fresh results every, say, four hours. Not bad!
If we want a faster turnaround, we could, as you suggest, identify a subset of tests.
I don't think four hours for a full build/test cycle is unreasonable for a tinderbox, and I think with fairly recent hardware and the relaxed durability option for testing, that the time for a full build/test cycle could probably be quite a bit less than that, probably around two hours on top-of-the-line hardware. i.e. I think that running the full test suite in a tinderbox approach is a better idea than a subset of the tests.
OK, I think we're in agreement here then... Now it's just a mere matter of implementing it :)
Thanks,
David
andrew
