On Jul 9, 2014, at 12:48 AM, Paolo Carlini <paolo.carl...@oracle.com> wrote: > > On 07/09/2014 03:53 AM, Tim Shen wrote: >> +// This file is for general testcases in regex. >> +// We use a single file for multiple testcases because it takes too long to >> +// compile regex for each testcase in a single file. >> +// Normal testcases in other files should be moved into this file in the >> future. > Thus the plan would be *all* the regex testcases in a single file!?!
So, there is a balance here. The way I view it, imagine you had a 1024 CPU system, do you increase the testing time? If other parts of the test suite take 4 minutes, then you can take up to 4 minutes and be covered by that other long arm. You don’t want to break them up to 2.5 seconds a testcase for the theoretic beauty of 1 regexp per test case, when you can run 1000 of them in one run, using 2.48 seconds overhead plus 0.02 seconds per case. The other limit is how big the resulting program is. Too large, and it won’t test on a small memory system. Under a meg, and most people don’t care. As it climbs above that it starts mattering more. So I endorse batching them up to a limit, more than 5 minutes, and it is certainly way too big. Under 20 seconds, and I don’t see the problem. I’d likely stop batching at the 30 seconds mark, which is 1/10 of the usual test case limit of 300 seconds. Some people think running it up to 280 seconds is good for a single test case, this is wrong. On a slow system, they blow the limit, and on a multi user system that is not completely unloaded, they blow the limit.
