Could DrDr say "This build is not the latest" or "The latest push is 234234"?
On 08/08/2011 11:37 AM, Jay McCarthy wrote: > It is useful to test all of them to find out when errors start. It > doesn't do the newest first, because then the calculation of "new > issue" wouldn't make any sense, because you wouldn't have the previous > push's tests. > > Jay > > On Mon, Aug 8, 2011 at 11:34 AM, Jon Rafkind <rafk...@cs.utah.edu> wrote: >> Another request: could DrDr process the latest push first? Its a little >> annoying to get emails for tests that failed when the latest push fixes >> them but DrDr is so far behind. Is there any benefit to testing all the >> intermediate pushes? >> >> On 08/08/2011 09:56 AM, Vincent St-Amour wrote: >>> I love DrDr, but there's a small thing that annoys me about it. >>> >>> Some tests are prone to intermittent failures. For example, some >>> benchmarks need to create a file, and several benchmarks share the >>> same file, which leads to race conditions. Similarly, some DrRacket >>> tests sometimes fail for focus reasons. >>> >>> So, whenever someone pushes, they may get failures from these tests, >>> then have go look at the actual errors, and try to figure out if they >>> actually broke something or not. >>> >>> (Or, they ignore these failures, which is bad.) >>> >>> Here are two potential solutions. Let's assume that I just pushed >>> something, and a test started failing. >>> >>> - Have DrDr send me email for every push about the broken test for as >>> long as it fails. If I get email more than once, it's likely that I >>> actually broke something. If I only get email once, the problem went >>> away on its own, and was likely an intermittent failure. >>> >>> - Have the possiblity to flag some tests as intermittent (something >>> like `drdr:random'), and only report failures for these tests if >>> they fail twice in a row. This would reduce the amount of noise, >>> since I expect most of these tests to pass most of the time. Actual >>> breakage would still be detected, since it's unlikely that such >>> failures would go away on their own. Detection would happen one push >>> late, but that shouldn't be too much of an issue. >>> >>> Or, maybe only notify the pusher after two failures in a row, but >>> notify the responsible person right away. >>> >>> Any thoughts? >>> >>> Vincent >>> _________________________________________________ >>> For list-related administrative tasks: >>> http://lists.racket-lang.org/listinfo/dev >> _________________________________________________ >> For list-related administrative tasks: >> http://lists.racket-lang.org/listinfo/dev >> > > _________________________________________________ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev