Your wish is my command. On Mon, Aug 8, 2011 at 10:00 AM, Robby Findler <ro...@eecs.northwestern.edu> wrote: > PS: I'm also happy if this class of tests only emails the responsible > person, and not the pusher. > > Robby > > On Mon, Aug 8, 2011 at 10:59 AM, Robby Findler > <ro...@eecs.northwestern.edu> wrote: >> I like the two-times-in-a-row thought. >> >> FWIW, please try to avoid race conditions of the second kind. >> >> I think the drracket test suites are special because they fail >> not-so-often and I don't actually know how to fix them. If either of >> those weren't true then I'd say they should just not run in drdr. (So >> the race-condition/using the same file thing fails this test.) >> >> Robby >> >> On Mon, Aug 8, 2011 at 10:56 AM, Vincent St-Amour <stamo...@ccs.neu.edu> >> wrote: >>> >>> I love DrDr, but there's a small thing that annoys me about it. >>> >>> Some tests are prone to intermittent failures. For example, some >>> benchmarks need to create a file, and several benchmarks share the >>> same file, which leads to race conditions. Similarly, some DrRacket >>> tests sometimes fail for focus reasons. >>> >>> So, whenever someone pushes, they may get failures from these tests, >>> then have go look at the actual errors, and try to figure out if they >>> actually broke something or not. >>> >>> (Or, they ignore these failures, which is bad.) >>> >>> Here are two potential solutions. Let's assume that I just pushed >>> something, and a test started failing. >>> >>> - Have DrDr send me email for every push about the broken test for as >>> long as it fails. If I get email more than once, it's likely that I >>> actually broke something. If I only get email once, the problem went >>> away on its own, and was likely an intermittent failure. >>> >>> - Have the possiblity to flag some tests as intermittent (something >>> like `drdr:random'), and only report failures for these tests if >>> they fail twice in a row. This would reduce the amount of noise, >>> since I expect most of these tests to pass most of the time. Actual >>> breakage would still be detected, since it's unlikely that such >>> failures would go away on their own. Detection would happen one push >>> late, but that shouldn't be too much of an issue. >>> >>> Or, maybe only notify the pusher after two failures in a row, but >>> notify the responsible person right away. >>> >>> Any thoughts? >>> >>> Vincent >>> _________________________________________________ >>> For list-related administrative tasks: >>> http://lists.racket-lang.org/listinfo/dev >>> >> > > _________________________________________________ > For list-related administrative tasks: > http://lists.racket-lang.org/listinfo/dev
-- Jay McCarthy <j...@cs.byu.edu> Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay "The glory of God is Intelligence" - D&C 93 _________________________________________________ For list-related administrative tasks: http://lists.racket-lang.org/listinfo/dev
