On Fri, Apr 21, 2017 at 10:34:58PM -0700, Andres Freund wrote: > I've a bunch of tests, but I don't quite know whether we can expose all > of them via classical tests. There are several easy ones that I > definitely want to add (import "empty" snapshot; import snapshot with > running xacts; create snapshot, perform some ddl, import snapshot, > perform some ddl, check things work reasonably crazy), but there's > enough others that are just probabilistic. I was wondering about adding > a loop that simply runs for like 30s and then quits or such, but who > knows.
If the probabilistic test catches the bug even 5% of the time in typical configurations, the buildfarm will rapidly identify any regression. I'd choose a 7s test that detects the bug 5% of the time over a 30s test that detects it 99% of the time. (When I wrote src/bin/pgbench/t/001_pgbench.pl for a probabilistic bug, I sized that test to finish in 1s and catch its bug half the time. In its case, only two buildfarm members were able to demonstrate the original bug, so 5% detection would have been too low.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
