On 2017-05-01 12:32:07 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-05-01 08:46:47 -0400, Tom Lane wrote: > >> 30sec is kind of a big lump from a buildfarm standpoint, especially if > >> you mean "it runs for 30s on my honkin' fast workstation". I'm fine > >> with individual tests that run for ~ 1sec. > > > I was more thinking of pgench -T$XX, rather than constant number of > > iterations. I currently can reproduce the issues within like 3-4 > > minutes, so 5s is probably not quite sufficient to get decent coverage. > > Adding a five-minute pgbench run to the buildfarm sequence is definitely > going to get you ridden out of town on a rail.
Right - that was referring to Noah's comment upthread: On 2017-04-29 14:42:01 -0700, Noah Misch wrote: > If the probabilistic test catches the bug even 5% of the time in typical > configurations, the buildfarm will rapidly identify any regression. I'd > choose a 7s test that detects the bug 5% of the time over a 30s test that > detects it 99% of the time. (When I wrote src/bin/pgbench/t/001_pgbench.pl > for a probabilistic bug, I sized that test to finish in 1s and catch its bug > half the time. In its case, only two buildfarm members were able to > demonstrate the original bug, so 5% detection would have been too low.) and I suspect that you'd not find these within 5s within sufficient time, because the detection rate would be too low. > But quite aside from the question of whether we can afford the cycles, > it seems like the wrong approach. IMO the buildfarm is mainly for > verifying portability, not for trying to prove that race-like > conditions don't exist. In most situations we're going out of our way > to ensure reproduceability of tests we add to the buildfarm sequence; > but it seems like this is looking for irreproducible results. Yea, I wondered about that upthread as well. But the tests are quite useful nonetheless. Wonder about adding them simply as a separate target. - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
