On 1/22/16 12:14 PM, Andres Freund wrote:
On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:
Ideally reviewers shouldn't be doing any testing, because the tests
that are part of the patch should answer every question they would
have, but I don't see that happening until we have a separate
automation-only target that we don't care how long it takes to run.

I think that's completely wrong.

Yes, more tests are good, and we need a place for longer running
tests. But assuming that every patch author will create a testsuite that
covers every angle is just about akin to assuming every submitter will
deliver perfect, bug free code. And we know how well that turns out.

I think actively trying to break a feature, and postgres in general, is
one of the most important tasks of reviewers and testers. And with that
I don't mean trying to run "make check". Look e.g. at the tests Jeff
Janes has performed, what the recent plug tests of Tomas Vondra brought
to light, or at what the full page write checker tool of Heikki's

IIRC Jeff's tests are scripted and obviously the page write checker is as well. I don't recall the exact methodology Tomas was using but I suspect it could also be scripted if you had a way to pull the plug via software (via a power management unit or maybe kill -9 of a VM). All of that is stuff that can and should be automated. Presumably it won't ever be part of the Makefile tests, but that's fine. Heck, the test scripts could stay in completely separate repos.

Where the code lives isn't the issue; it's getting stuff like this automated so humans can go back do doing things that can't be automated.
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:

Reply via email to