On Mon, Nov 9, 2015 at 2:02 AM, Craig Ringer <cr...@2ndquadrant.com> wrote: > On 25 June 2015 at 00:52, Robert Haas <robertmh...@gmail.com> wrote: >> I agree that we can do much better at testing than we traditionally >> have done, and I think pretty much everyone in the room for the >> developer unconference session on testing at PGCon was also in >> agreement with that. I really like the idea of taking purpose-built >> testing frameworks - like the one that Heikki created for the WAL >> format changes - and polishing them to the point where they can go >> into core. That's more work, of course, but very beneficial. > > Something that'd really help with that, IMO, would be weakening the > requirement that everything be C or be core Perl. Instead require that > configure detect whether or not facilities for some tests are present, > and have them fail with a clean warning indicating they were skipped > for lack of pre-requisites at 'make' time. > > I don't see that as significantly different to having some buildfarm > animals not bother to configure or test the PLs, SSL, etc. I > understand why adding to the mix required for the core server build > isn't acceptable, but hopefully separate test suites can be more > flexible. A free-for-all of languages and tools doesn't make sense, > but I'd like to see, say, python and the 'unittest' module added, and > to see acceptance of tests using psycopg2 to stream and decode WAL > from a slot. > > Thoughts?
I actually kind of hate this sort of thing. It's already the case that some of the TAP tests are skilled if you don't have the prerequisites, and while that solves the problem of spurious test failures, it also means that the tests which have those dependencies are run in fewer and fewer places. Now I'm not dead set against changing anything at all here, but I want to point out that we just added the TAP framework and already there are proposals (like Kevin's snapshot too old patch) to require DBD::Pg for some tests, which a lot of people aren't going to have, and I know you and probably a few other people would like python, which is fair enough, but other people are going to want other things, and pretty soon we we end up with a situation - if we're not already there - where for a developer or packager to get a machine to the point where it runs "all the tests" is a major undertaking. Accepting more tools also increases the breadth of knowledge expected of committers and patch authors. If I commit a patch and it turns out something breaks, I'm expected to fix that. Now if it's in C or Perl, I can. If it's in Python, I can't. If we add Python to the list of things we test with, there will (I'm sure) be other committers who can fix the C and Python stuff, but have no clue about the Perl. The more frameworks we support, the worse this gets. And woe betide a patch author whose feature requires adjustment of existing tests - if that person actually notices the problem before commit, as opposed to the buildfarm finding it, that person now needs to learn enough of every language in which we support tests to update the existing tests and add any new ones which are needed. I'm starting to get the hang of isolation tests, but I still don't *really* understand TAP tests, really, and anything new we add is a whole new obstacle. I think this is sort of an inherent problem with long-running projects. Everybody knows, for example, that the toolchain we're using to build our documentation is pretty crufty. But if we switched to something else that used a different input format, then every back-patchable doc fix would have to be done twice, once for the old toolchain and once for the new, which means that at least some people would have to understand both, for a good 5-6 years. We'd also have to get that new toolchain working on every platform we support, and all of our packagers would have to switch to it, and of course we'd want it to support all the output formats we have today. Yet, if we were starting over, I think there's about zero chance we'd pick this approach. It's just that switching now is a job and a half, and we don't want to take that on likely. Similarly with testing frameworks - except those are even worse, because nobody's saying that any of the stuff we have now will ever go away. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (email@example.com) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers