* Tom Lane (t...@sss.pgh.pa.us) wrote: > Realistically though, how much would code coverage info have helped? > Code coverage on a back branch would not have told you about whether > it leaves blobs behind in the final regression DB state. Code coverage > on HEAD might have helped you notice some aspects of this failure, but > it would not have told you about the same query failing before 7.4 > that worked later.
The code coverage report is exactly what I was using to figure out what was being tested in pg_dump and what wasn't. Many of the tests that are included in the new TAP testing framework that I wrote for pg_dump were specifically to provide code coverage and did improve the report. If the regression tests in older versions were updated to make sure that all the capabilities of pg_dump in those versions were tested then my testing with the regression test databases would have shown that the newer version of pg_dump wasn't handling those cases correctly. That would require more comprehensive testing to be done in those back-branches though, which would require more than just the code coverage tool being included, that's true. Another approach to this would be to figure out a way for the newer testing framework in HEAD to be run against older versions, though we'd need to have a field which indicates which version of PG a given test should be run against as there are certainly tests of newer capabilities than older versions supported. Ultimately, I'm afraid we may have to just punt on the idea of this kind of testing being done using the same testing structure that exists in HEAD and is used in the buildfarm. That would be unfortunate, but I'm not quite sure how you could have a buildfarm member than runs every major version between 8.0 and HEAD and knows how to tell the HEAD build-system what all the ports are for all those versions to connect to and run tests against. Thanks! Stephen
Description: Digital signature