On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano <gits...@pobox.com> wrote: > Ævar Arnfjörð Bjarmason <ava...@gmail.com> writes: > >>> Well, it is one thing to place git-annex under CI to make sure its >>> latest and greatest works together well with our latest and greatest >>> (and it may be something we want to see happen), but driving its >>> tests from our testsuite sounds like a tail wagging the dog, at >>> least to me. >> >> To me this is just a question of: >> >> * Is it the case that git-annex tests for a lot of edge cases we don't >> test for: Yes, probably. As evidenced by them spotting this >> regression, and not us. > > And I'd encourage them to keep doing so.
The point of this patch is that we can do this more systematically and reliably, not have people discover this sort of thing after a major release. I.e. we can be pro-active about this instead of waiting for bug reports to roll in. >> * We can (and should) add a test for the specific breakage we caused >> in 2.13.0, but that's no replacement for other things annex may be >> covering & we may be missing which'll catch future breakages. >> >> * It's a pretty established practice to test a library (git) along >> with its consumers (e.g. annex) before a major release. > > I am not so sure about the division of labor. What you are > advocating would work _ONLY_ if we test with a perfect & bug-free > version of the consumers. If they are also a moving target, then > I do not think it is worth it. After all, we are *not* in the > business of testing these consumers. > > Unless I misunderstood you and you were saying that we freeze a > version, or a set of versions, of customer that is/are known to pass > their own tests, and test the combination of that frozen version of > the customer with our daily development. If that is the case, then > I would agree that we are using their test to test us, not them. > But I somehow didn't get that impression, hence my reaction. The test I'm adding tests the release version of git-annex, so I think in practice we don't have to worry about random changes of theirs producing false positives for us. The utility of this test is that sometime close to release someone (e.g. me) can run it, if it fails let's see if it fails on the last release version of ours, if so it's probably upstream breakage, or like with the 2.13.0 release if it's OK on 2.12.0 it's our bug. It'll never trip some random tester up since you need to explicitly opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these objections. This incurs no burden on either devs, packagers or users, and would have demonstrably detected an issue we'd rather have wanted to know about pre-release than post-release as is the case now.
