Chris, On Fri, Dec 7, 2018 at 1:24 PM Chris Lamb <[email protected]> wrote: > > Could you elaborate more why? You do mention that changing 'Test- > Architectures' (for example) should not trigger the rebuilding of > packages but is that really a big concern? I mean, I throw away > everything before running any test and it's not excessively slow to do > so. >
First of all, it is not urgent. My hope is that the test suite will eventually permutate through several degrees of freedom. The goal is to get as many *test cases* out of the test specifications as possible. That idea was formed in reaction to commit ccb387b4. There, a problem arose when a test specification was filled with 'Package-Architecture: any' while the harness provided 'amd64' (due to a particular implementation of Test-Architectures). The check should have worked either way. Next, please consider that the current defaults for test cases are package format '1.0 (native)' with 'Package-Architecture: all'. That is not representative of the archive at all. As you know from my merge requests, I went on to parametrize many templates. I never submitted branches changing the default package architecture to 'any', because they seemed like half-way solutions. The right way is to permutate for each degree of freedom through all possible values. When one test specification is turned into many test cases, some data changes and some does not. The template fill data and the test options can vary from one test case to another, while the constraints do not. In fact, the constraints are currently stored up one directory. The distinction between fill data and test options, on the other hand, is helpful to avoid confusion. Among other things, it will safely allow the field Package-Architecture to be renamed back to Architecture (hoping to address your request here: https://salsa.debian.org/lintian/lintian/merge_requests/46#note_46510). As for rebuilding, perhaps I am more time sensitive. I run the full suite dozens of times a day and would like to bring it down from six minutes over here to about one (if my experiments hold up). When adding permutations for package architecture and package format alone (and there are additional degrees of freedom), the 670~ test specifications turn into 6000 test cases, or more. Rebuilding is not a problem with a single test specification or with hand edits, but rather when such processes are automated. That being said, I am not set one way or another. Please let's revisit this proposal when my 'variations' branch is ready for your consideration.
