We've had some more discussions regarding this topic within the CI
team. Here's a brief summary of the recommendation we agreed upon:

Tests may be written in different ways, but are enabled in a
standard way directly in the package git repository as defined
by the Standard Test Interface. [1]

Test code can be stored directly in the package git repository
(recommended as default) or fetched from another location hosted
in the Fedora infrastructure such as the Tests Namespace. [2] The
decision where the test code is stored is up to the package maintainer.

[1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface
[2] https://fedoraproject.org/wiki/CI/Share_Test_Code


On 1 March 2018 at 18:06, Honza Horak <hho...@redhat.com> wrote:
> On 03/01/2018 04:00 PM, Petr Šplíchal wrote:
>>>> To sum up what I've heard so far from the developer side:
>>>> * I would like to enable tests for my component (yes, I want)
>>>> * I will take care of them (really, I see the benefit in CI)
>>>> * I want to easily collaborate on tests with qe (direct commits)
>>>> * I don't want to give qe commit access to the rpms dist git
>>> This is quite likely the crux of the problem.
>>> Personally, I'm perfectly happy to give write access to any repo
>>> to people who have shown that they know what they are doing.
>>> We have pull requests in dist-git pagure now, and I think this is
>>> a better approach:
>>> 1. ask QE contributors to submit PRs
>>> 2. if enough cooperation happens and trust is established, give
>>>    privileges to write to the repo directly, possibly with an agreement
>>>    that this specific person should only touch tests, and not the
>>>    packaging.
>>> I think it's perfectly fine to never get to point 2.: many many
>>> upstream projects require a review from a second person, or sometimes
>>> even two reviews before a PR is merged, which means that one _never_
>>> merges their own PR, and another contributor is always involved. We
>>> usually don't do this in dist-git, but I'm quite sure that requiring
>>> reviews for dist-git changes would raise quality and catch many silly
>>> mistakes. Either way, it's nowadays possible to cooperate using a
>>> single repo without fully trusting the other person, frictionlessly.
>> Good point. However, if there is a qe/devel team which prefers to
>> collaborate on tests in a separate repo because this is the most
>> efficient way they found so far, why should we stop them and try
>> to enforce a less efficient workflow? (Which they can workaround
>> by a different git repo.)
>>>> * I want to efficiently maintain tests long-term
>>>> * I want to have just a single place for my tests (no duplication)
>>>> * I don't want to backport new tests to old branches (enable once)
>>>> * I want to easily enable testing for all supported branches
>>> Those four items depend strongly on the package. My thinking is
>>> biased by some specific usecases (mainly systemd), but I'm sure many
>>> other packages are like that: a lot of tests would be for new
>>> functionality, and then the tests _are_ tied to the branch being
>>> tested.
>>> While I agree that keeping tests separate avoids a bit of effort
>>> required to update multiple branches, this effort isn't very big. If
>>> the tests indeed apply cleanly to all branches, then it's just a
>>> matter of doing 'git merge' or 'git cherry-pick' once per branch.
>>>> * I want to keep rpms dist git clean (just metadata & patches)
>>> Meh.
>>>> * I want to run all available relevant tests (not to list them)
>>>> * I want to execute set of tests based on a tag (e.g. Tier1)
>>> Those two sound like stuff that should be solved in the tooling,
>>> whatever is used to run tests.
>>>> * I want an easy way to clone tests (fedpkg clone tests/pkg)
>>> Tests alone make no sense, you need something to test, and
>>> cloning one repo is easier than cloning two repos, so there's no
>>> advantage to the split here.
>> But "fedpkg clone tests" is easier than cloning from a "random"
>> git repository where I was forced to save my tests because I was
>> not allowed to save them in Fedora tests namespace.
>>>> I believe the tests namespace resolves them all.
>>> None of those arguments convince me that separate repos for tests are
>>> a good default. Sure, there are specific packages where this will make
>>> sense, or specific packagers with idiosyncratic workflows, but I'd
>>> rather "start small", with the simple configuration, and only move
>>> specific packages to the more complicated setup once that proves to
>>> be required.
>> Why default? Test namespace should be an option. Not the default.
>> Storing tests directly in dist git is and will be possible.
>> Anybody who finds this as a better way can do so. But why
>> enforcing this approach to all?
> +1 Exactly! I don't really see any reason why to force people to follow one
> way when there are obvious disadvantages. All concerns against tests/
> namespace in this thread are based on expectations that it won't work, but
> how can anybody know at this point? The only proof we have is that the
> tests/ namespace works internally.
> I believe that what we really need here is flexibility -- let people find
> the way that works best for them.
> Honza
> _______________________________________________
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Reply via email to