We've had some more discussions regarding this topic within the CI
team. Here's a brief summary of the recommendation we agreed upon:
Tests may be written in different ways, but are enabled in a
standard way directly in the package git repository as defined
by the Standard Test Interface. 
Test code can be stored directly in the package git repository
(recommended as default) or fetched from another location hosted
in the Fedora infrastructure such as the Tests Namespace.  The
decision where the test code is stored is up to the package maintainer.
On 1 March 2018 at 18:06, Honza Horak <hho...@redhat.com> wrote:
> On 03/01/2018 04:00 PM, Petr Šplíchal wrote:
>>>> To sum up what I've heard so far from the developer side:
>>>> * I would like to enable tests for my component (yes, I want)
>>>> * I will take care of them (really, I see the benefit in CI)
>>>> * I want to easily collaborate on tests with qe (direct commits)
>>>> * I don't want to give qe commit access to the rpms dist git
>>> This is quite likely the crux of the problem.
>>> Personally, I'm perfectly happy to give write access to any repo
>>> to people who have shown that they know what they are doing.
>>> We have pull requests in dist-git pagure now, and I think this is
>>> a better approach:
>>> 1. ask QE contributors to submit PRs
>>> 2. if enough cooperation happens and trust is established, give
>>> privileges to write to the repo directly, possibly with an agreement
>>> that this specific person should only touch tests, and not the
>>> I think it's perfectly fine to never get to point 2.: many many
>>> upstream projects require a review from a second person, or sometimes
>>> even two reviews before a PR is merged, which means that one _never_
>>> merges their own PR, and another contributor is always involved. We
>>> usually don't do this in dist-git, but I'm quite sure that requiring
>>> reviews for dist-git changes would raise quality and catch many silly
>>> mistakes. Either way, it's nowadays possible to cooperate using a
>>> single repo without fully trusting the other person, frictionlessly.
>> Good point. However, if there is a qe/devel team which prefers to
>> collaborate on tests in a separate repo because this is the most
>> efficient way they found so far, why should we stop them and try
>> to enforce a less efficient workflow? (Which they can workaround
>> by a different git repo.)
>>>> * I want to efficiently maintain tests long-term
>>>> * I want to have just a single place for my tests (no duplication)
>>>> * I don't want to backport new tests to old branches (enable once)
>>>> * I want to easily enable testing for all supported branches
>>> Those four items depend strongly on the package. My thinking is
>>> biased by some specific usecases (mainly systemd), but I'm sure many
>>> other packages are like that: a lot of tests would be for new
>>> functionality, and then the tests _are_ tied to the branch being
>>> While I agree that keeping tests separate avoids a bit of effort
>>> required to update multiple branches, this effort isn't very big. If
>>> the tests indeed apply cleanly to all branches, then it's just a
>>> matter of doing 'git merge' or 'git cherry-pick' once per branch.
>>>> * I want to keep rpms dist git clean (just metadata & patches)
>>>> * I want to run all available relevant tests (not to list them)
>>>> * I want to execute set of tests based on a tag (e.g. Tier1)
>>> Those two sound like stuff that should be solved in the tooling,
>>> whatever is used to run tests.
>>>> * I want an easy way to clone tests (fedpkg clone tests/pkg)
>>> Tests alone make no sense, you need something to test, and
>>> cloning one repo is easier than cloning two repos, so there's no
>>> advantage to the split here.
>> But "fedpkg clone tests" is easier than cloning from a "random"
>> git repository where I was forced to save my tests because I was
>> not allowed to save them in Fedora tests namespace.
>>>> I believe the tests namespace resolves them all.
>>> None of those arguments convince me that separate repos for tests are
>>> a good default. Sure, there are specific packages where this will make
>>> sense, or specific packagers with idiosyncratic workflows, but I'd
>>> rather "start small", with the simple configuration, and only move
>>> specific packages to the more complicated setup once that proves to
>>> be required.
>> Why default? Test namespace should be an option. Not the default.
>> Storing tests directly in dist git is and will be possible.
>> Anybody who finds this as a better way can do so. But why
>> enforcing this approach to all?
> +1 Exactly! I don't really see any reason why to force people to follow one
> way when there are obvious disadvantages. All concerns against tests/
> namespace in this thread are based on expectations that it won't work, but
> how can anybody know at this point? The only proof we have is that the
> tests/ namespace works internally.
> I believe that what we really need here is flexibility -- let people find
> the way that works best for them.
> devel mailing list -- firstname.lastname@example.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
devel mailing list -- email@example.com
To unsubscribe send an email to devel-le...@lists.fedoraproject.org