Re: New "tests" namespace to share test code

[Petr Šplíchal]

Hi!

We've had some more discussions regarding this topic within the CI
team. Here's a brief summary of the recommendation we agreed upon:

Tests may be written in different ways, but are enabled in a
standard way directly in the package git repository as defined
by the Standard Test Interface. [1]

Test code can be stored directly in the package git repository
(recommended as default) or fetched from another location hosted
in the Fedora infrastructure such as the Tests Namespace. [2] The
decision where the test code is stored is up to the package maintainer.

[1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface
[2] https://fedoraproject.org/wiki/CI/Share_Test_Code

psss...

On 1 March 2018 at 18:06, Honza Horak  wrote:
> On 03/01/2018 04:00 PM, Petr Šplíchal wrote:

 To sum up what I've heard so far from the developer side:

 * I would like to enable tests for my component (yes, I want)
 * I will take care of them (really, I see the benefit in CI)
 * I want to easily collaborate on tests with qe (direct commits)
 * I don't want to give qe commit access to the rpms dist git
>>>
>>> This is quite likely the crux of the problem.
>>>
>>> Personally, I'm perfectly happy to give write access to any repo
>>> to people who have shown that they know what they are doing.
>>> We have pull requests in dist-git pagure now, and I think this is
>>> a better approach:
>>> 1. ask QE contributors to submit PRs
>>> 2. if enough cooperation happens and trust is established, give
>>>privileges to write to the repo directly, possibly with an agreement
>>>that this specific person should only touch tests, and not the
>>>packaging.
>>>
>>> I think it's perfectly fine to never get to point 2.: many many
>>> upstream projects require a review from a second person, or sometimes
>>> even two reviews before a PR is merged, which means that one _never_
>>> merges their own PR, and another contributor is always involved. We
>>> usually don't do this in dist-git, but I'm quite sure that requiring
>>> reviews for dist-git changes would raise quality and catch many silly
>>> mistakes. Either way, it's nowadays possible to cooperate using a
>>> single repo without fully trusting the other person, frictionlessly.
>>
>>
>> Good point. However, if there is a qe/devel team which prefers to
>> collaborate on tests in a separate repo because this is the most
>> efficient way they found so far, why should we stop them and try
>> to enforce a less efficient workflow? (Which they can workaround
>> by a different git repo.)
>>
 * I want to efficiently maintain tests long-term
 * I want to have just a single place for my tests (no duplication)
 * I don't want to backport new tests to old branches (enable once)
 * I want to easily enable testing for all supported branches
>>>
>>> Those four items depend strongly on the package. My thinking is
>>> biased by some specific usecases (mainly systemd), but I'm sure many
>>> other packages are like that: a lot of tests would be for new
>>> functionality, and then the tests _are_ tied to the branch being
>>> tested.
>>>
>>> While I agree that keeping tests separate avoids a bit of effort
>>> required to update multiple branches, this effort isn't very big. If
>>> the tests indeed apply cleanly to all branches, then it's just a
>>> matter of doing 'git merge' or 'git cherry-pick' once per branch.
>>>
 * I want to keep rpms dist git clean (just metadata & patches)
>>>
>>> Meh.
>>>
 * I want to run all available relevant tests (not to list them)
 * I want to execute set of tests based on a tag (e.g. Tier1)
>>>
>>> Those two sound like stuff that should be solved in the tooling,
>>> whatever is used to run tests.
>>>
 * I want an easy way to clone tests (fedpkg clone tests/pkg)
>>>
>>> Tests alone make no sense, you need something to test, and
>>> cloning one repo is easier than cloning two repos, so there's no
>>> advantage to the split here.
>>
>>
>> But "fedpkg clone tests" is easier than cloning from a "random"
>> git repository where I was forced to save my tests because I was
>> not allowed to save them in Fedora tests namespace.
>>
 I believe the tests namespace resolves them all.
>>>
>>> None of those arguments convince me that separate repos for tests are
>>> a good default. Sure, there are specific packages where this will make
>>> sense, or specific packagers with idiosyncratic workflows, but I'd
>>> rather "start small", with the simple configuration, and only move
>>> specific packages to the more complicated setup once that proves to
>>> be required.
>>
>>
>> Why default? Test namespace should be an option. Not the default.
>> Storing tests directly in dist git is and will be possible.
>> Anybody who finds this as a better way can do so. But why
>> enforcing this approach to all?
>
>
> +1 Exactly! I don't really see any reason why to force people to follow one
> way when there are obvious 

Re: New "tests" namespace to share test code

[Honza Horak]

On 02/25/2018 08:59 PM, Pierre-Yves Chibon wrote:

On Sun, Feb 25, 2018 at 12:12:33PM +, Zbigniew Jędrzejewski-Szmek wrote:

On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:

On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:

Hi!

During the last days there have been concerns raised regarding
what is an appropriate content for the tests namespace. [1] My
original idea was to enable sharing tests even across branches of
the same component, not only for tests to be used by completely
different packages. The initial examples might have been a bit
misleading in this respect. One of the main points still holds:


  * Tests might follow a different branching pattern than the
dist-git repo, leading to code duplication


 From the feedback from developers I feel they always keep on mind
and care a lot about the maintenance costs. So it perfectly makes
sense to me if they want to keep and maintain tests in a separate
repo instead of merging/cherry-picking between dist-git branches.

When, for a particular package, it is the most efficient way to
maintain tests in a separate repo why should we discourage from
this approach? There are packages where it makes more sense to
store test code directly in dist-git. And it is still an option.
But why should we enforce this for all?


My worries are basically that this mechanism is hiding the tests from the
package maintainers. It splits the concerns between people maintaining the
artifacts and people maintaining the tests, which is exactly what the original
plan to bring CI was *not* about.
The idea has always been to bring the tests where the code lives, so that both
can be worked on as one, to make tests be a concern of the package maintainer
very much like updating the artifact (rpms, image..) is, while getting support
from QE for them (the tests).

In addition, this is what I fear most:
The tests for the package are stored elsewhere, we're not sure where (the tests
namespace, a random git repo on the internet...), the packager update package
and the tests fail.
What do you think will happen?
a) the packager will look for $random_place_of_the_internet where tests are and
spend time trying to fix them.
b) the packager will turn off the tests because they get in the way

If the packager wants to turn off the tests, they will have to go through
dist-git to do it, they may very well end up turning the tests off anyway, but
if the tests are right there, they may as well have a quick look at them to see
if they can fix them quickly before deciding.

In addition, if the packager turn the tests off and the people maintaining the
tests do not realize that, they will end up spending time maintaining
$somewhere_else tests that aren't being used.
Having them interact directly with the dist-git repo will increase the chances
that they see/realize when something is turned off.

If that means we have less tests in dist-git because the maintainers do not want
them, I'd say so be it.
In the long term this is their loss, they are the ones who will get the bug
reports and will have to deal with them, they are the ones who will have to dive
into old code rather than going back into something that is still fresh in their
mind.
As long as, it is clear that they have been approached and that it is their
choice to not merge pull-requests adding tests, I think the people offering to
help should not be the ones blamed.


pingou, I share your thinking in general, but I think your concern is
overstated. Petr's original e-mail suggests that the new separate
namespace should be used *in* *preference* to the in-dist-git, and your
reply concentrates on that. But I think that we want to have both
possibilities, because both will work best for different cases, and it's
just the emphasis that needs to change.

I agree that having tests in a separate namespace is not any simpler
or more manageable for most packages. Thus, I think that *by* *default*
we should put tests in dist-git. IMHO this will apply to 90% of all
tests and packages.

OTOH, there will be cases where it'll be useful to split the tests out
into a separate area. The documentation for tests/ has the example of
shared tests for bash/zhs/dash. I'm sure that there will be other
cases, like testing of stacks consisting of multiple packages
apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial
to keep the test separate. The management of permissions in this case
will be _much_ more complicated, because to meaningfully update the
tests sometimes it'll be necessary to change the test repo and _each_
of the packages. But having them separate is better than randomly
sticking the shared tests in dist-git for one of the involved packages.
Nevertheless, because of the overhead of more repos and additional
permissions, IMHO this should never be advertised as the default
approach, but as an "advanced" possibility.


I violently agree with this :) I am very much advocating for using the test

Re: New "tests" namespace to share test code

[Honza Horak]

On 03/01/2018 04:00 PM, Petr Šplíchal wrote:

To sum up what I've heard so far from the developer side:

* I would like to enable tests for my component (yes, I want)
* I will take care of them (really, I see the benefit in CI)
* I want to easily collaborate on tests with qe (direct commits)
* I don't want to give qe commit access to the rpms dist git

This is quite likely the crux of the problem.

Personally, I'm perfectly happy to give write access to any repo
to people who have shown that they know what they are doing.
We have pull requests in dist-git pagure now, and I think this is
a better approach:
1. ask QE contributors to submit PRs
2. if enough cooperation happens and trust is established, give
   privileges to write to the repo directly, possibly with an agreement
   that this specific person should only touch tests, and not the
   packaging.

I think it's perfectly fine to never get to point 2.: many many
upstream projects require a review from a second person, or sometimes
even two reviews before a PR is merged, which means that one _never_
merges their own PR, and another contributor is always involved. We
usually don't do this in dist-git, but I'm quite sure that requiring
reviews for dist-git changes would raise quality and catch many silly
mistakes. Either way, it's nowadays possible to cooperate using a
single repo without fully trusting the other person, frictionlessly.


Good point. However, if there is a qe/devel team which prefers to
collaborate on tests in a separate repo because this is the most
efficient way they found so far, why should we stop them and try
to enforce a less efficient workflow? (Which they can workaround
by a different git repo.)


* I want to efficiently maintain tests long-term
* I want to have just a single place for my tests (no duplication)
* I don't want to backport new tests to old branches (enable once)
* I want to easily enable testing for all supported branches

Those four items depend strongly on the package. My thinking is
biased by some specific usecases (mainly systemd), but I'm sure many
other packages are like that: a lot of tests would be for new
functionality, and then the tests _are_ tied to the branch being
tested.

While I agree that keeping tests separate avoids a bit of effort
required to update multiple branches, this effort isn't very big. If
the tests indeed apply cleanly to all branches, then it's just a
matter of doing 'git merge' or 'git cherry-pick' once per branch.


* I want to keep rpms dist git clean (just metadata & patches)

Meh.


* I want to run all available relevant tests (not to list them)
* I want to execute set of tests based on a tag (e.g. Tier1)

Those two sound like stuff that should be solved in the tooling,
whatever is used to run tests.


* I want an easy way to clone tests (fedpkg clone tests/pkg)

Tests alone make no sense, you need something to test, and
cloning one repo is easier than cloning two repos, so there's no
advantage to the split here.


But "fedpkg clone tests" is easier than cloning from a "random"
git repository where I was forced to save my tests because I was
not allowed to save them in Fedora tests namespace.


I believe the tests namespace resolves them all.

None of those arguments convince me that separate repos for tests are
a good default. Sure, there are specific packages where this will make
sense, or specific packagers with idiosyncratic workflows, but I'd
rather "start small", with the simple configuration, and only move
specific packages to the more complicated setup once that proves to
be required.


Why default? Test namespace should be an option. Not the default.
Storing tests directly in dist git is and will be possible.
Anybody who finds this as a better way can do so. But why
enforcing this approach to all?


+1 Exactly! I don't really see any reason why to force people to follow 
one way when there are obvious disadvantages. All concerns against 
tests/ namespace in this thread are based on expectations that it won't 
work, but how can anybody know at this point? The only proof we have is 
that the tests/ namespace works internally.


I believe that what we really need here is flexibility -- let people 
find the way that works best for them.


Honza
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

>> To sum up what I've heard so far from the developer side:
>>
>> * I would like to enable tests for my component (yes, I want)
>> * I will take care of them (really, I see the benefit in CI)
>> * I want to easily collaborate on tests with qe (direct commits)
>> * I don't want to give qe commit access to the rpms dist git
> This is quite likely the crux of the problem.
>
> Personally, I'm perfectly happy to give write access to any repo
> to people who have shown that they know what they are doing.
> We have pull requests in dist-git pagure now, and I think this is
> a better approach:
> 1. ask QE contributors to submit PRs
> 2. if enough cooperation happens and trust is established, give
>   privileges to write to the repo directly, possibly with an agreement
>   that this specific person should only touch tests, and not the
>   packaging.
>
> I think it's perfectly fine to never get to point 2.: many many
> upstream projects require a review from a second person, or sometimes
> even two reviews before a PR is merged, which means that one _never_
> merges their own PR, and another contributor is always involved. We
> usually don't do this in dist-git, but I'm quite sure that requiring
> reviews for dist-git changes would raise quality and catch many silly
> mistakes. Either way, it's nowadays possible to cooperate using a
> single repo without fully trusting the other person, frictionlessly.

Good point. However, if there is a qe/devel team which prefers to
collaborate on tests in a separate repo because this is the most
efficient way they found so far, why should we stop them and try
to enforce a less efficient workflow? (Which they can workaround
by a different git repo.)

>> * I want to efficiently maintain tests long-term
>> * I want to have just a single place for my tests (no duplication)
>> * I don't want to backport new tests to old branches (enable once)
>> * I want to easily enable testing for all supported branches
> Those four items depend strongly on the package. My thinking is
> biased by some specific usecases (mainly systemd), but I'm sure many
> other packages are like that: a lot of tests would be for new
> functionality, and then the tests _are_ tied to the branch being
> tested.
>
> While I agree that keeping tests separate avoids a bit of effort
> required to update multiple branches, this effort isn't very big. If
> the tests indeed apply cleanly to all branches, then it's just a
> matter of doing 'git merge' or 'git cherry-pick' once per branch.
>
>> * I want to keep rpms dist git clean (just metadata & patches)
> Meh.
>
>> * I want to run all available relevant tests (not to list them)
>> * I want to execute set of tests based on a tag (e.g. Tier1)
> Those two sound like stuff that should be solved in the tooling,
> whatever is used to run tests.
>
>> * I want an easy way to clone tests (fedpkg clone tests/pkg)
> Tests alone make no sense, you need something to test, and
> cloning one repo is easier than cloning two repos, so there's no
> advantage to the split here.

But "fedpkg clone tests" is easier than cloning from a "random"
git repository where I was forced to save my tests because I was
not allowed to save them in Fedora tests namespace.

>> I believe the tests namespace resolves them all.
> None of those arguments convince me that separate repos for tests are
> a good default. Sure, there are specific packages where this will make
> sense, or specific packagers with idiosyncratic workflows, but I'd
> rather "start small", with the simple configuration, and only move
> specific packages to the more complicated setup once that proves to
> be required.

Why default? Test namespace should be an option. Not the default.
Storing tests directly in dist git is and will be possible.
Anybody who finds this as a better way can do so. But why
enforcing this approach to all?

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Zbigniew Jędrzejewski-Szmek]

On Thu, Mar 01, 2018 at 02:44:37PM +0100, Petr Šplíchal wrote:
> > > > My worries are basically that this mechanism is hiding the
> > > > tests from the package maintainers. It splits the concerns
> > > > between people maintaining the artifacts and people
> > > > maintaining the tests, which is exactly what the original
> > > > plan to bring CI was *not* about.
> > >
> > > I do not see a split here but rather potential for enhanced or
> > > extended collaboration. Repositories in the tests namespace
> > > can be set up with more open access than rpms git. In this way
> > > devel can invite qe for direct collaboration on tests.
> >
> > I don't see how that's not possible with tests being in
> > dist-git.
> 
> What I've heard from devels is that they often don't feel like
> giving direct commit access to the rpms dist git to QE as spec
> files and packaging in general is a big responsiblity. But they
> are quite OK with directly collaborating with QE on test coverage.
> 
> > If the tests are defined in dist-git and adding new tests means
> > committing these changes to dist-git's yaml file, how does that
> > solve the concern about "noise" in the git commits?
> 
> Not necessarily. You could use advanced metadata to select desired
> tests only once. The very first proof of concept of using Flexible
> Metadata Format for testing selinux quite nicely illustrates this:
> 
> https://src.fedoraproject.org/tests/selinux/pull-request/1
> https://github.com/psss/fmf
> 
> Instead of listing individual tests you could say: Run all tests
> for this selinux component + all Tier1 selinux integration tests:
> 
> fmf --key test --filter 'component:libselinux | tags:Tier1'
> 
> For tests covering multiple components the benefit of the tests
> namespace approach is unquestionable. However, I see benefit of
> the approach for single components as well (and developers seem
> to often feel the same): That is not having to cherry-pick/merge
> test coverage between supported branches. So even for these cases
> there is the benefit of sharing the test code.
> 
> I believe that extending the integration testing coverage is one
> the most essential goals of Fedora CI. This is the crucial piece
> of the puzzle how to make an Always Ready Operating System: It is
> to ensure that packages play nicely together. So I guess there
> will be quite many tests which will benefit from this approach.
> 
> Do we really want/need to assess every new request for a tests
> repo if it is good-enough for the test namespace? Will there be
> a committee to decide whether a particular set of tests has a
> potential to be used by other components in the future or not? :)
> 
> To sum up what I've heard so far from the developer side:
> 
> * I would like to enable tests for my component (yes, I want)
> * I will take care of them (really, I see the benefit in CI)
> * I want to easily collaborate on tests with qe (direct commits)
> * I don't want to give qe commit access to the rpms dist git
This is quite likely the crux of the problem.

Personally, I'm perfectly happy to give write access to any repo
to people who have shown that they know what they are doing.
We have pull requests in dist-git pagure now, and I think this is
a better approach:
1. ask QE contributors to submit PRs
2. if enough cooperation happens and trust is established, give
  privileges to write to the repo directly, possibly with an agreement
  that this specific person should only touch tests, and not the
  packaging.

I think it's perfectly fine to never get to point 2.: many many
upstream projects require a review from a second person, or sometimes
even two reviews before a PR is merged, which means that one _never_
merges their own PR, and another contributor is always involved. We
usually don't do this in dist-git, but I'm quite sure that requiring
reviews for dist-git changes would raise quality and catch many silly
mistakes. Either way, it's nowadays possible to cooperate using a
single repo without fully trusting the other person, frictionlessly.

> * I want to efficiently maintain tests long-term
> * I want to have just a single place for my tests (no duplication)
> * I don't want to backport new tests to old branches (enable once)
> * I want to easily enable testing for all supported branches
Those four items depend strongly on the package. My thinking is
biased by some specific usecases (mainly systemd), but I'm sure many
other packages are like that: a lot of tests would be for new
functionality, and then the tests _are_ tied to the branch being
tested.

While I agree that keeping tests separate avoids a bit of effort
required to update multiple branches, this effort isn't very big. If
the tests indeed apply cleanly to all branches, then it's just a
matter of doing 'git merge' or 'git cherry-pick' once per branch.

> * I want to keep rpms dist git clean (just metadata & patches)
Meh.

> * I want to run all available relevant tests (not to list them)
> * I want to 

Re: New "tests" namespace to share test code

[Petr Šplíchal]

> > > My worries are basically that this mechanism is hiding the
> > > tests from the package maintainers. It splits the concerns
> > > between people maintaining the artifacts and people
> > > maintaining the tests, which is exactly what the original
> > > plan to bring CI was *not* about.
> >
> > I do not see a split here but rather potential for enhanced or
> > extended collaboration. Repositories in the tests namespace
> > can be set up with more open access than rpms git. In this way
> > devel can invite qe for direct collaboration on tests.
>
> I don't see how that's not possible with tests being in
> dist-git.

What I've heard from devels is that they often don't feel like
giving direct commit access to the rpms dist git to QE as spec
files and packaging in general is a big responsiblity. But they
are quite OK with directly collaborating with QE on test coverage.

> If the tests are defined in dist-git and adding new tests means
> committing these changes to dist-git's yaml file, how does that
> solve the concern about "noise" in the git commits?

Not necessarily. You could use advanced metadata to select desired
tests only once. The very first proof of concept of using Flexible
Metadata Format for testing selinux quite nicely illustrates this:

https://src.fedoraproject.org/tests/selinux/pull-request/1
https://github.com/psss/fmf

Instead of listing individual tests you could say: Run all tests
for this selinux component + all Tier1 selinux integration tests:

fmf --key test --filter 'component:libselinux | tags:Tier1'

For tests covering multiple components the benefit of the tests
namespace approach is unquestionable. However, I see benefit of
the approach for single components as well (and developers seem
to often feel the same): That is not having to cherry-pick/merge
test coverage between supported branches. So even for these cases
there is the benefit of sharing the test code.

I believe that extending the integration testing coverage is one
the most essential goals of Fedora CI. This is the crucial piece
of the puzzle how to make an Always Ready Operating System: It is
to ensure that packages play nicely together. So I guess there
will be quite many tests which will benefit from this approach.

Do we really want/need to assess every new request for a tests
repo if it is good-enough for the test namespace? Will there be
a committee to decide whether a particular set of tests has a
potential to be used by other components in the future or not? :)

To sum up what I've heard so far from the developer side:

* I would like to enable tests for my component (yes, I want)
* I will take care of them (really, I see the benefit in CI)
* I want to easily collaborate on tests with qe (direct commits)
* I don't want to give qe commit access to the rpms dist git
* I want to efficiently maintain tests long-term
* I want to have just a single place for my tests (no duplication)
* I want to keep rpms dist git clean (just metadata & patches)
* I want to easily enable testing for all supported branches
* I don't want to backport new tests to old branches (enable once)
* I want to run all available relevant tests (not to list them)
* I want to execute set of tests based on a tag (e.g. Tier1)
* I want an easy way to clone tests (fedpkg clone tests/pkg)

I believe the tests namespace resolves them all.

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Pierre-Yves Chibon]

On Sun, Feb 25, 2018 at 12:12:33PM +, Zbigniew Jędrzejewski-Szmek wrote:
> On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:
> > On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
> > > Hi!
> > > 
> > > During the last days there have been concerns raised regarding
> > > what is an appropriate content for the tests namespace. [1] My
> > > original idea was to enable sharing tests even across branches of
> > > the same component, not only for tests to be used by completely
> > > different packages. The initial examples might have been a bit
> > > misleading in this respect. One of the main points still holds:
> > > 
> > > >  * Tests might follow a different branching pattern than the
> > > >dist-git repo, leading to code duplication
> > > 
> > > From the feedback from developers I feel they always keep on mind
> > > and care a lot about the maintenance costs. So it perfectly makes
> > > sense to me if they want to keep and maintain tests in a separate
> > > repo instead of merging/cherry-picking between dist-git branches.
> > > 
> > > When, for a particular package, it is the most efficient way to
> > > maintain tests in a separate repo why should we discourage from
> > > this approach? There are packages where it makes more sense to
> > > store test code directly in dist-git. And it is still an option.
> > > But why should we enforce this for all?
> > 
> > My worries are basically that this mechanism is hiding the tests from the
> > package maintainers. It splits the concerns between people maintaining the
> > artifacts and people maintaining the tests, which is exactly what the 
> > original
> > plan to bring CI was *not* about.
> > The idea has always been to bring the tests where the code lives, so that 
> > both
> > can be worked on as one, to make tests be a concern of the package 
> > maintainer
> > very much like updating the artifact (rpms, image..) is, while getting 
> > support
> > from QE for them (the tests).
> > 
> > In addition, this is what I fear most:
> > The tests for the package are stored elsewhere, we're not sure where (the 
> > tests
> > namespace, a random git repo on the internet...), the packager update 
> > package
> > and the tests fail.
> > What do you think will happen?
> > a) the packager will look for $random_place_of_the_internet where tests are 
> > and
> >spend time trying to fix them.
> > b) the packager will turn off the tests because they get in the way
> > 
> > If the packager wants to turn off the tests, they will have to go through
> > dist-git to do it, they may very well end up turning the tests off anyway, 
> > but
> > if the tests are right there, they may as well have a quick look at them to 
> > see
> > if they can fix them quickly before deciding.
> > 
> > In addition, if the packager turn the tests off and the people maintaining 
> > the
> > tests do not realize that, they will end up spending time maintaining
> > $somewhere_else tests that aren't being used.
> > Having them interact directly with the dist-git repo will increase the 
> > chances
> > that they see/realize when something is turned off.
> > 
> > If that means we have less tests in dist-git because the maintainers do not 
> > want
> > them, I'd say so be it.
> > In the long term this is their loss, they are the ones who will get the bug
> > reports and will have to deal with them, they are the ones who will have to 
> > dive
> > into old code rather than going back into something that is still fresh in 
> > their
> > mind.
> > As long as, it is clear that they have been approached and that it is their
> > choice to not merge pull-requests adding tests, I think the people offering 
> > to
> > help should not be the ones blamed.
> 
> pingou, I share your thinking in general, but I think your concern is
> overstated. Petr's original e-mail suggests that the new separate
> namespace should be used *in* *preference* to the in-dist-git, and your
> reply concentrates on that. But I think that we want to have both
> possibilities, because both will work best for different cases, and it's
> just the emphasis that needs to change.
> 
> I agree that having tests in a separate namespace is not any simpler
> or more manageable for most packages. Thus, I think that *by* *default*
> we should put tests in dist-git. IMHO this will apply to 90% of all
> tests and packages.
> 
> OTOH, there will be cases where it'll be useful to split the tests out
> into a separate area. The documentation for tests/ has the example of
> shared tests for bash/zhs/dash. I'm sure that there will be other
> cases, like testing of stacks consisting of multiple packages
> apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial
> to keep the test separate. The management of permissions in this case
> will be _much_ more complicated, because to meaningfully update the
> tests sometimes it'll be necessary to change the test repo and _each_
> of the packages. But having them 

Re: New "tests" namespace to share test code

[Zbigniew Jędrzejewski-Szmek]

On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:
> On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
> > Hi!
> > 
> > During the last days there have been concerns raised regarding
> > what is an appropriate content for the tests namespace. [1] My
> > original idea was to enable sharing tests even across branches of
> > the same component, not only for tests to be used by completely
> > different packages. The initial examples might have been a bit
> > misleading in this respect. One of the main points still holds:
> > 
> > >  * Tests might follow a different branching pattern than the
> > >dist-git repo, leading to code duplication
> > 
> > From the feedback from developers I feel they always keep on mind
> > and care a lot about the maintenance costs. So it perfectly makes
> > sense to me if they want to keep and maintain tests in a separate
> > repo instead of merging/cherry-picking between dist-git branches.
> > 
> > When, for a particular package, it is the most efficient way to
> > maintain tests in a separate repo why should we discourage from
> > this approach? There are packages where it makes more sense to
> > store test code directly in dist-git. And it is still an option.
> > But why should we enforce this for all?
> 
> My worries are basically that this mechanism is hiding the tests from the
> package maintainers. It splits the concerns between people maintaining the
> artifacts and people maintaining the tests, which is exactly what the original
> plan to bring CI was *not* about.
> The idea has always been to bring the tests where the code lives, so that both
> can be worked on as one, to make tests be a concern of the package maintainer
> very much like updating the artifact (rpms, image..) is, while getting support
> from QE for them (the tests).
> 
> In addition, this is what I fear most:
> The tests for the package are stored elsewhere, we're not sure where (the 
> tests
> namespace, a random git repo on the internet...), the packager update package
> and the tests fail.
> What do you think will happen?
> a) the packager will look for $random_place_of_the_internet where tests are 
> and
>spend time trying to fix them.
> b) the packager will turn off the tests because they get in the way
> 
> If the packager wants to turn off the tests, they will have to go through
> dist-git to do it, they may very well end up turning the tests off anyway, but
> if the tests are right there, they may as well have a quick look at them to 
> see
> if they can fix them quickly before deciding.
> 
> In addition, if the packager turn the tests off and the people maintaining the
> tests do not realize that, they will end up spending time maintaining
> $somewhere_else tests that aren't being used.
> Having them interact directly with the dist-git repo will increase the chances
> that they see/realize when something is turned off.
> 
> If that means we have less tests in dist-git because the maintainers do not 
> want
> them, I'd say so be it.
> In the long term this is their loss, they are the ones who will get the bug
> reports and will have to deal with them, they are the ones who will have to 
> dive
> into old code rather than going back into something that is still fresh in 
> their
> mind.
> As long as, it is clear that they have been approached and that it is their
> choice to not merge pull-requests adding tests, I think the people offering to
> help should not be the ones blamed.

Hi,

pingou, I share your thinking in general, but I think your concern is
overstated. Petr's original e-mail suggests that the new separate
namespace should be used *in* *preference* to the in-dist-git, and your
reply concentrates on that. But I think that we want to have both
possibilities, because both will work best for different cases, and it's
just the emphasis that needs to change.

I agree that having tests in a separate namespace is not any simpler
or more manageable for most packages. Thus, I think that *by* *default*
we should put tests in dist-git. IMHO this will apply to 90% of all
tests and packages.

OTOH, there will be cases where it'll be useful to split the tests out
into a separate area. The documentation for tests/ has the example of
shared tests for bash/zhs/dash. I'm sure that there will be other
cases, like testing of stacks consisting of multiple packages
apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial
to keep the test separate. The management of permissions in this case
will be _much_ more complicated, because to meaningfully update the
tests sometimes it'll be necessary to change the test repo and _each_
of the packages. But having them separate is better than randomly
sticking the shared tests in dist-git for one of the involved packages.
Nevertheless, because of the overhead of more repos and additional
permissions, IMHO this should never be advertised as the default
approach, but as an "advanced" possibility.

Zbyszek

Re: New "tests" namespace to share test code

[Pierre-Yves Chibon]

On Fri, Feb 23, 2018 at 10:33:57AM +0100, Pavel Raiskup wrote:
> On Thursday, February 22, 2018 3:50:38 PM CET Pierre-Yves Chibon wrote:
> > On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
> > > When, for a particular package, it is the most efficient way to
> > > maintain tests in a separate repo why should we discourage from
> > > this approach? There are packages where it makes more sense to
> > > store test code directly in dist-git. And it is still an option.
> > > But why should we enforce this for all?
> > 
> > My worries are basically that this mechanism is hiding the tests from
> > the package maintainers. It splits the concerns between people
> > maintaining the artifacts and people maintaining the tests, which is
> > exactly what the original plan to bring CI was *not* about.
> 
> Since we'll have to "hook" the remote repo into dist-git code, I must be
> missing something.  Where do you see the hidden magic?
> 
> > The idea has always been to bring the tests where the code lives, so
> > that both can be worked on as one, to make tests be a concern of the
> > package maintainer very much like updating the artifact (rpms, image..)
> > is, while getting support from QE for them (the tests).
> 
> If the tests live in separate repo, they still can "work as one".

How do they work as one if they are both working in different places?

> > In addition, this is what I fear most:
> > The tests for the package are stored elsewhere, we're not sure where (the 
> > tests
> > namespace, a random git repo on the internet...), the packager update 
> > package
> > and the tests fail.
> > What do you think will happen?
> > a) the packager will look for $random_place_of_the_internet where tests are 
> > and
> >spend time trying to fix them.
> 
> Yes, please don't force us to use random_place_of_the_internet, but rather
> allow maintainers to use standardized place.
> 
> Then e.g. 'rpmtests/' repo can automatically grant commit access
> to package maintainers of 'rpms/' to not overcomplicate things,
> but still, we can grant non-packagers access to 'rpmtests/'.

We're working on figuring a way to give non-packagers commit to dist-git, so
giving more access would work on the main repo as well.

> What about proventesters, could we grant them access to all tests, or could we
> create new role?
> 
> > b) the packager will turn off the tests because they get in the way
> 
> Polemics (even below) about turning-off the tests is FUD.  Since tests are
> opt-in feature, why would anyone thought this way?

I may very well be miss-seeing this, but what I see is people refusing to add
tests to their repository because it doesn't suit their workflow.
So I am thinking that if they finally think "whatever let's just merge" because
they don't want to cause troubles, whenever the tests are going to impact that
workflow even more because they fail, chances are high that said person will
just turn them off.
I'd be happy to be wrong about that though.

> > If the packager wants to turn off the tests, they will have to go through
> > dist-git to do it, they may very well end up turning the tests off anyway, 
> > but
> > if the tests are right there, they may as well have a quick look at them to 
> > see
> > if they can fix them quickly before deciding.
> > 
> > In addition, if the packager turn the tests off and the people maintaining 
> > the
> > tests do not realize that, they will end up spending time maintaining
> > $somewhere_else tests that aren't being used.
> > Having them interact directly with the dist-git repo will increase the
> > chances
> > that they see/realize when something is turned off.
> >
> > If that means we have less tests in dist-git because the maintainers do
> > not want them, I'd say so be it.  In the long term this is their loss,
> > they are the ones who will get the bug reports and will have to deal
> > with them, they are the ones who will have to dive into old code rather
> > than going back into something that is still fresh in their mind.
> 
> There's no loss, no.  Everything is as transparent as with in-dist-git
> testing, except it gives us more flexibility
> - to allow more people to contribute.

Pull-requests and improvement to our dist-git set up seems to me better ways to
achieve this.

> - keep the package's git smaller and concentrated on packaging

That one is fair.

> > As long as, it is clear that they have been approached and that it is their
> > choice to not merge pull-requests adding tests, I think the people offering 
> > to
> > help should not be the ones blamed.
> 
> The help here is very much welcome, I wouldn't blame anyone though (I for
> one only wait for a bit more flexible form of CI).
> 
> Btw., the far more important problem is that we can not setup CI for
> non-atomic host packages, and that we can not use the CI for Rawhide

Definitely agreed on this one, I've been pushing for it for the better part of a
year, I'd be happy to see more people pushing for 

Re: New "tests" namespace to share test code

[Pierre-Yves Chibon]

On Fri, Feb 23, 2018 at 10:19:35AM +0100, Petr Šplíchal wrote:
> On 22 February 2018 at 15:50, Pierre-Yves Chibon  wrote:
> > On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
> >> Hi!
> >>
> >> During the last days there have been concerns raised regarding
> >> what is an appropriate content for the tests namespace. [1] My
> >> original idea was to enable sharing tests even across branches of
> >> the same component, not only for tests to be used by completely
> >> different packages. The initial examples might have been a bit
> >> misleading in this respect. One of the main points still holds:
> >>
> >> >  * Tests might follow a different branching pattern than the
> >> >dist-git repo, leading to code duplication
> >>
> >> From the feedback from developers I feel they always keep on mind
> >> and care a lot about the maintenance costs. So it perfectly makes
> >> sense to me if they want to keep and maintain tests in a separate
> >> repo instead of merging/cherry-picking between dist-git branches.
> >>
> >> When, for a particular package, it is the most efficient way to
> >> maintain tests in a separate repo why should we discourage from
> >> this approach? There are packages where it makes more sense to
> >> store test code directly in dist-git. And it is still an option.
> >> But why should we enforce this for all?
> 
> Thanks for sharing your thoughts, Pierre.
> 
> > My worries are basically that this mechanism is hiding the tests
> > from the package maintainers. It splits the concerns between
> > people maintaining the artifacts and people maintaining the
> > tests, which is exactly what the original plan to bring CI was
> > *not* about.
> 
> I do not see a split here but rather potential for enhanced or
> extended collaboration. Repositories in the tests namespace can be
> set up with more open access than rpms git. In this way devel can
> invite qe for direct collaboration on tests. 

I don't see how that's not possible with tests being in dist-git.

> That's one of the
> reasons why we should try to fix the non-packager access as soon
> as possible.
> 
> https://pagure.io/fedora-infrastructure/issue/6361

Agreed and being worked on.
 
> > The idea has always been to bring the tests where the code
> > lives, so that both can be worked on as one, to make tests be a
> > concern of the package maintainer very much like updating the
> > artifact (rpms, image..) is, while getting support from QE for
> > them (the tests).
> 
> I see the benefits of having code with tests at the very same
> place, I find this very useful especially for unit tests in the
> project upstream. However, the situation here is a bit different
> in two respects:
> 
> 1) The rpms dist git does not include package code itself but
> rather build metadata only (mainly spec files describing how to
> build the package) so it makes very good sense to host at this
> place testing metadata only (how to test the package). This is
> quite nicely consistent approach.

I still believe it creates a separation of concerns that is in opposition with
the original idea.

> 2) As far as I can see, within the Always Ready Operating System
> effort we are not looking to cover every and each feature in all
> packages but rather ensuring the OS works as a whole. Thus basic
> functionality and integration tests which guard the stability of
> the packages are more appropriate for testing. These do not need
> to be kept so close to the package code.
> 
> I believe we don't want to duplicate upstream unit testing here.
> We should probably clarify more which test types are appropriate
> or expected in the Fedora CI. As part of the docs consolidation I
> added a brief section about test types to the main CI portal:
> 
> https://fedoraproject.org/wiki/CI#Test_Types
> 
> Does it decribe well what we are looking for? Shall we extend it?
> 
> > In addition, this is what I fear most:
> > The tests for the package are stored elsewhere, we're not sure
> > where (the tests namespace, a random git repo on the
> > internet...), the packager update package and the tests fail.
> 
> I see no hiding of tests from maintainers: Tests are clearly
> referenced and discoverable 

Referenced maybe, discoverable, much less than if they are right there.

> from the rpms dist git as defined by
> the Standard Test Interface and devels have access, see:
> 
> https://fedoraproject.org/wiki/CI/Share_Test_Code#Examples

If the tests are defined in dist-git and adding new tests means committing these
changes to dist-git's yaml file, how does that solve the concern about "noise"
in the git commits?

It seems to me that whether the commit touches just the yaml file or the yaml
and tweak the tests themselves, the outcome is the same.

> Devel decides which tests are good for testing, it does not matter
> where the test code lives. However there needs to be trust to the
> referenced repo. But this is the same as with any other open
> source 

Re: New "tests" namespace to share test code

[Pavel Raiskup]

On Thursday, February 22, 2018 3:50:38 PM CET Pierre-Yves Chibon wrote:
> On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
> > When, for a particular package, it is the most efficient way to
> > maintain tests in a separate repo why should we discourage from
> > this approach? There are packages where it makes more sense to
> > store test code directly in dist-git. And it is still an option.
> > But why should we enforce this for all?
> 
> My worries are basically that this mechanism is hiding the tests from
> the package maintainers. It splits the concerns between people
> maintaining the artifacts and people maintaining the tests, which is
> exactly what the original plan to bring CI was *not* about.

Since we'll have to "hook" the remote repo into dist-git code, I must be
missing something.  Where do you see the hidden magic?

> The idea has always been to bring the tests where the code lives, so
> that both can be worked on as one, to make tests be a concern of the
> package maintainer very much like updating the artifact (rpms, image..)
> is, while getting support from QE for them (the tests).

If the tests live in separate repo, they still can "work as one".

> In addition, this is what I fear most:
> The tests for the package are stored elsewhere, we're not sure where (the 
> tests
> namespace, a random git repo on the internet...), the packager update package
> and the tests fail.
> What do you think will happen?
> a) the packager will look for $random_place_of_the_internet where tests are 
> and
>spend time trying to fix them.

Yes, please don't force us to use random_place_of_the_internet, but rather
allow maintainers to use standardized place.

Then e.g. 'rpmtests/' repo can automatically grant commit access
to package maintainers of 'rpms/' to not overcomplicate things,
but still, we can grant non-packagers access to 'rpmtests/'.

What about proventesters, could we grant them access to all tests, or could we
create new role?

> b) the packager will turn off the tests because they get in the way

Polemics (even below) about turning-off the tests is FUD.  Since tests are
opt-in feature, why would anyone thought this way?

> If the packager wants to turn off the tests, they will have to go through
> dist-git to do it, they may very well end up turning the tests off anyway, but
> if the tests are right there, they may as well have a quick look at them to 
> see
> if they can fix them quickly before deciding.
> 
> In addition, if the packager turn the tests off and the people maintaining the
> tests do not realize that, they will end up spending time maintaining
> $somewhere_else tests that aren't being used.
> Having them interact directly with the dist-git repo will increase the
> chances
> that they see/realize when something is turned off.
>
> If that means we have less tests in dist-git because the maintainers do
> not want them, I'd say so be it.  In the long term this is their loss,
> they are the ones who will get the bug reports and will have to deal
> with them, they are the ones who will have to dive into old code rather
> than going back into something that is still fresh in their mind.

There's no loss, no.  Everything is as transparent as with in-dist-git
testing, except it gives us more flexibility
- to allow more people to contribute.
- keep the package's git smaller and concentrated on packaging

> As long as, it is clear that they have been approached and that it is their
> choice to not merge pull-requests adding tests, I think the people offering to
> help should not be the ones blamed.

The help here is very much welcome, I wouldn't blame anyone though (I for
one only wait for a bit more flexible form of CI).

Btw., the far more important problem is that we can not setup CI for
non-atomic host packages, and that we can not use the CI for Rawhide
gating.

Pavel


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

On 22 February 2018 at 15:50, Pierre-Yves Chibon  wrote:
> On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
>> Hi!
>>
>> During the last days there have been concerns raised regarding
>> what is an appropriate content for the tests namespace. [1] My
>> original idea was to enable sharing tests even across branches of
>> the same component, not only for tests to be used by completely
>> different packages. The initial examples might have been a bit
>> misleading in this respect. One of the main points still holds:
>>
>> >  * Tests might follow a different branching pattern than the
>> >dist-git repo, leading to code duplication
>>
>> From the feedback from developers I feel they always keep on mind
>> and care a lot about the maintenance costs. So it perfectly makes
>> sense to me if they want to keep and maintain tests in a separate
>> repo instead of merging/cherry-picking between dist-git branches.
>>
>> When, for a particular package, it is the most efficient way to
>> maintain tests in a separate repo why should we discourage from
>> this approach? There are packages where it makes more sense to
>> store test code directly in dist-git. And it is still an option.
>> But why should we enforce this for all?

Thanks for sharing your thoughts, Pierre.

> My worries are basically that this mechanism is hiding the tests
> from the package maintainers. It splits the concerns between
> people maintaining the artifacts and people maintaining the
> tests, which is exactly what the original plan to bring CI was
> *not* about.

I do not see a split here but rather potential for enhanced or
extended collaboration. Repositories in the tests namespace can be
set up with more open access than rpms git. In this way devel can
invite qe for direct collaboration on tests. That's one of the
reasons why we should try to fix the non-packager access as soon
as possible.

https://pagure.io/fedora-infrastructure/issue/6361

> The idea has always been to bring the tests where the code
> lives, so that both can be worked on as one, to make tests be a
> concern of the package maintainer very much like updating the
> artifact (rpms, image..) is, while getting support from QE for
> them (the tests).

I see the benefits of having code with tests at the very same
place, I find this very useful especially for unit tests in the
project upstream. However, the situation here is a bit different
in two respects:

1) The rpms dist git does not include package code itself but
rather build metadata only (mainly spec files describing how to
build the package) so it makes very good sense to host at this
place testing metadata only (how to test the package). This is
quite nicely consistent approach.

2) As far as I can see, within the Always Ready Operating System
effort we are not looking to cover every and each feature in all
packages but rather ensuring the OS works as a whole. Thus basic
functionality and integration tests which guard the stability of
the packages are more appropriate for testing. These do not need
to be kept so close to the package code.

I believe we don't want to duplicate upstream unit testing here.
We should probably clarify more which test types are appropriate
or expected in the Fedora CI. As part of the docs consolidation I
added a brief section about test types to the main CI portal:

https://fedoraproject.org/wiki/CI#Test_Types

Does it decribe well what we are looking for? Shall we extend it?

> In addition, this is what I fear most:
> The tests for the package are stored elsewhere, we're not sure
> where (the tests namespace, a random git repo on the
> internet...), the packager update package and the tests fail.

I see no hiding of tests from maintainers: Tests are clearly
referenced and discoverable from the rpms dist git as defined by
the Standard Test Interface and devels have access, see:

https://fedoraproject.org/wiki/CI/Share_Test_Code#Examples

Devel decides which tests are good for testing, it does not matter
where the test code lives. However there needs to be trust to the
referenced repo. But this is the same as with any other open
source project.

> What do you think will happen?
> a) the packager will look for $random_place_of_the_internet
> where tests are and spend time trying to fix them.
> b) the packager will turn off the tests because they get in the way
>
> If the packager wants to turn off the tests, they will have to
> go through dist-git to do it, they may very well end up turning
> the tests off anyway, but if the tests are right there, they may
> as well have a quick look at them to see if they can fix them
> quickly before deciding.

It's not random_place_of_the_internet, it's well selected and
trusted source where devel can (usually) directly contribute. I
meet devels around who see benefit in having tests enabled. So
I don't share your concern about turning the test off so easily.

> In addition, if the packager turn the tests off and the 

Re: New "tests" namespace to share test code

[Pierre-Yves Chibon]

On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
> Hi!
> 
> During the last days there have been concerns raised regarding
> what is an appropriate content for the tests namespace. [1] My
> original idea was to enable sharing tests even across branches of
> the same component, not only for tests to be used by completely
> different packages. The initial examples might have been a bit
> misleading in this respect. One of the main points still holds:
> 
> >  * Tests might follow a different branching pattern than the
> >dist-git repo, leading to code duplication
> 
> From the feedback from developers I feel they always keep on mind
> and care a lot about the maintenance costs. So it perfectly makes
> sense to me if they want to keep and maintain tests in a separate
> repo instead of merging/cherry-picking between dist-git branches.
> 
> When, for a particular package, it is the most efficient way to
> maintain tests in a separate repo why should we discourage from
> this approach? There are packages where it makes more sense to
> store test code directly in dist-git. And it is still an option.
> But why should we enforce this for all?

My worries are basically that this mechanism is hiding the tests from the
package maintainers. It splits the concerns between people maintaining the
artifacts and people maintaining the tests, which is exactly what the original
plan to bring CI was *not* about.
The idea has always been to bring the tests where the code lives, so that both
can be worked on as one, to make tests be a concern of the package maintainer
very much like updating the artifact (rpms, image..) is, while getting support
from QE for them (the tests).

In addition, this is what I fear most:
The tests for the package are stored elsewhere, we're not sure where (the tests
namespace, a random git repo on the internet...), the packager update package
and the tests fail.
What do you think will happen?
a) the packager will look for $random_place_of_the_internet where tests are and
   spend time trying to fix them.
b) the packager will turn off the tests because they get in the way

If the packager wants to turn off the tests, they will have to go through
dist-git to do it, they may very well end up turning the tests off anyway, but
if the tests are right there, they may as well have a quick look at them to see
if they can fix them quickly before deciding.

In addition, if the packager turn the tests off and the people maintaining the
tests do not realize that, they will end up spending time maintaining
$somewhere_else tests that aren't being used.
Having them interact directly with the dist-git repo will increase the chances
that they see/realize when something is turned off.

If that means we have less tests in dist-git because the maintainers do not want
them, I'd say so be it.
In the long term this is their loss, they are the ones who will get the bug
reports and will have to deal with them, they are the ones who will have to dive
into old code rather than going back into something that is still fresh in their
mind.
As long as, it is clear that they have been approached and that it is their
choice to not merge pull-requests adding tests, I think the people offering to
help should not be the ones blamed.


Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Adam Williamson]

On Thu, 2018-02-15 at 13:11 +0100, Petr Šplíchal wrote:
> 
> > The question 'who decides which tests block which packages' is
> > left a bit up in the air, but in fact no more so than it already
> > was for package-specific tests...
> 
> Right. Do we want to change this? Specify this more strictly?
> Like exactly defining requirements which an Always Ready Operating
> System has to meet? And then mapping these requirements to the
> test coverage? (Here again FMF mentioned above might come handy.)

Maybe we should just *formalize* it a bit, so folks know where they
stand. So far we've basically done three things that implicitly
describe the policy on this:

1) Bodhi has a mechanism by which the person submitting an update can
specify some tests that block the update

2) FESCo accepted, voted on and approved a ticket which involved a
decision that certain tests blocked some/all updates

3) We built and deployed WaiverDB and Greenwave such that Greenwave
currently accepts waivers submitted to WaiverDB by...(anybody??? this
point I'm not sure on)

So, effectively, we have communicated that FESCo can set mandatory
policies here, and packagers can also make voluntary choices. Point 3)
is rather a crucial one, and I don't know that it's been formally
discussed anywhere; I think we have been rather heavily focused on
building the tools as opposed to setting the policy so far. But we've
at least clearly implied so far that FESCo claims to some extent the
power to set automated test gating requirements for updates, and that
packagers have some ability to extend those requirements voluntarily,
and also submit waivers. I guess the natural thing to do from here, if
we want to be a bit more explicit about it, would be to extend the
Updates Policy:

https://fedoraproject.org/wiki/Updates_Policy

either in-line or with a subsidiary document, to define this stuff a
bit more clearly/formally.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

On 14 February 2018 at 18:18, Adam Williamson
 wrote:
> On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
>> Hi!
>>
>> During the last days there have been concerns raised regarding
>> what is an appropriate content for the tests namespace. [1] My
>> original idea was to enable sharing tests even across branches
>> of the same component, not only for tests to be used by
>> completely different packages. The initial examples might have
>> been a bit misleading in this respect. One of the main points
>> still holds:
>>
>> >  * Tests might follow a different branching pattern than the
>> >  dist-git repo, leading to code duplication
>>
>> From the feedback from developers I feel they always keep on
>> mind and care a lot about the maintenance costs. So it
>> perfectly makes sense to me if they want to keep and maintain
>> tests in a separate repo instead of merging/cherry-picking
>> between dist-git branches.
>>
>> When, for a particular package, it is the most efficient way to
>> maintain tests in a separate repo why should we discourage from
>> this approach? There are packages where it makes more sense to
>> store test code directly in dist-git. And it is still an
>> option.  But why should we enforce this for all?
>>
>> Please share your thoughts and real-life examples. For those
>> who are not familiar with the topic there is a new wiki page
>> with a summary of the Share Test Code approach [2].
>
> I don't have any problem with the concept *in itself*; in fact I
> know of another reason to have something like it. That is
> 'generic' tests, tests we want to run on all packages, or at
> least a very large set of packages - like the tests currently
> running in Taskotron, which are generally run on all packages
> (rpmgrill, rpmdeplint...) or a large subset (the Python tests).

Yes, sharing tests useful for multiple packages makes sense in
general. Test code for generic tests can be stored basically
anywhere but not in specific components. However I see some
additional benefits of having a dedicated tests namespace:

* Source of inspiration and real-life examples (at one place)
* Potential for workflow automation (tests CI, fedpkg --tests)
* Easy cherry-picking/backporting upstream/downstream tests

The difference I see when comparing these shared tests to generic
tests mentioned above like rpmdeplint is that you don't want to
reference these from the individual package repos (e.g. using the
Standard Test Interface), but just run them on all packages.

If the expectation is to run tests only on a subset of rpms then a
question comes: How to store the list of relevant components or
criteria to decide which packages are to be tested or not. Here
the Flexible Metadata Format might come handy in the future:

https://github.com/psss/fmf

> What I did think was maybe a concern is that we should figure
> out in advance the squishy human consequences of different
> technical approaches.
>
> Basically this boiled down to "who is responsible for these
> 'shared' tests, and who gets to decide which 'shared' tests
> can/should block packages?"
>
> Looking at the proposal, I think it actually has at least
> workable initial explicit/implicit answers to this, if I'm
> reading it correctly.  Anyone can create a shared test
> repository (and is therefore "responsible" for maintaining it),
> but the definition of which tests are run on which packages
> remains in the package repositories. Thus the package
> maintainer(s) retain the ultimate choice over which tests are
> run against their packages (and thus can pick which shared tests
> are run, and disable shared tests if they are no longer properly
> testing their package).

Yes. This is exactly true. The maintainer of the package repo has
always the last word on what is to be executed for qualification.
It can be done by referencing only suitable tests from the shared
repo or selecting an appropriate branch or even a specific commit
to ensure that future test development does not break stable
gating (and only manually update reference after review of shared
test updates) or even completely disable tests by removing the
reference.

> The question 'who decides which tests block which packages' is
> left a bit up in the air, but in fact no more so than it already
> was for package-specific tests...

Right. Do we want to change this? Specify this more strictly?
Like exactly defining requirements which an Always Ready Operating
System has to meet? And then mapping these requirements to the
test coverage? (Here again FMF mentioned above might come handy.)

psss...

> --
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
> http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Adam Williamson]

On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
> Hi!
> 
> During the last days there have been concerns raised regarding
> what is an appropriate content for the tests namespace. [1] My
> original idea was to enable sharing tests even across branches of
> the same component, not only for tests to be used by completely
> different packages. The initial examples might have been a bit
> misleading in this respect. One of the main points still holds:
> 
> >  * Tests might follow a different branching pattern than the
> >dist-git repo, leading to code duplication
> 
> From the feedback from developers I feel they always keep on mind
> and care a lot about the maintenance costs. So it perfectly makes
> sense to me if they want to keep and maintain tests in a separate
> repo instead of merging/cherry-picking between dist-git branches.
> 
> When, for a particular package, it is the most efficient way to
> maintain tests in a separate repo why should we discourage from
> this approach? There are packages where it makes more sense to
> store test code directly in dist-git. And it is still an option.
> But why should we enforce this for all?
> 
> Please share your thoughts and real-life examples. For those who
> are not familiar with the topic there is a new wiki page with a
> summary of the Share Test Code approach [2].

I don't have any problem with the concept *in itself*; in fact I know
of another reason to have something like it. That is 'generic' tests,
tests we want to run on all packages, or at least a very large set of
packages - like the tests currently running in Taskotron, which are
generally run on all packages (rpmgrill, rpmdeplint...) or a large
subset (the Python tests).

What I did think was maybe a concern is that we should figure out in
advance the squishy human consequences of different technical
approaches.

Basically this boiled down to "who is responsible for these 'shared'
tests, and who gets to decide which 'shared' tests can/should block
packages?"

Looking at the proposal, I think it actually has at least workable
initial explicit/implicit answers to this, if I'm reading it correctly.
Anyone can create a shared test repository (and is therefore
"responsible" for maintaining it), but the definition of which tests
are run on which packages remains in the package repositories. Thus the
package maintainer(s) retain the ultimate choice over which tests are
run against their packages (and thus can pick which shared tests are
run, and disable shared tests if they are no longer properly testing
their package).

The question 'who decides which tests block which packages' is left a
bit up in the air, but in fact no more so than it already was for
package-specific tests...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Honza Horak]

On 02/14/2018 05:28 PM, Petr Šplíchal wrote:

Hi!

During the last days there have been concerns raised regarding
what is an appropriate content for the tests namespace. [1] My
original idea was to enable sharing tests even across branches of
the same component, not only for tests to be used by completely
different packages. The initial examples might have been a bit
misleading in this respect. One of the main points still holds:


  * Tests might follow a different branching pattern than the
dist-git repo, leading to code duplication


 From the feedback from developers I feel they always keep on mind
and care a lot about the maintenance costs. So it perfectly makes
sense to me if they want to keep and maintain tests in a separate
repo instead of merging/cherry-picking between dist-git branches.


+1

From my PoV, it makes perfect sense to share tests between branches, or 
between different versions of packages available in modules streams (soon).


Honza


When, for a particular package, it is the most efficient way to
maintain tests in a separate repo why should we discourage from
this approach? There are packages where it makes more sense to
store test code directly in dist-git. And it is still an option.
But why should we enforce this for all?

Please share your thoughts and real-life examples. For those who
are not familiar with the topic there is a new wiki page with a
summary of the Share Test Code approach [2].

psss...

[1] https://pagure.io/fedora-infrastructure/issue/6695
[2] https://fedoraproject.org/wiki/CI/Share_Test_Code

On 7 December 2017 at 10:38, Petr Splichal  wrote:

Hi!

While working on adding CI tests [0] using the Standard Test
Interface a need arose to have a shared git repository where tests
could be stored:

  * A large number of test files makes a dist-git repository more
difficult to maintain

  * Tests might follow a different branching pattern than the
dist-git repo, leading to code duplication

  * Shared maintenance for tests sometimes benefits from different
access levels than the release dist-git repository

The plan is to create a new “tests” namespace in Fedora git/pagure
dedicated to storing the shared test code. To enable execution of
these tests by the CI pipeline, tests.yml file in dist-git will be
used to link the tests in the standard way as defined by the
Standard Test Interface [1].

This approach should help to efficiently maintain tests & minimize
test code duplication. Using a dedicated git repo for test code
also means to keep dist-git more as a place for storing metadata
only: Build metadata (spec file = how to build the package) and
test metadata (tests.yml = how to test the package) rather than
mixing spec files with test code itself.

Please note that this does not mean that all tests should now go
into this new namespace. You can still link tests directly from
upstream (like GitHub) or any other source. Also, for unit tests
it makes more sense to be kept directly with the project source
and executed there. Shared tests namespace in Fedora will be
suitable especially for functional and integration testing which
should help to continuously ensure the OS works as a whole.

For more detailed motivation and real-life examples see the Share
Test Code proposal on the Fedora CI list [2]. If you have any
questions or comments feel free to share them here or in the
pagure issue requesting the new namespace:

https://pagure.io/fedora-infrastructure/issue/6478

Thanks.

psss...

[0] https://fedoraproject.org/wiki/CI
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
[2] 
https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

Hi!

During the last days there have been concerns raised regarding
what is an appropriate content for the tests namespace. [1] My
original idea was to enable sharing tests even across branches of
the same component, not only for tests to be used by completely
different packages. The initial examples might have been a bit
misleading in this respect. One of the main points still holds:

>  * Tests might follow a different branching pattern than the
>dist-git repo, leading to code duplication

From the feedback from developers I feel they always keep on mind
and care a lot about the maintenance costs. So it perfectly makes
sense to me if they want to keep and maintain tests in a separate
repo instead of merging/cherry-picking between dist-git branches.

When, for a particular package, it is the most efficient way to
maintain tests in a separate repo why should we discourage from
this approach? There are packages where it makes more sense to
store test code directly in dist-git. And it is still an option.
But why should we enforce this for all?

Please share your thoughts and real-life examples. For those who
are not familiar with the topic there is a new wiki page with a
summary of the Share Test Code approach [2].

psss...

[1] https://pagure.io/fedora-infrastructure/issue/6695
[2] https://fedoraproject.org/wiki/CI/Share_Test_Code

On 7 December 2017 at 10:38, Petr Splichal  wrote:
> Hi!
>
> While working on adding CI tests [0] using the Standard Test
> Interface a need arose to have a shared git repository where tests
> could be stored:
>
>  * A large number of test files makes a dist-git repository more
>difficult to maintain
>
>  * Tests might follow a different branching pattern than the
>dist-git repo, leading to code duplication
>
>  * Shared maintenance for tests sometimes benefits from different
>access levels than the release dist-git repository
>
> The plan is to create a new “tests” namespace in Fedora git/pagure
> dedicated to storing the shared test code. To enable execution of
> these tests by the CI pipeline, tests.yml file in dist-git will be
> used to link the tests in the standard way as defined by the
> Standard Test Interface [1].
>
> This approach should help to efficiently maintain tests & minimize
> test code duplication. Using a dedicated git repo for test code
> also means to keep dist-git more as a place for storing metadata
> only: Build metadata (spec file = how to build the package) and
> test metadata (tests.yml = how to test the package) rather than
> mixing spec files with test code itself.
>
> Please note that this does not mean that all tests should now go
> into this new namespace. You can still link tests directly from
> upstream (like GitHub) or any other source. Also, for unit tests
> it makes more sense to be kept directly with the project source
> and executed there. Shared tests namespace in Fedora will be
> suitable especially for functional and integration testing which
> should help to continuously ensure the OS works as a whole.
>
> For more detailed motivation and real-life examples see the Share
> Test Code proposal on the Fedora CI list [2]. If you have any
> questions or comments feel free to share them here or in the
> pagure issue requesting the new namespace:
>
> https://pagure.io/fedora-infrastructure/issue/6478
>
> Thanks.
>
> psss...
>
> [0] https://fedoraproject.org/wiki/CI
> [1] https://fedoraproject.org/wiki/Changes/InvokingTests
> [2] 
> https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

New "tests" namespace to share test code

[Petr Splichal]

Hi!

While working on adding CI tests [0] using the Standard Test
Interface a need arose to have a shared git repository where tests
could be stored:

 * A large number of test files makes a dist-git repository more
   difficult to maintain

 * Tests might follow a different branching pattern than the
   dist-git repo, leading to code duplication

 * Shared maintenance for tests sometimes benefits from different
   access levels than the release dist-git repository

The plan is to create a new “tests” namespace in Fedora git/pagure
dedicated to storing the shared test code. To enable execution of
these tests by the CI pipeline, tests.yml file in dist-git will be
used to link the tests in the standard way as defined by the
Standard Test Interface [1].

This approach should help to efficiently maintain tests & minimize
test code duplication. Using a dedicated git repo for test code
also means to keep dist-git more as a place for storing metadata
only: Build metadata (spec file = how to build the package) and
test metadata (tests.yml = how to test the package) rather than
mixing spec files with test code itself.

Please note that this does not mean that all tests should now go
into this new namespace. You can still link tests directly from
upstream (like GitHub) or any other source. Also, for unit tests
it makes more sense to be kept directly with the project source
and executed there. Shared tests namespace in Fedora will be
suitable especially for functional and integration testing which
should help to continuously ensure the OS works as a whole.

For more detailed motivation and real-life examples see the Share
Test Code proposal on the Fedora CI list [2]. If you have any
questions or comments feel free to share them here or in the
pagure issue requesting the new namespace:

https://pagure.io/fedora-infrastructure/issue/6478

Thanks.

psss...

[0] https://fedoraproject.org/wiki/CI
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
[2] 
https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org