Flexible Metadata Format

[Petr Splichal]

Hi!

In order to keep test execution efficient when number of test
cases grows, it is crucial to maintain corresponding metadata,
which define some aspects of how the test coverage is executed.
For example limiting environment combinations where the test is
relevant or selecting a subset of important test cases for quick
verification of essential features when testing a security update.

Within the BaseOS QE team we were thinking (for a long time) about
an efficient metadata solution which would cover our use cases and
would be open source. Recently we've been involved in the Upstream
First initiative which increased the need for an open metadata
solution which would enable us to more easily share test code
between Red Hat Enterprise Linux and Fedora.

We've put together a draft solution which covers some of the most
important stories we've gathered so far. It does not cover all use
cases and it is not complete. In this early stage we would like to
invite others who might have similar use cases to gather your
feedback, share your experience or even join the project:

https://fedoraproject.org/wiki/Flexible_Metadata_Format

The page lists some of our core user stories as well as a couple of
real-life examples to demonstrate proposed features of the format.
Can you see similar user stories in your team? Is this something
that could be useful for you as well? Do you know of a different
solution for these use cases? Any other relevant ideas?

To illustrate where we could be heading: In the ideal future there
could be just a single test case for a particular feature stored
in public with a single set of metadata attached close to the test
code and together used for testing in both upstream and downstream
without need to duplicate the test code (maintain both copies).

This proposal does not suggest in any way to replace tests.yml [1]
files defined by the Standard Test Interface. The new format could
serve as an extension for selecting the right tests to be executed
(e.g. filtering tests by tag instead of listing them manually).

Looking forward to your feedback!

psss...

[1] https://fedoraproject.org/wiki/CI/Tests
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

New "tests" namespace to share test code

[Petr Splichal]

Hi!

While working on adding CI tests [0] using the Standard Test
Interface a need arose to have a shared git repository where tests
could be stored:

 * A large number of test files makes a dist-git repository more
   difficult to maintain

 * Tests might follow a different branching pattern than the
   dist-git repo, leading to code duplication

 * Shared maintenance for tests sometimes benefits from different
   access levels than the release dist-git repository

The plan is to create a new “tests” namespace in Fedora git/pagure
dedicated to storing the shared test code. To enable execution of
these tests by the CI pipeline, tests.yml file in dist-git will be
used to link the tests in the standard way as defined by the
Standard Test Interface [1].

This approach should help to efficiently maintain tests & minimize
test code duplication. Using a dedicated git repo for test code
also means to keep dist-git more as a place for storing metadata
only: Build metadata (spec file = how to build the package) and
test metadata (tests.yml = how to test the package) rather than
mixing spec files with test code itself.

Please note that this does not mean that all tests should now go
into this new namespace. You can still link tests directly from
upstream (like GitHub) or any other source. Also, for unit tests
it makes more sense to be kept directly with the project source
and executed there. Shared tests namespace in Fedora will be
suitable especially for functional and integration testing which
should help to continuously ensure the OS works as a whole.

For more detailed motivation and real-life examples see the Share
Test Code proposal on the Fedora CI list [2]. If you have any
questions or comments feel free to share them here or in the
pagure issue requesting the new namespace:

https://pagure.io/fedora-infrastructure/issue/6478

Thanks.

psss...

[0] https://fedoraproject.org/wiki/CI
[1] https://fedoraproject.org/wiki/Changes/InvokingTests
[2] 
https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org