Flexible Metadata Format: Define Essential Attributes

[Petr Šplíchal]

Hi!

During the last weeks we've been experimenting with the Flexible
Metadata Format proof of concept on some real-life components:
There's a small SELinux example showing how FMF could be used for
filtering relevant tests [1], Jakub Krysl successfully used FMF
for generating different device setups for his VDO testing [2] and
Jan Scotka is now integrating FMF with the Meta Test Family.

As the next step we would like to start discussions about the test
metadata content, that is to define essential attributes which
should be stored close to the test code, directly in git repo. If
you are interested in contributing to this effort, please join our
discussions which will happen on the Fedora CI list.

I've created an initial draft for the first couple of attributes
on the Fedora wiki [3]. Please, review and share your thoughts.
Thanks.

psss...

[1] https://src.fedoraproject.org/tests/selinux/pull-request/1
[2] http://fmf.readthedocs.io/en/latest/examples.html#setups
[3] https://fedoraproject.org/wiki/CI/Metadata

On 23 January 2018 at 09:18, Petr Šplíchal <pspli...@redhat.com> wrote:
> Hi,
>
> simple proof of concept is ready for experimenting:
>
> https://github.com/psss/fmf
> http://fmf.readthedocs.io/
>
> Looking for the first impressions & feedback. Thanks.
>
> psss...
>
> On 8 January 2018 at 15:49, Petr Splichal <pspli...@redhat.com> wrote:
>>
>> Hi!
>>
>> In order to keep test execution efficient when number of test
>> cases grows, it is crucial to maintain corresponding metadata,
>> which define some aspects of how the test coverage is executed.
>> For example limiting environment combinations where the test is
>> relevant or selecting a subset of important test cases for quick
>> verification of essential features when testing a security update.
>>
>> Within the BaseOS QE team we were thinking (for a long time) about
>> an efficient metadata solution which would cover our use cases and
>> would be open source. Recently we've been involved in the Upstream
>> First initiative which increased the need for an open metadata
>> solution which would enable us to more easily share test code
>> between Red Hat Enterprise Linux and Fedora.
>>
>> We've put together a draft solution which covers some of the most
>> important stories we've gathered so far. It does not cover all use
>> cases and it is not complete. In this early stage we would like to
>> invite others who might have similar use cases to gather your
>> feedback, share your experience or even join the project:
>>
>> https://fedoraproject.org/wiki/Flexible_Metadata_Format
>>
>> The page lists some of our core user stories as well as a couple of
>> real-life examples to demonstrate proposed features of the format.
>> Can you see similar user stories in your team? Is this something
>> that could be useful for you as well? Do you know of a different
>> solution for these use cases? Any other relevant ideas?
>>
>> To illustrate where we could be heading: In the ideal future there
>> could be just a single test case for a particular feature stored
>> in public with a single set of metadata attached close to the test
>> code and together used for testing in both upstream and downstream
>> without need to duplicate the test code (maintain both copies).
>>
>> This proposal does not suggest in any way to replace tests.yml [1]
>> files defined by the Standard Test Interface. The new format could
>> serve as an extension for selecting the right tests to be executed
>> (e.g. filtering tests by tag instead of listing them manually).
>>
>> Looking forward to your feedback!
>>
>> psss...
>>
>> [1] https://fedoraproject.org/wiki/CI/Tests
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Flexible Metadata Format

[Petr Šplíchal]

Hi,

simple proof of concept is ready for experimenting:

https://github.com/psss/fmf
http://fmf.readthedocs.io/

Looking for the first impressions & feedback. Thanks.

psss...

On 8 January 2018 at 15:49, Petr Splichal  wrote:

> Hi!
>
> In order to keep test execution efficient when number of test
> cases grows, it is crucial to maintain corresponding metadata,
> which define some aspects of how the test coverage is executed.
> For example limiting environment combinations where the test is
> relevant or selecting a subset of important test cases for quick
> verification of essential features when testing a security update.
>
> Within the BaseOS QE team we were thinking (for a long time) about
> an efficient metadata solution which would cover our use cases and
> would be open source. Recently we've been involved in the Upstream
> First initiative which increased the need for an open metadata
> solution which would enable us to more easily share test code
> between Red Hat Enterprise Linux and Fedora.
>
> We've put together a draft solution which covers some of the most
> important stories we've gathered so far. It does not cover all use
> cases and it is not complete. In this early stage we would like to
> invite others who might have similar use cases to gather your
> feedback, share your experience or even join the project:
>
> https://fedoraproject.org/wiki/Flexible_Metadata_Format
>
> The page lists some of our core user stories as well as a couple of
> real-life examples to demonstrate proposed features of the format.
> Can you see similar user stories in your team? Is this something
> that could be useful for you as well? Do you know of a different
> solution for these use cases? Any other relevant ideas?
>
> To illustrate where we could be heading: In the ideal future there
> could be just a single test case for a particular feature stored
> in public with a single set of metadata attached close to the test
> code and together used for testing in both upstream and downstream
> without need to duplicate the test code (maintain both copies).
>
> This proposal does not suggest in any way to replace tests.yml [1]
> files defined by the Standard Test Interface. The new format could
> serve as an extension for selecting the right tests to be executed
> (e.g. filtering tests by tag instead of listing them manually).
>
> Looking forward to your feedback!
>
> psss...
>
> [1] https://fedoraproject.org/wiki/CI/Tests
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Flexible Metadata Format

[Petr Splichal]

Hi!

In order to keep test execution efficient when number of test
cases grows, it is crucial to maintain corresponding metadata,
which define some aspects of how the test coverage is executed.
For example limiting environment combinations where the test is
relevant or selecting a subset of important test cases for quick
verification of essential features when testing a security update.

Within the BaseOS QE team we were thinking (for a long time) about
an efficient metadata solution which would cover our use cases and
would be open source. Recently we've been involved in the Upstream
First initiative which increased the need for an open metadata
solution which would enable us to more easily share test code
between Red Hat Enterprise Linux and Fedora.

We've put together a draft solution which covers some of the most
important stories we've gathered so far. It does not cover all use
cases and it is not complete. In this early stage we would like to
invite others who might have similar use cases to gather your
feedback, share your experience or even join the project:

https://fedoraproject.org/wiki/Flexible_Metadata_Format

The page lists some of our core user stories as well as a couple of
real-life examples to demonstrate proposed features of the format.
Can you see similar user stories in your team? Is this something
that could be useful for you as well? Do you know of a different
solution for these use cases? Any other relevant ideas?

To illustrate where we could be heading: In the ideal future there
could be just a single test case for a particular feature stored
in public with a single set of metadata attached close to the test
code and together used for testing in both upstream and downstream
without need to duplicate the test code (maintain both copies).

This proposal does not suggest in any way to replace tests.yml [1]
files defined by the Standard Test Interface. The new format could
serve as an extension for selecting the right tests to be executed
(e.g. filtering tests by tag instead of listing them manually).

Looking forward to your feedback!

psss...

[1] https://fedoraproject.org/wiki/CI/Tests
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org