tmt hint 02: Under The Hood

[Petr Šplíchal]

Hi!

Based on the feedback from users we've put together the second
chapter of the Test Management Tool guide: Let's together dive a
little bit more and look Under The Hood [0] of tmt to see how
plans, tests and stories work together on a couple of examples:

Plans [1] are used to enable testing and group relevant tests
together. They describe how to discover tests for execution, how
to provision the environment, how to prepare it for testing, how
to execute tests, report results and finally how to finish the
test job.

provision:
how: container
image: fedora:33
prepare:
how: install
package: wget
execute:
how: tmt
script: wget http://example.org/

Tests [2] define attributes which are closely related to
individual test cases such as the test script, framework,
directory path where the test should be executed, maximum test
duration or packages required to run the test.

summary: Fetch an example web page
test: wget http://example.org/
require: wget
duration: 1m

Stories [3] can be used to track implementation, test and
documentation coverage for individual features or requirements.
Thanks to this you can track everything in one place, including
the project implementation progress.

story:
As a user I want to see more detailed information for
particular command.
example:
  - tmt test show -v
  - tmt test show -vvv
  - tmt test show --verbose

Core [4] attributes cover general metadata such as summary or
description for describing the content, the enabled attribute for
disabling and enabling tests, plans and stories and the link key
which can be used for tracking relations between objects.

description:
Different verbose levels can be enabled by using the
option several times.
link:
  - implemented-by: /tmt/cli.py
  - documented-by: /tmt/cli.py
  - verified-by: /tests/core/dry

Have a look at the whole chapter [0] to learn more details and get
some more context. See the Fedora Guide [6] to learn even more
about enabling tmt tests in the CI.

psss...

P.S. The tmt-1.7.0 [5] is available in Fedora repos and contains a
couple of new interesting features such as environment files,
parametrized plans, improved image guessing, experimental support
for reboot in the middle of the test plus a bunch of other
improvements and bugfixes.

[0] https://tmt.readthedocs.io/en/latest/guide.html#under-the-hood
[1] https://tmt.readthedocs.io/en/latest/guide.html#plans
[2] https://tmt.readthedocs.io/en/latest/guide.html#tests
[3] https://tmt.readthedocs.io/en/latest/guide.html#stories
[4] https://tmt.readthedocs.io/en/latest/guide.html#core
[5] https://github.com/psss/tmt/releases/tag/1.7.0
[6] https://docs.fedoraproject.org/en-US/ci/tmt/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

tmt hint 01

[Petr Šplíchal]

Hi!

After the initial hint [1] describing the very first steps with
tmt let's have a look at the available test execution options.
The following user story was at the very beginning of tmt:

As a tester or developer, I want to easily run tests
in my preferred environment.

Do you want to safely run tests without breaking your laptop? Use
the default provision method 'virtual' which will execute tests
under a virtual machine using libvirt with the help of testcloud:

tmt run

Would you like to execute tests a bit faster and don't need the
full virtualization support? Run them in a container using podman:

tmt run --all provision --how container

Do you have an already prepared box where everything's ready for
testing and you often ssh to it to experiment safely?

tmt run --all provision --how connect --guest my.test.box

Do you know exactly what the tests are doing and feel safe to
quickly run them directly on your localhost?

tmt run -a provision -h local

Note that some provision methods require additional dependencies.
Installing them is easy as well:

sudo dnf install -y tmt-provision-container
sudo dnf install -y tmt-provision-virtual

See the tmt guide [2] and examples [3] for some more inspiration.

Happy testing! :-)

psss...

[1] https://communityblog.fedoraproject.org/tmt-hints-basic-test/
[2] https://tmt.readthedocs.io/en/latest/guide.html
[3] https://tmt.readthedocs.io/en/latest/examples.html
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

tmt hint 00

[Petr Šplíchal]

Hi!

For those who still haven't heard: tmt [1] is now fully supported
in Packit [2], Fedora CI [3] and the RHEL CI. This means it is
possible to use the same concise and consistent config to enable
tests across all of them, more easily open source tests, share
test coverage across releases and run tests as early as possible.

In the coming weeks we'll be sharing short, bite-sized examples
demonstrating the tmt usage so that new users can get quickly
started and existing users do not miss various interesting and
useful features hidden under the hood.

Here we go with the first set of examples showing how to quickly
enable a simple smoke test for your component, assuming you are in
your project git repository:

sudo dnf install -y tmt
cd git/fedora/rpms/foo
tmt init --template mini
vim plans/example.fmf

Adjust the example plan to run the desired command:

summary: Basic smoke test
execute:
script: foo --version

The very minimal config is really just two lines:

execute:
script: make test

Now submit the pull request and wait for the results:

git add .
git checkout -b smoke-test
git commit -m "Enable a simple smoke test"
git push fork -u smoke-test

Eager to learn more? Not patient enough to wait for the results
from the CI pipeline? Willing to safely execute tests from your
laptop right now? Check the rest of the first chapter of our brand
new guide to learn more [4].

psss...

P.S. Join the #tmt channel if you have any questions.

[1] https://tmt.readthedocs.io/
[2] https://packit.dev/docs/testing-farm/
[3] https://docs.fedoraproject.org/en-US/ci/tmt/
[4] https://tmt.readthedocs.io/en/stable/guide.html#the-first-steps
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: src.fedoraproject.org branch conversion to rawhide/main tomorrow

[Petr Šplíchal]

On Wed, 3 Feb 2021 at 17:42, Petr Lautrbach  wrote:

> Petr Lautrbach  writes:
>
> > Kevin Fenzi  writes:
> >
> >> Greetings everyone.
> >>
> >> We finally have everything in place and hopefully tested to make the
> >> switch tomorrow from master to rawhide/main branches for
> >> src.fedoraproject.org.
> >>
> >> At 13:30UTC we will adjust pagure to reject pushes to 'master' and then
> >> will be moving all the branches over to rawhide/main. As soon as all
> >> packages are done, we will be adjusting pdc/hooks/existing pr's.
> >>
> >> We will be sending an additional email once changes are complete and
> >> hopefully any downtime will be limited.
> >>
> >> Once the change is completed you will want to checkout rawhide/main
> >> instead of master and update/recreate any existing forks you have.
> >>
> >> See
> >> https://fedoraproject.org/wiki/Changes/GitRepos-master-to-main
> >> for more information.
> >>
> >
> > The page says "This Change will move many repositories (see below) to
> > use a "main" branch as default." and "Not every namespace on dist-git
> > has a rawhide version. For example: containers do not have/use rawhide.
> > And having different default branches on different namespaces is not
> > very appealing."
> >
> > Now we have https://src.fedoraproject.org/tests/selinux/ with default
> > branch "rawhide". "rawhide" doesn't make sense in this repo as it
> > contains tests used on all Fedora versions and also downstream Red Hat
> > Enterprise Linux.
> >
> > Have I missed something?
> >
> > Petr
>
> I've tried to change it on my own:
>
> 1. create and push new branch "real-main"
> 2. set "real-main" as default branch
> 3.
> ^&^ git push origin :rawhide
> remote: Branch deletion is not allowed
> remote: Denied push for ref 'refs/heads/rawhide' for user 'plautrba'
> remote: All changes have been rejected
> To ssh://pkgs.fedoraproject.org/tests/selinux.git
>  ! [remote rejected]   rawhide (pre-receive hook declined)
> error: failed to push some refs to 'ssh://
> pkgs.fedoraproject.org/tests/selinux.git'
>
> 4. remove alias main -> rawhide at
> https://src.fedoraproject.org/tests/selinux/settings#gitbranch-tab
>
> pagure reports: "Alias deleted"
>
> 5. refresh the setting page
>
> 6. Alias "main" is still there
> https://src.fedoraproject.org/tests/selinux/settings#gitbranch-tab
>
> what now? revert changes back to misleading "rawhide"? use "master" as
> it's directly referenced in tests we use in Red Hat and solve strange
> default branch name later?
>

In addition to fixing the "rawhide" name, would it be possible to provide a
temporary symlink "master --> main"? At least in the fedora tests
namespace? As Petr mentioned, we have a bunch of tests which are referenced
using "master". All those tests are now broken. It would help us a lot if
there would be a 1-2 month transition period during which both "master" and
"main" would work until we fix the configs and tools, and get packages
released. Thanks.

psss...

Petr
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Package downgrades from f32 -> f33 (categorized list inside)

[Petr Šplíchal]

Hi!

> # Packages not included in mass rebuild?
>
> - tmt is newer in 32 than in 33:
>   0:0.20-1.fc32 > 0:0.19-1.fc33

Recently I was unable to build a fresh tmt package for rawhide
because of a missing build dependency:

https://koji.fedoraproject.org/koji/taskinfo?taskID=48098490
https://koji.fedoraproject.org/koji/taskinfo?taskID=50191342

Problem: conflicting requests
- nothing provides libguestfs-tools-c needed by
python3-testcloud-0.3.5-3.fc33.noarch

Seems the problem happens only for some build host architectures
(although tmt is noarch). Today I was able to build the package
successfully (on another build host). Any recommendation on how
to prevent these random failures? Thanks.

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: As a developer I want to easily execute/enable tests

[Petr Šplíchal]

Hi Dominik,

Thank you for your questions. See my comments inline.

On Thu, 23 Jul 2020 at 14:39, Dominik 'Rathann' Mierzejewski
 wrote:
>
> On Wednesday, 22 July 2020 at 14:49, Petr Šplíchal wrote:
> > Hi!
> >
> > I am writing today to share a proposal for how we can make it
> > easier to enable tests in Fedora CI.
> >
> > We've heard the user story mentioned in the subject many times.
> > Fedora developers have frequently mentioned the difficulties which
> > they face with enabling tests in Fedora CI. Some people don't
> > understand the complex yaml syntax. Other people find that they
> > can't easily reproduce issues revealed by Fedora CI on their
> > laptops. People have asked us to provide a tool for local testing:
> >
> > https://pagure.io/fedora-ci/general/issue/4
> >
> > A few of us have been working on a tool which we hope will improve
> > the situation considerably. The proof of concept below shows how
> > everyday developer tasks (related to testing) might look in the
> > near future:
> >
> > Enable basic smoke test for my component:
> >
> > dnf install -y tmt
> > tmt init --template mini
> > vim plans/example.fmf
> > git add . && git commit -m tests && git push
> >
> > Create a new test based on a shell/beakerlib template:
> >
> > tmt test create --template shell tests/basic
> > tmt test create --template beakerlib tests/advanced
> >
> > Run all/selected tests safely in a virtual machine:
> >
> > tmt run
> > tmt run test --name smoke
>
> I'd say it's good to have such option, but for quick testing a full VM
> is often too heavy and slow to boot. Even a container takes time to set
> up and run.

Yes, there are different needs and preferences related to the
environment where the tests should be executed. We want to provide
enough flexibility here so the 'provision' step allows you to
choose your preferred way.

> My primary environment would be local.

So for your case that would be 'provision --how local'. If you
have a test box prepared for testing you could also connect to it
using 'provision --how connect'.

> > Execute tests in my preferred environment:
> >
> > tmt run --all provision --how container
> > tmt run --all provision --how virtual
> > tmt run --all provision --how local
> > tmt run --all provision --how ...
>
> I find it difficult to guess what the above commands actually do.
> Do they run the tests or do they provision environments to run the
> tests? Or both?

The 'tmt run' command is used to run tests. By default it runs all
steps (discover, provision, prepare, execute, report, finish) so
the examples above would run tests in the requested environment.

It is also possible to run only selected steps: For example, one
of the use cases shared with us was 'just tell me what tests would
be run, do not run them', this can be achieved with the following:

tmt run discover

For detailed description of individual steps have a look here:
https://tmt.readthedocs.io/en/latest/spec/steps.html

> > I would like to ask for your feedback about this. What do you
> > think? Is this something that would be useful for you? What do
> > you think of the syntax? We would love to hear back from you
> > concerning what essential features you might expect from such a
> > tool. It would also be helpful to know if there is anything that
> > should be changed early in the tool design.
> >
> > For more examples and frequently asked questions have a look here:
> >
> > https://tmt.readthedocs.io/en/latest/examples.html
> > https://docs.fedoraproject.org/en-US/ci/tmt/
> >
> > Looking forward to your feedback and ideas! :)
>
> tmt examples are overcomplicated and not explained in enough detail.
> What is "metadata" and why do you need 4 "levels" of it?

Metadata levels allow to store all needed test execution data at
appropriate location, the levels are described here:
https://tmt.readthedocs.io/en/latest/spec.html

We want the config for simple/common use cases (such as running a
shell script as a test) as simple as possible so that users do not
have to study all the details but just create something like this:

execute:
script: ./run-my-test.sh

If one needs to maintain larger set of tests the L1 metadata can
come handy:
https://tmt.readthedocs.io/en/latest/spec/tests.html

> What are "tiers" for?

They are used to organize tests into groups by priority.
https://tmt.readthedocs.io/en/latest/spec/tests.html#tier

> Which items are mandatory and which are optional?

In order to enable tests you just have to define the 

As a developer I want to easily execute/enable tests

[Petr Šplíchal]

Hi!

I am writing today to share a proposal for how we can make it
easier to enable tests in Fedora CI.

We've heard the user story mentioned in the subject many times.
Fedora developers have frequently mentioned the difficulties which
they face with enabling tests in Fedora CI. Some people don't
understand the complex yaml syntax. Other people find that they
can't easily reproduce issues revealed by Fedora CI on their
laptops. People have asked us to provide a tool for local testing:

https://pagure.io/fedora-ci/general/issue/4

A few of us have been working on a tool which we hope will improve
the situation considerably. The proof of concept below shows how
everyday developer tasks (related to testing) might look in the
near future:

Enable basic smoke test for my component:

dnf install -y tmt
tmt init --template mini
vim plans/example.fmf
git add . && git commit -m tests && git push

Create a new test based on a shell/beakerlib template:

tmt test create --template shell tests/basic
tmt test create --template beakerlib tests/advanced

Run all/selected tests safely in a virtual machine:

tmt run
tmt run test --name smoke

Execute tests in my preferred environment:

tmt run --all provision --how container
tmt run --all provision --how virtual
tmt run --all provision --how local
tmt run --all provision --how ...

I would like to ask for your feedback about this. What do you
think? Is this something that would be useful for you? What do
you think of the syntax? We would love to hear back from you
concerning what essential features you might expect from such a
tool. It would also be helpful to know if there is anything that
should be changed early in the tool design.

For more examples and frequently asked questions have a look here:

https://tmt.readthedocs.io/en/latest/examples.html
https://docs.fedoraproject.org/en-US/ci/tmt/

Looking forward to your feedback and ideas! :)

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

tmt, a tool for testing

[Petr Šplíchal]

Hi!

With a couple of qe & devel guys who are involved in the CI
effort we've started work on creating tmt, a tool for testing.
Why? We've got a bunch of dreams [1] and we would like to:

* run, debug and develop tests in much more comfortable way [2]
* open source more tests and run them closer to the upstream
* make creating and enabling tests in the CI more user friendly

We've put together the first proof-of-concept which outlines the
command line syntax. We would like to ask for early feedback and
invite anybody who would like to collaborate on this to join us:

https://github.com/psss/tmt

The easiest way how to start experimenting with the latest bits is
to install the tool directly from the git repo as it contains also
a set of real life tests/plans/stories for instant experimenting:

mkvirtualenv tmt # use [3] to install virtualenvwrapper
git clone https://github.com/psss/tmt
cd tmt
pip install -e .
tmt
tmt test ls
tmt plan show
tmt story show create
tmt story coverage --implemented
...

By creating the tool we aim to implement the L1 and L2 metadata
specification [4] and cover user stories [5] which we've gathered
so far. Feel free to review existing stories and provide feedback
or create new stories which are not covered yet.

Also note, that the repository has enabled Packit / Testing Farm
integration [6] which shows how you can automatically get test
results on copr builds directly in your pull requests, e.g. [7].

We are looking for any feedback.

psss...

[1] https://tmt.readthedocs.io/en/latest/dreams.html
[2] https://pagure.io/fedora-ci/general/issue/4
[3] dnf install -y python3-virtualenvwrapper && bash
[4] https://pagure.io/fedora-ci/metadata
[5] https://tmt.readthedocs.io/en/latest/stories.html
[6] https://packit.dev/testing-farm/
[7] https://github.com/psss/tmt/pull/9
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Running a test from dist-git

[Petr Šplíchal]

On Thu, 11 Oct 2018 at 09:49, Richard W.M. Jones  wrote:
>
> On Wed, Oct 10, 2018 at 11:26:01PM +0200, Miroslav Vadkerti wrote:
> > Hi,
> >
> > sorry for the late response :(
> >
> > On Tue, Oct 2, 2018 at 9:42 AM Richard W.M. Jones  wrote:
> >
> > > On Mon, Oct 01, 2018 at 11:26:59AM +0200, Miroslav Vadkerti wrote:
> > > > For this time I rescheduled the run:
> > > >
> > > >
> > > >
> > > https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-build-pipeline/detail/fedora-rawhide-build-pipeline/850/pipeline
> > >
> > > It appears to have failed, but I've got no clue why.  How do I see
> > > the output of the command?
> > >
> >
> > Ugh, logs are now gone. Which package were you testing? We can take a look
> > what was the problem.
>
> https://src.fedoraproject.org/rpms/libguestfs
>
> The test is to run a single command (libguestfs-test-tool).  This
> command is already installed by the package, so there's no need for
> any test harness or script.  If the command fails (ie. exit code != 0)
> then it's vital we see the complete output in order to diagnose the
> problem.
>
> https://src.fedoraproject.org/rpms/libguestfs/blob/master/f/tests/tests.yml

I've created pull request with fixed tests.yml file:

https://src.fedoraproject.org/rpms/libguestfs/pull-request/4

In the pull request you would directly see link to the test
results including the complete output of the test. Unfortunately
there is currently issue with propagating results to Pagure:

https://pagure.io/fedora-ci/general/issue/10

We are currently looking into this, it should be fixed soon.
Meanwhile it's still possible to see results of the pipeline here:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/activity/
https://fedoraproject.org/wiki/CI/Pipeline#Instances

The latest libguestfs job is this:

https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/jenkins/fedora-rawhide-pr-pipeline/detail/fedora-rawhide-pr-pipeline/459/pipeline

Test output can be found under that "Artifact" tab:

https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-pr-pipeline/459/artifact/package-tests/logs/PASS_libguestfs-test-tool.log

Hope this helps.

psss...

> Ideally we'd want to run the test as root and non-root, but for now I
> just had it run the test as root.
>
> > > I think it would be easier if we could just drop a shell script into
> > > tests/ and have it run that and display the output.  The current
> > > system seems elaborately overengineered and I don't understand why.
> > >
> >
> > Well, it was decided that we will use Ansible to wrap around our tests. Not
> > my idea. I agree the results could be presented better though, even if the
> > runner is Ansbile. We are preparing a dashboard to view the result of
> > testing, to give a better user experience in the future.
>
> It wasn't very obvious to start off with, but it seems as if the
> "standard test roles" are designed for the simple case of "just run a
> script".  However the use of YAML combined with the fact that you
> can't easily run a test locally makes the whole thing very awkward.
> (Also for some reason I didn't investigate yet, my copy of emacs
> appears to hate YAML.)
>
> > If the pipeline goes through (i.e. tests were run), the pipeline ends up
> > with Pass (test passed) or Unstable state (tests failed). If the test
> > failed, the pipeline did not execute the testing. In the former case you
> > should find logs in the Artifacts tab in Jenkins (test results are prefixed
> > with PASS_ or FAIL_). If you encounter an error, we are here to help an
> > investigate.
>
> Do the tests run if we submit scratch builds?

AFAIK, testing on scratch builds is not currently enabled.

>
> Anyway feel free to submit a new build of libguestfs if you want to
> see if you can make the test run.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Consistent CI Messages

[Petr Šplíchal]

On Tue, 21 Aug 2018 at 15:22, Jeremy Cline  wrote:
>
> Hi Petr,
>
> On 08/21/2018 11:50 AM, Petr Šplíchal wrote:
> > Hi,
> >
> > as part of bringing upstream and downstream workflows related to
> > testing one more step closer together and allow easier automation
> > tools development and sharing between Fedora, Red Hat Enterprise
> > Linux and other products, the CI team is proposing to use a
> > consistent format for the CI related messages:
> >
> > https://pagure.io/fedora-ci/messages
> >
> > The specification (currently describing koji-build and brew-build
> > messages) is written in self-documented YAML files defining a
> > JSON SCHEMA which can be used to validate the message format.
> > Included is also a set of example messages to get a quick start.
> >
> > Please, review the proposed format and share your feedback.
> > Thanks!
>
> I've only skimmed the format and I don't work on anything that consumes
> these messages, so I can't provide great feedback on that. A couple of
> quick notes on the schema itself, though:

Thanks for the review, Jeremy.

> * "generated_at" is in a human-readable format, but doesn't include the
>   timezone (UTC). It would be nice to include that, or use a machine-
>   readable format and keep it documented as being UTC. You should also
>   be able to enforce this with JSON schema's "date-time" type if you
>   opt to go the human-readable route.

Yes, this is expected to be UTC as defined by the spec here:
https://pagure.io/fedora-ci/messages/blob/master/f/schemas/common.yaml#_20
Also the description suggests using the ISO format so I guess we
should be set here.

> * URLs can be of type "uri" and JSON schema will enforce that.

That for the hint. I've updated schemas to specify format as well
(simply using uri as type did not work):
https://pagure.io/fedora-ci/messages/c/c0cdee9
Unfortunately it seems "python-jsonschema" does not enforce this.

> I'm curious, are you aware of https://fedora-messaging.readthedocs.io/
> and generally the plan for Fedora infrastructure to move to AMQP?
> fedora-messaging uses JSON schema as well and enforces the schema when
> sending or receiving.

Didn't know that. Thanks for the pointer! Will have a look. Do you
know what's the time frame when the migration should happen?

psss...

>
>
> --
> Jeremy Cline
> XMPP: jer...@jcline.org
> IRC:  jcline
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/APFPP6LKQSQGVA2TMGXWMM2CQSYFVTE7/

Consistent CI Messages

[Petr Šplíchal]

Hi,

as part of bringing upstream and downstream workflows related to
testing one more step closer together and allow easier automation
tools development and sharing between Fedora, Red Hat Enterprise
Linux and other products, the CI team is proposing to use a
consistent format for the CI related messages:

https://pagure.io/fedora-ci/messages

The specification (currently describing koji-build and brew-build
messages) is written in self-documented YAML files defining a
JSON SCHEMA which can be used to validate the message format.
Included is also a set of example messages to get a quick start.

Please, review the proposed format and share your feedback.
Thanks!

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/3DZK35E6WWWVFHTPVQCR4KKRG4DHOPE2/

Flexible Metadata Format: Define Essential Attributes

[Petr Šplíchal]

Hi!

During the last weeks we've been experimenting with the Flexible
Metadata Format proof of concept on some real-life components:
There's a small SELinux example showing how FMF could be used for
filtering relevant tests [1], Jakub Krysl successfully used FMF
for generating different device setups for his VDO testing [2] and
Jan Scotka is now integrating FMF with the Meta Test Family.

As the next step we would like to start discussions about the test
metadata content, that is to define essential attributes which
should be stored close to the test code, directly in git repo. If
you are interested in contributing to this effort, please join our
discussions which will happen on the Fedora CI list.

I've created an initial draft for the first couple of attributes
on the Fedora wiki [3]. Please, review and share your thoughts.
Thanks.

psss...

[1] https://src.fedoraproject.org/tests/selinux/pull-request/1
[2] http://fmf.readthedocs.io/en/latest/examples.html#setups
[3] https://fedoraproject.org/wiki/CI/Metadata

On 23 January 2018 at 09:18, Petr Šplíchal <pspli...@redhat.com> wrote:
> Hi,
>
> simple proof of concept is ready for experimenting:
>
> https://github.com/psss/fmf
> http://fmf.readthedocs.io/
>
> Looking for the first impressions & feedback. Thanks.
>
> psss...
>
> On 8 January 2018 at 15:49, Petr Splichal <pspli...@redhat.com> wrote:
>>
>> Hi!
>>
>> In order to keep test execution efficient when number of test
>> cases grows, it is crucial to maintain corresponding metadata,
>> which define some aspects of how the test coverage is executed.
>> For example limiting environment combinations where the test is
>> relevant or selecting a subset of important test cases for quick
>> verification of essential features when testing a security update.
>>
>> Within the BaseOS QE team we were thinking (for a long time) about
>> an efficient metadata solution which would cover our use cases and
>> would be open source. Recently we've been involved in the Upstream
>> First initiative which increased the need for an open metadata
>> solution which would enable us to more easily share test code
>> between Red Hat Enterprise Linux and Fedora.
>>
>> We've put together a draft solution which covers some of the most
>> important stories we've gathered so far. It does not cover all use
>> cases and it is not complete. In this early stage we would like to
>> invite others who might have similar use cases to gather your
>> feedback, share your experience or even join the project:
>>
>> https://fedoraproject.org/wiki/Flexible_Metadata_Format
>>
>> The page lists some of our core user stories as well as a couple of
>> real-life examples to demonstrate proposed features of the format.
>> Can you see similar user stories in your team? Is this something
>> that could be useful for you as well? Do you know of a different
>> solution for these use cases? Any other relevant ideas?
>>
>> To illustrate where we could be heading: In the ideal future there
>> could be just a single test case for a particular feature stored
>> in public with a single set of metadata attached close to the test
>> code and together used for testing in both upstream and downstream
>> without need to duplicate the test code (maintain both copies).
>>
>> This proposal does not suggest in any way to replace tests.yml [1]
>> files defined by the Standard Test Interface. The new format could
>> serve as an extension for selecting the right tests to be executed
>> (e.g. filtering tests by tag instead of listing them manually).
>>
>> Looking forward to your feedback!
>>
>> psss...
>>
>> [1] https://fedoraproject.org/wiki/CI/Tests
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

Hi!

We've had some more discussions regarding this topic within the CI
team. Here's a brief summary of the recommendation we agreed upon:

Tests may be written in different ways, but are enabled in a
standard way directly in the package git repository as defined
by the Standard Test Interface. [1]

Test code can be stored directly in the package git repository
(recommended as default) or fetched from another location hosted
in the Fedora infrastructure such as the Tests Namespace. [2] The
decision where the test code is stored is up to the package maintainer.

[1] https://fedoraproject.org/wiki/CI/Standard_Test_Interface
[2] https://fedoraproject.org/wiki/CI/Share_Test_Code

psss...

On 1 March 2018 at 18:06, Honza Horak <hho...@redhat.com> wrote:
> On 03/01/2018 04:00 PM, Petr Šplíchal wrote:
>>>>
>>>> To sum up what I've heard so far from the developer side:
>>>>
>>>> * I would like to enable tests for my component (yes, I want)
>>>> * I will take care of them (really, I see the benefit in CI)
>>>> * I want to easily collaborate on tests with qe (direct commits)
>>>> * I don't want to give qe commit access to the rpms dist git
>>>
>>> This is quite likely the crux of the problem.
>>>
>>> Personally, I'm perfectly happy to give write access to any repo
>>> to people who have shown that they know what they are doing.
>>> We have pull requests in dist-git pagure now, and I think this is
>>> a better approach:
>>> 1. ask QE contributors to submit PRs
>>> 2. if enough cooperation happens and trust is established, give
>>>privileges to write to the repo directly, possibly with an agreement
>>>that this specific person should only touch tests, and not the
>>>packaging.
>>>
>>> I think it's perfectly fine to never get to point 2.: many many
>>> upstream projects require a review from a second person, or sometimes
>>> even two reviews before a PR is merged, which means that one _never_
>>> merges their own PR, and another contributor is always involved. We
>>> usually don't do this in dist-git, but I'm quite sure that requiring
>>> reviews for dist-git changes would raise quality and catch many silly
>>> mistakes. Either way, it's nowadays possible to cooperate using a
>>> single repo without fully trusting the other person, frictionlessly.
>>
>>
>> Good point. However, if there is a qe/devel team which prefers to
>> collaborate on tests in a separate repo because this is the most
>> efficient way they found so far, why should we stop them and try
>> to enforce a less efficient workflow? (Which they can workaround
>> by a different git repo.)
>>
>>>> * I want to efficiently maintain tests long-term
>>>> * I want to have just a single place for my tests (no duplication)
>>>> * I don't want to backport new tests to old branches (enable once)
>>>> * I want to easily enable testing for all supported branches
>>>
>>> Those four items depend strongly on the package. My thinking is
>>> biased by some specific usecases (mainly systemd), but I'm sure many
>>> other packages are like that: a lot of tests would be for new
>>> functionality, and then the tests _are_ tied to the branch being
>>> tested.
>>>
>>> While I agree that keeping tests separate avoids a bit of effort
>>> required to update multiple branches, this effort isn't very big. If
>>> the tests indeed apply cleanly to all branches, then it's just a
>>> matter of doing 'git merge' or 'git cherry-pick' once per branch.
>>>
>>>> * I want to keep rpms dist git clean (just metadata & patches)
>>>
>>> Meh.
>>>
>>>> * I want to run all available relevant tests (not to list them)
>>>> * I want to execute set of tests based on a tag (e.g. Tier1)
>>>
>>> Those two sound like stuff that should be solved in the tooling,
>>> whatever is used to run tests.
>>>
>>>> * I want an easy way to clone tests (fedpkg clone tests/pkg)
>>>
>>> Tests alone make no sense, you need something to test, and
>>> cloning one repo is easier than cloning two repos, so there's no
>>> advantage to the split here.
>>
>>
>> But "fedpkg clone tests" is easier than cloning from a "random"
>> git repository where I was forced to save my tests because I was
>> not allowed to save them in Fedora tests namespace.
>>
>>>> I believe the tests namespace resolves them all.
>>>
>>&g

Re: New "tests" namespace to share test code

[Petr Šplíchal]

>> To sum up what I've heard so far from the developer side:
>>
>> * I would like to enable tests for my component (yes, I want)
>> * I will take care of them (really, I see the benefit in CI)
>> * I want to easily collaborate on tests with qe (direct commits)
>> * I don't want to give qe commit access to the rpms dist git
> This is quite likely the crux of the problem.
>
> Personally, I'm perfectly happy to give write access to any repo
> to people who have shown that they know what they are doing.
> We have pull requests in dist-git pagure now, and I think this is
> a better approach:
> 1. ask QE contributors to submit PRs
> 2. if enough cooperation happens and trust is established, give
>   privileges to write to the repo directly, possibly with an agreement
>   that this specific person should only touch tests, and not the
>   packaging.
>
> I think it's perfectly fine to never get to point 2.: many many
> upstream projects require a review from a second person, or sometimes
> even two reviews before a PR is merged, which means that one _never_
> merges their own PR, and another contributor is always involved. We
> usually don't do this in dist-git, but I'm quite sure that requiring
> reviews for dist-git changes would raise quality and catch many silly
> mistakes. Either way, it's nowadays possible to cooperate using a
> single repo without fully trusting the other person, frictionlessly.

Good point. However, if there is a qe/devel team which prefers to
collaborate on tests in a separate repo because this is the most
efficient way they found so far, why should we stop them and try
to enforce a less efficient workflow? (Which they can workaround
by a different git repo.)

>> * I want to efficiently maintain tests long-term
>> * I want to have just a single place for my tests (no duplication)
>> * I don't want to backport new tests to old branches (enable once)
>> * I want to easily enable testing for all supported branches
> Those four items depend strongly on the package. My thinking is
> biased by some specific usecases (mainly systemd), but I'm sure many
> other packages are like that: a lot of tests would be for new
> functionality, and then the tests _are_ tied to the branch being
> tested.
>
> While I agree that keeping tests separate avoids a bit of effort
> required to update multiple branches, this effort isn't very big. If
> the tests indeed apply cleanly to all branches, then it's just a
> matter of doing 'git merge' or 'git cherry-pick' once per branch.
>
>> * I want to keep rpms dist git clean (just metadata & patches)
> Meh.
>
>> * I want to run all available relevant tests (not to list them)
>> * I want to execute set of tests based on a tag (e.g. Tier1)
> Those two sound like stuff that should be solved in the tooling,
> whatever is used to run tests.
>
>> * I want an easy way to clone tests (fedpkg clone tests/pkg)
> Tests alone make no sense, you need something to test, and
> cloning one repo is easier than cloning two repos, so there's no
> advantage to the split here.

But "fedpkg clone tests" is easier than cloning from a "random"
git repository where I was forced to save my tests because I was
not allowed to save them in Fedora tests namespace.

>> I believe the tests namespace resolves them all.
> None of those arguments convince me that separate repos for tests are
> a good default. Sure, there are specific packages where this will make
> sense, or specific packagers with idiosyncratic workflows, but I'd
> rather "start small", with the simple configuration, and only move
> specific packages to the more complicated setup once that proves to
> be required.

Why default? Test namespace should be an option. Not the default.
Storing tests directly in dist git is and will be possible.
Anybody who finds this as a better way can do so. But why
enforcing this approach to all?

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

> > > My worries are basically that this mechanism is hiding the
> > > tests from the package maintainers. It splits the concerns
> > > between people maintaining the artifacts and people
> > > maintaining the tests, which is exactly what the original
> > > plan to bring CI was *not* about.
> >
> > I do not see a split here but rather potential for enhanced or
> > extended collaboration. Repositories in the tests namespace
> > can be set up with more open access than rpms git. In this way
> > devel can invite qe for direct collaboration on tests.
>
> I don't see how that's not possible with tests being in
> dist-git.

What I've heard from devels is that they often don't feel like
giving direct commit access to the rpms dist git to QE as spec
files and packaging in general is a big responsiblity. But they
are quite OK with directly collaborating with QE on test coverage.

> If the tests are defined in dist-git and adding new tests means
> committing these changes to dist-git's yaml file, how does that
> solve the concern about "noise" in the git commits?

Not necessarily. You could use advanced metadata to select desired
tests only once. The very first proof of concept of using Flexible
Metadata Format for testing selinux quite nicely illustrates this:

https://src.fedoraproject.org/tests/selinux/pull-request/1
https://github.com/psss/fmf

Instead of listing individual tests you could say: Run all tests
for this selinux component + all Tier1 selinux integration tests:

fmf --key test --filter 'component:libselinux | tags:Tier1'

For tests covering multiple components the benefit of the tests
namespace approach is unquestionable. However, I see benefit of
the approach for single components as well (and developers seem
to often feel the same): That is not having to cherry-pick/merge
test coverage between supported branches. So even for these cases
there is the benefit of sharing the test code.

I believe that extending the integration testing coverage is one
the most essential goals of Fedora CI. This is the crucial piece
of the puzzle how to make an Always Ready Operating System: It is
to ensure that packages play nicely together. So I guess there
will be quite many tests which will benefit from this approach.

Do we really want/need to assess every new request for a tests
repo if it is good-enough for the test namespace? Will there be
a committee to decide whether a particular set of tests has a
potential to be used by other components in the future or not? :)

To sum up what I've heard so far from the developer side:

* I would like to enable tests for my component (yes, I want)
* I will take care of them (really, I see the benefit in CI)
* I want to easily collaborate on tests with qe (direct commits)
* I don't want to give qe commit access to the rpms dist git
* I want to efficiently maintain tests long-term
* I want to have just a single place for my tests (no duplication)
* I want to keep rpms dist git clean (just metadata & patches)
* I want to easily enable testing for all supported branches
* I don't want to backport new tests to old branches (enable once)
* I want to run all available relevant tests (not to list them)
* I want to execute set of tests based on a tag (e.g. Tier1)
* I want an easy way to clone tests (fedpkg clone tests/pkg)

I believe the tests namespace resolves them all.

psss...
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

On 22 February 2018 at 15:50, Pierre-Yves Chibon <pin...@pingoured.fr> wrote:
> On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
>> Hi!
>>
>> During the last days there have been concerns raised regarding
>> what is an appropriate content for the tests namespace. [1] My
>> original idea was to enable sharing tests even across branches of
>> the same component, not only for tests to be used by completely
>> different packages. The initial examples might have been a bit
>> misleading in this respect. One of the main points still holds:
>>
>> >  * Tests might follow a different branching pattern than the
>> >dist-git repo, leading to code duplication
>>
>> From the feedback from developers I feel they always keep on mind
>> and care a lot about the maintenance costs. So it perfectly makes
>> sense to me if they want to keep and maintain tests in a separate
>> repo instead of merging/cherry-picking between dist-git branches.
>>
>> When, for a particular package, it is the most efficient way to
>> maintain tests in a separate repo why should we discourage from
>> this approach? There are packages where it makes more sense to
>> store test code directly in dist-git. And it is still an option.
>> But why should we enforce this for all?

Thanks for sharing your thoughts, Pierre.

> My worries are basically that this mechanism is hiding the tests
> from the package maintainers. It splits the concerns between
> people maintaining the artifacts and people maintaining the
> tests, which is exactly what the original plan to bring CI was
> *not* about.

I do not see a split here but rather potential for enhanced or
extended collaboration. Repositories in the tests namespace can be
set up with more open access than rpms git. In this way devel can
invite qe for direct collaboration on tests. That's one of the
reasons why we should try to fix the non-packager access as soon
as possible.

https://pagure.io/fedora-infrastructure/issue/6361

> The idea has always been to bring the tests where the code
> lives, so that both can be worked on as one, to make tests be a
> concern of the package maintainer very much like updating the
> artifact (rpms, image..) is, while getting support from QE for
> them (the tests).

I see the benefits of having code with tests at the very same
place, I find this very useful especially for unit tests in the
project upstream. However, the situation here is a bit different
in two respects:

1) The rpms dist git does not include package code itself but
rather build metadata only (mainly spec files describing how to
build the package) so it makes very good sense to host at this
place testing metadata only (how to test the package). This is
quite nicely consistent approach.

2) As far as I can see, within the Always Ready Operating System
effort we are not looking to cover every and each feature in all
packages but rather ensuring the OS works as a whole. Thus basic
functionality and integration tests which guard the stability of
the packages are more appropriate for testing. These do not need
to be kept so close to the package code.

I believe we don't want to duplicate upstream unit testing here.
We should probably clarify more which test types are appropriate
or expected in the Fedora CI. As part of the docs consolidation I
added a brief section about test types to the main CI portal:

https://fedoraproject.org/wiki/CI#Test_Types

Does it decribe well what we are looking for? Shall we extend it?

> In addition, this is what I fear most:
> The tests for the package are stored elsewhere, we're not sure
> where (the tests namespace, a random git repo on the
> internet...), the packager update package and the tests fail.

I see no hiding of tests from maintainers: Tests are clearly
referenced and discoverable from the rpms dist git as defined by
the Standard Test Interface and devels have access, see:

https://fedoraproject.org/wiki/CI/Share_Test_Code#Examples

Devel decides which tests are good for testing, it does not matter
where the test code lives. However there needs to be trust to the
referenced repo. But this is the same as with any other open
source project.

> What do you think will happen?
> a) the packager will look for $random_place_of_the_internet
> where tests are and spend time trying to fix them.
> b) the packager will turn off the tests because they get in the way
>
> If the packager wants to turn off the tests, they will have to
> go through dist-git to do it, they may very well end up turning
> the tests off anyway, but if the tests are right there, they may
> as well have a quick look at them to see if they can fix them
> quickly before deciding.

It's not random_place_of_the_internet, it's well selected and
trusted source where de

Re: New "tests" namespace to share test code

[Petr Šplíchal]

On 14 February 2018 at 18:18, Adam Williamson
<adamw...@fedoraproject.org> wrote:
> On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
>> Hi!
>>
>> During the last days there have been concerns raised regarding
>> what is an appropriate content for the tests namespace. [1] My
>> original idea was to enable sharing tests even across branches
>> of the same component, not only for tests to be used by
>> completely different packages. The initial examples might have
>> been a bit misleading in this respect. One of the main points
>> still holds:
>>
>> >  * Tests might follow a different branching pattern than the
>> >  dist-git repo, leading to code duplication
>>
>> From the feedback from developers I feel they always keep on
>> mind and care a lot about the maintenance costs. So it
>> perfectly makes sense to me if they want to keep and maintain
>> tests in a separate repo instead of merging/cherry-picking
>> between dist-git branches.
>>
>> When, for a particular package, it is the most efficient way to
>> maintain tests in a separate repo why should we discourage from
>> this approach? There are packages where it makes more sense to
>> store test code directly in dist-git. And it is still an
>> option.  But why should we enforce this for all?
>>
>> Please share your thoughts and real-life examples. For those
>> who are not familiar with the topic there is a new wiki page
>> with a summary of the Share Test Code approach [2].
>
> I don't have any problem with the concept *in itself*; in fact I
> know of another reason to have something like it. That is
> 'generic' tests, tests we want to run on all packages, or at
> least a very large set of packages - like the tests currently
> running in Taskotron, which are generally run on all packages
> (rpmgrill, rpmdeplint...) or a large subset (the Python tests).

Yes, sharing tests useful for multiple packages makes sense in
general. Test code for generic tests can be stored basically
anywhere but not in specific components. However I see some
additional benefits of having a dedicated tests namespace:

* Source of inspiration and real-life examples (at one place)
* Potential for workflow automation (tests CI, fedpkg --tests)
* Easy cherry-picking/backporting upstream/downstream tests

The difference I see when comparing these shared tests to generic
tests mentioned above like rpmdeplint is that you don't want to
reference these from the individual package repos (e.g. using the
Standard Test Interface), but just run them on all packages.

If the expectation is to run tests only on a subset of rpms then a
question comes: How to store the list of relevant components or
criteria to decide which packages are to be tested or not. Here
the Flexible Metadata Format might come handy in the future:

https://github.com/psss/fmf

> What I did think was maybe a concern is that we should figure
> out in advance the squishy human consequences of different
> technical approaches.
>
> Basically this boiled down to "who is responsible for these
> 'shared' tests, and who gets to decide which 'shared' tests
> can/should block packages?"
>
> Looking at the proposal, I think it actually has at least
> workable initial explicit/implicit answers to this, if I'm
> reading it correctly.  Anyone can create a shared test
> repository (and is therefore "responsible" for maintaining it),
> but the definition of which tests are run on which packages
> remains in the package repositories. Thus the package
> maintainer(s) retain the ultimate choice over which tests are
> run against their packages (and thus can pick which shared tests
> are run, and disable shared tests if they are no longer properly
> testing their package).

Yes. This is exactly true. The maintainer of the package repo has
always the last word on what is to be executed for qualification.
It can be done by referencing only suitable tests from the shared
repo or selecting an appropriate branch or even a specific commit
to ensure that future test development does not break stable
gating (and only manually update reference after review of shared
test updates) or even completely disable tests by removing the
reference.

> The question 'who decides which tests block which packages' is
> left a bit up in the air, but in fact no more so than it already
> was for package-specific tests...

Right. Do we want to change this? Specify this more strictly?
Like exactly defining requirements which an Always Ready Operating
System has to meet? And then mapping these requirements to the
test coverage? (Here again FMF mentioned above might come handy.)

psss...

> --
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
> http://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: New "tests" namespace to share test code

[Petr Šplíchal]

Hi!

During the last days there have been concerns raised regarding
what is an appropriate content for the tests namespace. [1] My
original idea was to enable sharing tests even across branches of
the same component, not only for tests to be used by completely
different packages. The initial examples might have been a bit
misleading in this respect. One of the main points still holds:

>  * Tests might follow a different branching pattern than the
>dist-git repo, leading to code duplication

From the feedback from developers I feel they always keep on mind
and care a lot about the maintenance costs. So it perfectly makes
sense to me if they want to keep and maintain tests in a separate
repo instead of merging/cherry-picking between dist-git branches.

When, for a particular package, it is the most efficient way to
maintain tests in a separate repo why should we discourage from
this approach? There are packages where it makes more sense to
store test code directly in dist-git. And it is still an option.
But why should we enforce this for all?

Please share your thoughts and real-life examples. For those who
are not familiar with the topic there is a new wiki page with a
summary of the Share Test Code approach [2].

psss...

[1] https://pagure.io/fedora-infrastructure/issue/6695
[2] https://fedoraproject.org/wiki/CI/Share_Test_Code

On 7 December 2017 at 10:38, Petr Splichal  wrote:
> Hi!
>
> While working on adding CI tests [0] using the Standard Test
> Interface a need arose to have a shared git repository where tests
> could be stored:
>
>  * A large number of test files makes a dist-git repository more
>difficult to maintain
>
>  * Tests might follow a different branching pattern than the
>dist-git repo, leading to code duplication
>
>  * Shared maintenance for tests sometimes benefits from different
>access levels than the release dist-git repository
>
> The plan is to create a new “tests” namespace in Fedora git/pagure
> dedicated to storing the shared test code. To enable execution of
> these tests by the CI pipeline, tests.yml file in dist-git will be
> used to link the tests in the standard way as defined by the
> Standard Test Interface [1].
>
> This approach should help to efficiently maintain tests & minimize
> test code duplication. Using a dedicated git repo for test code
> also means to keep dist-git more as a place for storing metadata
> only: Build metadata (spec file = how to build the package) and
> test metadata (tests.yml = how to test the package) rather than
> mixing spec files with test code itself.
>
> Please note that this does not mean that all tests should now go
> into this new namespace. You can still link tests directly from
> upstream (like GitHub) or any other source. Also, for unit tests
> it makes more sense to be kept directly with the project source
> and executed there. Shared tests namespace in Fedora will be
> suitable especially for functional and integration testing which
> should help to continuously ensure the OS works as a whole.
>
> For more detailed motivation and real-life examples see the Share
> Test Code proposal on the Fedora CI list [2]. If you have any
> questions or comments feel free to share them here or in the
> pagure issue requesting the new namespace:
>
> https://pagure.io/fedora-infrastructure/issue/6478
>
> Thanks.
>
> psss...
>
> [0] https://fedoraproject.org/wiki/CI
> [1] https://fedoraproject.org/wiki/Changes/InvokingTests
> [2] 
> https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Flexible Metadata Format

[Petr Šplíchal]

Hi,

simple proof of concept is ready for experimenting:

https://github.com/psss/fmf
http://fmf.readthedocs.io/

Looking for the first impressions & feedback. Thanks.

psss...

On 8 January 2018 at 15:49, Petr Splichal  wrote:

> Hi!
>
> In order to keep test execution efficient when number of test
> cases grows, it is crucial to maintain corresponding metadata,
> which define some aspects of how the test coverage is executed.
> For example limiting environment combinations where the test is
> relevant or selecting a subset of important test cases for quick
> verification of essential features when testing a security update.
>
> Within the BaseOS QE team we were thinking (for a long time) about
> an efficient metadata solution which would cover our use cases and
> would be open source. Recently we've been involved in the Upstream
> First initiative which increased the need for an open metadata
> solution which would enable us to more easily share test code
> between Red Hat Enterprise Linux and Fedora.
>
> We've put together a draft solution which covers some of the most
> important stories we've gathered so far. It does not cover all use
> cases and it is not complete. In this early stage we would like to
> invite others who might have similar use cases to gather your
> feedback, share your experience or even join the project:
>
> https://fedoraproject.org/wiki/Flexible_Metadata_Format
>
> The page lists some of our core user stories as well as a couple of
> real-life examples to demonstrate proposed features of the format.
> Can you see similar user stories in your team? Is this something
> that could be useful for you as well? Do you know of a different
> solution for these use cases? Any other relevant ideas?
>
> To illustrate where we could be heading: In the ideal future there
> could be just a single test case for a particular feature stored
> in public with a single set of metadata attached close to the test
> code and together used for testing in both upstream and downstream
> without need to duplicate the test code (maintain both copies).
>
> This proposal does not suggest in any way to replace tests.yml [1]
> files defined by the Standard Test Interface. The new format could
> serve as an extension for selecting the right tests to be executed
> (e.g. filtering tests by tag instead of listing them manually).
>
> Looking forward to your feedback!
>
> psss...
>
> [1] https://fedoraproject.org/wiki/CI/Tests
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Attempting to contact unresponsive maintainers - gabbayo and psss

[Petr Šplíchal]

Hi!

2016-06-02 13:35 GMT+02:00 David Kutalek :
>
> Hi,
>
> wrt psss - former email: pspli...@redhat.com:
>
> Afaik:
>  - he would like to continue with did (with other e-mail address)

Yes, I'm taking did as my personal hobby project and will continue
with the development. Ondřej Pták offered help with the Fedora
packaging stuff so I will shared the necessary rights with him.

>  - not sure about status-report (should be obsoleted by did?)

This can be safely obsoleted. In fact, the package has not been
released at all. I've closed the duplicated review request bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1213739

>  - python-nitrate should be transferred to Leoš Pol 
>and Martin Frodl  as a co-maintainer

Yes, Leoš & Martin accepted to take care of python-nitrate and
they should have all access rights to build packages. This has
been already smoke-tested with the latest release.

I forgot to update my email address in FAS. This should be fixed
now. Thanks for the reminder. Hope everything is now clear.

psss...

> Petr, Leoš, Martin, please can you comment?
>
> Thanks,
> David Kutalek
>
>  Forwarded Message 
> Subject:Attempting to contact unresponsive maintainers - gabbayo and 
> psss
> Date:   Wed, 1 Jun 2016 15:50:09 -0600
> From:   Kevin Fenzi 
> Reply-To:   Development discussions related to Fedora
> 
> Organization:   Scrye, INC
> To: devel@lists.fedoraproject.org
>
>
>
> Greetings, we've been told that the email addresses
> for these package maintainers are no longer valid.  I'm starting the
> unresponsive maintainer policy to find out if they are still interested
> in maintaining their packages (and if so, have them update their email
> addresses in FAS).  If they're not interested in maintaining or we
> can't locate them I'll have FESCo orphan the packages so that others
> can take them over.
>
> If you have a way to contact these maintainers, please let them
> know that we'd appreciate knowing what to do with their packages.
> Thanks!
>
> gabbayo - former email: ogab...@redhat.com
>
> Point of contact:
>
>  rpms/hsakmt -- AMD's HSA thunk library ( master f24 f23 f22 epel7 )
>
> Co-maintainer:
>
>  rpms/glew -- The OpenGL Extension Wrangler Library ( master f24 f23
>  f22 )
>  rpms/pixman -- Pixel manipulation library ( master f24 f23
>  f22 )
>
> psss - former email: pspli...@redhat.com
>
> Point of contact:
>
>  rpms/did -- What did you do last week, month, year? ( master f24
>f23 f22 epel7 el6 )
>  rpms/python-nitrate -- Python API for the Nitrate test case management
>system ( master f24 f23 f22 epel7 el6)
>  rpms/status-report -- Generate status report stats for selected
>date range ( master f24 f23 f22 epel7 el6 )
>
>
>
>
> --
> David Kutalek,
> Senior Quality Engineer, Red Hat, Inc.
> CDK QE / BaseOS QE Apps
--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org