Fedora lacks integration testing (unit testing done during build is not
enough). Taskotron will be able to fill some gaps in the future, so
maintainers will be able to set-up various tasks after their component
is built. But even before this works we can benefit from having the
tests already available (and run them manually if needed).
Hereby, I'd like to get ideas and figure out answers for how and where
to keep the tests. A similar discussion already took place before, which
I'd like to continue in:
https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html
And some short discussion already took place here as well:
https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-October/000570.html
Some high level requirements:
* tests will be written by maintainers or broader community, not a
dedicated team
* tests will be easy to run on anybody's computer (but might be
potentially destructive; some secure environment will not be part of tests)
* tests will be run automatically after related components get built
(probably by Taskotron)
Where to keep tests?
a/ in current dist-git for related components (problem with sharing
parts of code, problem where to keep tests related for more components)
b/ in separate git with similar functionality as dist-git (needs new
infrastructure, components are not directly connected with tests, won't
make mess in current dist-git)
c/ in current dist-git but as ordinary components (no new infrastructure
needed but components are not directly connected with tests)
How to deliver tests?
a/ just use them directly from git (we need to keep some metadata for
dependencies anyway)
b/ package them as RPMs (we can keep metadata there; e.g. Taskotron will
run only tests that have "Provides: ci-tests(mariadb)" after mariadb is
built; we also might automate packaging tests to RPMs)
Structure for tests?
a/ similar to what components use (branches for Fedora versions)
b/ only one branch
Test maintainers should be allowed to behave the same as package
maintainers do -- one likes keeping branches the same and uses "%if
%fedora" macros, someone else likes specs clean and rather maintain more
different branches) -- we won't find one structure that would fit all,
so allowing both ways seems better.
Which framework to use?
People have no time to learn new things, so we should let them to write
the tests in any language and just define some conventions how to run them.
Cheers,
Honza
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
