On Sun, Feb 19, 2023 at 3:11 PM Maciej Barć wrote:
>
> What if developer configured an ebuild in a way that it downloads the
> test suite/files/data with USE=test?
>
> IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should
> be true (exists).
This is what I was afraid of with the name; it's not only about a
testsuite being there or not, it's about trusting that:
- There is a test suite.
- It was run.
- It passed, and passing is valuable.
Just existing is not sufficient, because we cannot gain any
information from an src_test that just downloaded but didn't run the
suite. That is not valuable information.
-A
>
> W dniu 19.02.2023 o 18:32, Michał Górny pisze:
> > Signed-off-by: Michał Górny
> > ---
> > glep-.ebuild | 132 +++
> > 1 file changed, 132 insertions(+)
> > create mode 100644 glep-.ebuild
> >
> > diff --git a/glep-.ebuild b/glep-.ebuild
> > new file mode 100644
> > index 000..9ee18ca
> > --- /dev/null
> > +++ b/glep-.ebuild
> > @@ -0,0 +1,132 @@
> > +---
> > +GLEP:
> > +Title: TEST_SUITE_PRESENT variable
> > +Author: Michał Górny
> > +Type: Standards Track
> > +Status: Draft
> > +Version: 1
> > +Created: 2023-02-19
> > +Last-Modified: 2023-02-19
> > +Post-History: 2023-02-19
> > +Content-Type: text/x-rst
> > +---
> > +
> > +
> > +Abstract
> > +
> > +
> > +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> > +the package features a test suite. It can be set either by the ebuild,
> > +the eclass or the default ``src_test`` implementation, and afterwards
> > +included in the package manager logs. This can aid in analyzing
> > +the results of automated package testing.
> > +
> > +
> > +Motivation
> > +==
> > +
> > +The deployment of new Python targets in Gentoo currently involves
> > +testing of a large number of Gentoo packages against the said target.
> > +This is currently done manually for the most part. It can be
> > +particularly time consuming if multiple individuals repeatedly test
> > +the same package only to determine that it remains incompatible with
> > +the new interpreter.
> > +
> > +The Python team wanted to explore the use of automation to aid this
> > +testing. Unfortunately, this faces a major problem: for the vast
> > +of majority of packages, the incompatibilities with new Python versions
> > +do not exhibit during the installation and can only be detected through
> > +running the test suite. The results of automated testing are therefore
> > +only meaningful if the package features a test phase.
> > +
> > +For packages using ``distutils-r1`` eclass, the presence of test suite
> > +can usually be easily determined through grepping for
> > +``distutils_enable_tests`` call or an explicit ``python_test()``
> > +function. Even then, it seems sensible to work towards a more generic
> > +approach to tell whether a package had a test suite or not,
> > +and therefore whether a particular successful automated testing result
> > +means that the package actually passed tests or only confirmed that
> > +the Python files were copied successfully.
> > +
> > +An explicit indication whether a test suite was present can be presented
> > +by the package manager as part of logs, along with the result of running
> > +the test phase. Afterwards, these logs can be used to determine which
> > +packages were actually tested.
> > +
> > +
> > +Specification
> > +=
> > +
> > +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> > +by a ``src_test()`` implementation to indicate whether the package
> > +featured a test suite. It can take three values:
> > +
> > +- ``yes`` indicating that a test suite was run
> > +- ``indeterminate`` indicating that it was not possible to clearly
> > + determine whether the test suite was present or not (this could be
> > + a case e.g. when a generic test command is run and it does not
> > + indicate whether any tests were found)
> > +- ``no`` indicating that no test suite was run
> > +
> > +This variable *should* be set by eclasses defining the ``src_test()``
> > +phase. If the package in question is using ``src_test()`` defined
> > +by an eclass that does not declare it explicitly, the PM must assume
> > +``indeterminate``.
> > +
> > +The variable *may* be set by an ebuild defining the ``src_test()``
> > +phase. If the ebuild does not define it explicitly, the PM must assume
> > +``yes``.
> > +
> > +The default ``src_test()`` implementation as defined by the PMS sets
> > +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> > +target, and to ``no`` if neither of the targets is found.
> > +
> > +
> > +Rationale
> > +=
> > +
> > +The use of ternary flag makes it possible to clearly represent all three
> > +possible outcomes while navigating the defaults defined in the GLEP.
> > +The flag is set in ``src_test()``, so that runtime conditions (such
> > +as the results
