Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
Oooh, that would be perfect! Tomas On 6/8/20 10:57 AM, Florian Festi wrote: May be https://github.com/rpm-software-management/rpm/pull/1256 does the trick. Comments welcome! Florian On 6/5/20 4:39 PM, Igor Raits wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote: Hi, I think it would be useful to have a standard way of disabling the running of tests during RPM build (in the %check section of a spec file). I see a lot of packages already having %bcond's or other macro definitions to archieve this, but each package has their own way, there's no real standard. Thus you have to first look into the spec, locate the appropriate %bcond or macro name and only then you can disable the tests. I would like to propose two approaches: (a) Add a *SHOULD* rule to the guidelines that specifies what is the preferred way to conditionalize the tests. (b) Or, if that's too strong, mention in the guidelines the common methods that are being used (e.g. %bcond tests and %bcond check) so that new packagers have something to use. What do you think? I'd like to have this finally be implemented in https://github.com/rpm-software-management/rpm/issues/316. That way it would be simply rpmbuild --nocheck or define %_without_check 1 which would skip %check section entirely. For now, all Rust crates just have `%bcond_without check` so using `-- without check` works just fine there. Since this would be more generic thing to the RPM ecosystem, adding rpm-ecosystem@ to the copy. Tomas ___ packaging mailing list -- packag...@lists.fedoraproject.org To unsubscribe send an email to packaging-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org - -- Igor Raits -BEGIN PGP SIGNATURE- iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS 5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2 ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8 zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z 7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM= =JCw+ -END PGP SIGNATURE- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
On Fri, Jun 5, 2020 at 5:48 PM Nicolas Mailhot via devel
wrote:
> Some language ecosystems have very low quality unit tests ...
I'd discourage the "some test suites are bad, so let's disable them
all '' attitude.
Some very small and very stable testsuites may still be beneficial to
be run every time.
Though the "small" and "stable" again relies on the maintainer. (But
what does not, anyway ...)
On Fri, Jun 5, 2020 at 4:39 PM Miro Hrončok wrote:
> However rpmbuild itself doesn't support "CheckRequires", so the bcond often
> looks like this:
> %if %{with tests}
> BuildRequires: python3-pytest
> %endif
I fully agree. If the testsuite is not required to be run, sometimes
dozens of packages which are only test dependencies are not required
to be in the buildroot - thus they shouldn't be there.
---
Here's a little insight to huge MariaDB and MySQL packages:
MariaDB & MySQL have about 5000 tests in their testsuites nowadays
from which we run more than 4000.
Many of them scratch the whole DB and re-deploy it again, which *is*
time consuming.
While the build of the DB generally takes hours, the testsuite may
take ten times longer.
Historically on some slow-ass architectures, one build took about 2
days (looking at you 32-bit ARM)
Now if you add randomly failing tests - some because of the testsuite,
some because of the build machine (not enough ports available, or a
sudden heavy load triggered the test timeout, ...),
there's a huge need for decreasing the build time.
Currently, I implemented a macro which holds the "last tested version" value.
Once the maintainer (well, me), goes through the full testsuite for
all architectures, he can bump the macro value (to match the current
DB release version), and from then on, all later re-builds will run
only a very basic "sanity" testsuite which is short and stable, while
still check if the DB works at all.
Until a new release of the package is packed and the "last tested
version" value becomes lower than the current version and the full
testsuite needs to be run again.
This really helps anyone doing rebuilds and PRs.
Of course, that's not all. There are also implemented macros for "do
not run testsuite at all"; "run the full testsuite even though only
sanity check is required"; "if the testsuite will be run, ignore the
results (= always pass)", and so on.
Some may say such a huge testsuite shouldn't be run at the build time,
but well, this is just the upstream set of tests.
We have several other test frameworks with a lot of tests which are
triggered on each build. (e.g. installability tests, Red Hat internal
security regression tests, ... ) And the whole test suite again to
assure it passes in a real environment outside the buildroot.
--
Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat
--
On Mon, Jun 8, 2020 at 10:58 AM Florian Festi wrote:
>
> May be https://github.com/rpm-software-management/rpm/pull/1256 does the
> trick. Comments welcome!
>
> Florian
>
> On 6/5/20 4:39 PM, Igor Raits wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA512
> >
> > On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote:
> >> Hi,
> >> I think it would be useful to have a standard way of disabling the
> >> running of tests during RPM build (in the %check section of a spec
> >> file).
> >>
> >> I see a lot of packages already having %bcond's or other macro
> >> definitions to archieve this, but each package has their own way,
> >> there's no real standard. Thus you have to first look into the spec,
> >> locate the appropriate %bcond or macro name and only then you can
> >> disable the tests.
> >>
> >> I would like to propose two approaches:
> >>
> >> (a) Add a *SHOULD* rule to the guidelines that specifies what is the
> >> preferred way to conditionalize the tests.
> >>
> >> (b) Or, if that's too strong, mention in the guidelines the common
> >> methods that are being used (e.g. %bcond tests and %bcond check) so
> >> that
> >> new packagers have something to use.
> >>
> >> What do you think?
> >
> > I'd like to have this finally be implemented in
> > https://github.com/rpm-software-management/rpm/issues/316. That way it
> > would be simply rpmbuild --nocheck or define %_without_check 1 which
> > would skip %check section entirely.
> >
> > For now, all Rust crates just have `%bcond_without check` so using `--
> > without check` works just fine there.
> >
> > Since this would be more generic thing to the RPM ecosystem, adding
> > rpm-ecosystem@ to the copy.
> >
> >> Tomas
> >> ___
> >> packaging mailing list -- packag...@lists.fedoraproject.org
> >> To unsubscribe send an email to
> >> packaging-le...@lists.fedoraproject.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines:
> >> https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >>
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
May be https://github.com/rpm-software-management/rpm/pull/1256 does the trick. Comments welcome! Florian On 6/5/20 4:39 PM, Igor Raits wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote: >> Hi, >> I think it would be useful to have a standard way of disabling the >> running of tests during RPM build (in the %check section of a spec >> file). >> >> I see a lot of packages already having %bcond's or other macro >> definitions to archieve this, but each package has their own way, >> there's no real standard. Thus you have to first look into the spec, >> locate the appropriate %bcond or macro name and only then you can >> disable the tests. >> >> I would like to propose two approaches: >> >> (a) Add a *SHOULD* rule to the guidelines that specifies what is the >> preferred way to conditionalize the tests. >> >> (b) Or, if that's too strong, mention in the guidelines the common >> methods that are being used (e.g. %bcond tests and %bcond check) so >> that >> new packagers have something to use. >> >> What do you think? > > I'd like to have this finally be implemented in > https://github.com/rpm-software-management/rpm/issues/316. That way it > would be simply rpmbuild --nocheck or define %_without_check 1 which > would skip %check section entirely. > > For now, all Rust crates just have `%bcond_without check` so using `-- > without check` works just fine there. > > Since this would be more generic thing to the RPM ecosystem, adding > rpm-ecosystem@ to the copy. > >> Tomas >> ___ >> packaging mailing list -- packag...@lists.fedoraproject.org >> To unsubscribe send an email to >> packaging-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org > - -- > Igor Raits > -BEGIN PGP SIGNATURE- > > iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc > Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs > yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT > aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS > 5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2 > ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8 > zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq > FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z > 7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT > LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf > bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N > gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM= > =JCw+ > -END PGP SIGNATURE- > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
On 05. 06. 20 16:45, Tomas Orsava wrote: On 6/5/20 4:39 PM, Richard W.M. Jones wrote: On Fri, Jun 05, 2020 at 10:28:39AM -0400, Paul Wouters wrote: Or just a new option to rpmbuild that skips %check ? It exists already: rpmbuild --nocheck. It's not wired into the rest of the stack - eg. you cannot start a Koji build with checks disabled. IMHO that's a good thing, although when we first started doing the RISC-V bootstrap we initially and briefly used this option to disable the tests, for convenience of getting packages built. It might be a good thing for regular builds, but it's sorely missed in scratch builds. Generally, the ability to flip bconds in scratchbuilds would be a huge help over uploading the entire SRPM. Something like: $ fedpkg build --scratch --without tests --without optimizations -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
On 6/5/20 4:39 PM, Richard W.M. Jones wrote: On Fri, Jun 05, 2020 at 10:28:39AM -0400, Paul Wouters wrote: Or just a new option to rpmbuild that skips %check ? It exists already: rpmbuild --nocheck. It's not wired into the rest of the stack - eg. you cannot start a Koji build with checks disabled. IMHO that's a good thing, although when we first started doing the RISC-V bootstrap we initially and briefly used this option to disable the tests, for convenience of getting packages built. It might be a good thing for regular builds, but it's sorely missed in scratch builds. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
On Fri, Jun 05, 2020 at 10:28:39AM -0400, Paul Wouters wrote: > Or just a new option to rpmbuild that skips %check ? It exists already: rpmbuild --nocheck. It's not wired into the rest of the stack - eg. you cannot start a Koji build with checks disabled. IMHO that's a good thing, although when we first started doing the RISC-V bootstrap we initially and briefly used this option to disable the tests, for convenience of getting packages built. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Fri, 2020-06-05 at 16:10 +0200, Tomas Orsava wrote: > Hi, > I think it would be useful to have a standard way of disabling the > running of tests during RPM build (in the %check section of a spec > file). > > I see a lot of packages already having %bcond's or other macro > definitions to archieve this, but each package has their own way, > there's no real standard. Thus you have to first look into the spec, > locate the appropriate %bcond or macro name and only then you can > disable the tests. > > I would like to propose two approaches: > > (a) Add a *SHOULD* rule to the guidelines that specifies what is the > preferred way to conditionalize the tests. > > (b) Or, if that's too strong, mention in the guidelines the common > methods that are being used (e.g. %bcond tests and %bcond check) so > that > new packagers have something to use. > > What do you think? I'd like to have this finally be implemented in https://github.com/rpm-software-management/rpm/issues/316. That way it would be simply rpmbuild --nocheck or define %_without_check 1 which would skip %check section entirely. For now, all Rust crates just have `%bcond_without check` so using `-- without check` works just fine there. Since this would be more generic thing to the RPM ecosystem, adding rpm-ecosystem@ to the copy. > Tomas > ___ > packaging mailing list -- packag...@lists.fedoraproject.org > To unsubscribe send an email to > packaging-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org - -- Igor Raits -BEGIN PGP SIGNATURE- iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7aWQwACgkQEV1auJxc Hh6dyBAAmbJSCU0wtuET7vuXVVIeg7BeosaQF25/VoMwTSYGH3h36S9Gci9BRBgs yuque1uGnBaUQ74fsxBIMgGzapd73TvEY1M8PNnzHF3Miz0i0FgVhwnw3S9jvrTT aGqln2rE3L5jH0alII6pNOIqA67yPlYfb5+JtRazeO0KTarZuGOdemJsp6ONEKQS 5doQid6yrQvaUj90Xl2VpRY6goXx5FOQLDPb9DlaWlQDvBcVBJz5oaJ/VyxqCnC2 ObyLjMB9AXq+pBiot/50QDLTUCxKOkro1siBPxfswNCjpwRy6vDp6dyczHyQkhJ8 zFAHJQPWAr870WU3FMO/FirTv9yAqY6Je8jB+3EdxjzNuyBMTOT6Iq6r8Su/yxeq FcvDvUhlJ0OtWM8PfiIkaKpiSB9rzpuuM5hagPYqznLbqu5AeuTqAKojSyLbkK7Z 7fS+qABernfYqAVOlq7DkTaETh/0sAuIxhtwWXbbhz7vFPpbnsPdnyfUUGzFoIdT LBFnMOBQF0q4woTAhQRHez+VEH4ndiqZQGdYL8AJ9FtKeMZwwWmvl/r3ki/Hr5Yf bqETizKe4XBu5DxPRNN3+0RSi+TIXX11VeHtxIWeuGGErgdqq4EZkBfnmZlTb3/N gA8/3DUl9B4XRzGjnzq0AahOfIW1wNObh4pzlmoGN2jrG5odKPM= =JCw+ -END PGP SIGNATURE- ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Let's standardize the way to disable tests during RPM build?
Or just a new option to rpmbuild that skips %check ? Sent from my iPhone > On Jun 5, 2020, at 10:11, Tomas Orsava wrote: > > Hi, > I think it would be useful to have a standard way of disabling the running of > tests during RPM build (in the %check section of a spec file). > > I see a lot of packages already having %bcond's or other macro definitions to > archieve this, but each package has their own way, there's no real standard. > Thus you have to first look into the spec, locate the appropriate %bcond or > macro name and only then you can disable the tests. > > I would like to propose two approaches: > > (a) Add a *SHOULD* rule to the guidelines that specifies what is the > preferred way to conditionalize the tests. > > (b) Or, if that's too strong, mention in the guidelines the common methods > that are being used (e.g. %bcond tests and %bcond check) so that new > packagers have something to use. > > What do you think? > Tomas > ___ > packaging mailing list -- packag...@lists.fedoraproject.org > To unsubscribe send an email to packaging-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
