Re: Turning off tests leads to a different store item
Hi,
Simon Tournier writes:
> Hi Maxim,
>
> On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer
> wrote:
>
>>> I agree. On a side note, one of the issue is the time of some tests.
>>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>>> tests, think it will be good, re-launch “guix build”, another test
>>> failing, repeat. It could nice to be able to cache the result of the
>>> build phase.
>>
>> I usually end up working in the /tmp/guix-build-* failed build of a
>> package for these situations; then I don't need to rebuild the whole
>> thing for each test suite run, and can test changes directly without
>> proper patches while working toward a fix.
>
> Do you know all the command lines equivalent for each phase run by all
> the build systems? ;-)
It depends of the build system, but for gnu-build-system, it's typically
just sourcing environment-variables then running 'make check', I think.
Note that 'live' environment variables are captured in the
'environment-variables' file, which is handy.
> Well, this workflow seems appealing but it never works for me on
> concrete situations. Most of the time, my issue does not come from
> “what to do from command line” but “how to do it inside the Guix
> recipe”.
>
> For example, this test suite adjustment:
>
> (add-before 'install 'disable-network-tests
> (lambda _
> (substitute* "test/runtests.jl"
> (("\"async.jl") "# \"async.jl")
> (("\"client.jl") "# \"client.jl"))
[...]
> (("@testset.*Set-Cookie casing.*" all)
> (string-append all "return\n"
>
> To have the correct replacement rules, I almost never get it right at
> the very first try, and the feedback loop is poor. In some case, the
> replacement is done after 'unpack, so I can just kill the build process
> and check inside /tmp/guix-build-* if the output is the expected one.
I've used 'guix repl' in a failed build before, running the same
commands directly in the tree to observe their effects (e.g.,
substitutions) directly.
--
Thanks,
Maxim
Re: Turning off tests leads to a different store item
Hi Maxim,
On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer wrote:
>> I agree. On a side note, one of the issue is the time of some tests.
>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>> tests, think it will be good, re-launch “guix build”, another test
>> failing, repeat. It could nice to be able to cache the result of the
>> build phase.
>
> I usually end up working in the /tmp/guix-build-* failed build of a
> package for these situations; then I don't need to rebuild the whole
> thing for each test suite run, and can test changes directly without
> proper patches while working toward a fix.
Do you know all the command lines equivalent for each phase run by all
the build systems? ;-)
Well, this workflow seems appealing but it never works for me on
concrete situations. Most of the time, my issue does not come from
“what to do from command line” but “how to do it inside the Guix
recipe”.
For example, this test suite adjustment:
(add-before 'install 'disable-network-tests
(lambda _
(substitute* "test/runtests.jl"
(("\"async.jl") "# \"async.jl")
(("\"client.jl") "# \"client.jl"))
(substitute* "test/aws4.jl"
(("@testset.*HTTP.request with AWS authentication.*" all)
(string-append all "return\n")))
(substitute* "test/insert_layers.jl"
(("@testset.*Inserted final layer runs handler.*" all)
(string-append all "return\n")))
(substitute* "test/multipart.jl"
(("@testset \"Setting of Content-Type.*" all)
(string-append all "return\n"))
(("@testset \"Deprecation of .*" all)
(string-append all "return\n")))
(substitute* "test/websockets.jl"
(("@testset.*External Host.*" all)
(string-append all "return\n")))
(substitute* "test/messages.jl"
(("@testset.*Read methods.*" all)
(string-append all "return\n"))
(("@testset.*Body - .*" all)
(string-append all "return\n"))
(("@testset.*Write to file.*" all)
(string-append all "return\n")))
(substitute* "test/cookies.jl"
(("@testset.*Set-Cookie casing.*" all)
(string-append all "return\n"
To have the correct replacement rules, I almost never get it right at
the very first try, and the feedback loop is poor. In some case, the
replacement is done after 'unpack, so I can just kill the build process
and check inside /tmp/guix-build-* if the output is the expected one.
However, depending on the test suite, sometimes, it errors at the first
failing test and does not report all the failing cases. For example, in
the previous snippet, I could have to build then check, fail and repeat
something like 11 times, adding some try-error attempts for the correct
replacement rule.
Cheers,
simon
Re: Turning off tests leads to a different store item
Hi Simon, Simon Tournier writes: [...] >> I think the lower fruits are in looking at making the test suite of the >> few common offenders more robust (using libfaketime or the likes) to >> prevent (re)occurrences of time bombs in the future. > > I agree. On a side note, one of the issue is the time of some tests. > Sometimes, packaging is frustrating: build takes ages, then you fix some > tests, think it will be good, re-launch “guix build”, another test > failing, repeat. It could nice to be able to cache the result of the > build phase. I usually end up working in the /tmp/guix-build-* failed build of a package for these situations; then I don't need to rebuild the whole thing for each test suite run, and can test changes directly without proper patches while working toward a fix. -- Thanks, Maxim
Re: Turning off tests leads to a different store item
Hi,
On Wed, 08 Nov 2023 at 22:17, Maxim Cournoyer wrote:
> I agree it looks tricky to get it right (and even trickier to prove/test
> for it) :-).
Yeah. I have tried a rough “proof-of-concept” i.e., two derivations:
one which deletes ’check’ phase and the other which deletes ’build’
phase and depends on the former.
Re: Turning off tests leads to a different store item
Simon Tournier
Thu, 02 Nov 2023 18:02:18 +0100
id:86y1fgm6lh@gmail.com
https://lists.gnu.org/archive/html/help-guix/2023-11
https://yhetil.org/guix/86y1fgm6lh@gmail.com
In this simple case, it works. :-)
But to have something robust, IMHO, it would probably mean 1. create
other objects (record) different from and revamp the build
systems. And then 2. rebuild many packages for 3. evaluate the ratio
between the number of packages that works this way vs the number of
packages that fails this way. Oof! That’s a fun project… :-)
> I think the lower fruits are in looking at making the test suite of the
> few common offenders more robust (using libfaketime or the likes) to
> prevent (re)occurrences of time bombs in the future.
I agree. On a side note, one of the issue is the time of some tests.
Sometimes, packaging is frustrating: build takes ages, then you fix some
tests, think it will be good, re-launch “guix build”, another test
failing, repeat. It could nice to be able to cache the result of the
build phase.
Cheers,
simon
Re: Turning off tests leads to a different store item
Hi, Simon Tournier writes: > Hi, > > On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori wrote: > >> There is another way: simply preventing the tests from changing the >> resulting store item. For example, the package could first be built >> without tests and then that build tree could be copied to the build tree >> of the build with tests enabled. > > Somehow, the store would need to keep all the build intermediary > artifacts produced, right? For instance, consider the extreme case > where the build phase produces .o artifact files and the tests for > whatever reasons relies on these temporary artifacts. > > Well, we had a quick chat with Josselin and Andreas about separating > the tests at https://hpc.guix.info/events/2023/workshop/program/ > And my understanding of the rough conclusion we had: it is not easy > and the evil are in all the details. For example autotools: somehow > "make check" is connected in one way or the other to "make" and/or > "make install". Somehow, the complete build tree (with intermediary > artifacts as .o) should keep in the store. > > From a pragmatical point of view, there is packages where the tests > cannot be totally separated from from the temporary build, therefore > the question seems: how do these cases compare to the other regular > cases? What is the ratio? Is the rule about many corner cases > without a clear "regular"? Or are they just few corner cases? One easy-ish way, which would be kind of ugly because coupled to the specific file system capabilities (e.g Btrfs), would be to leverage CoW features and create a Btrfs snapshot at the beginning of the test suite, and reverting to it after it's run. But even that is not fullproof; that'd only protect the build directory, say, not the store location (some check phases are moved after installation). I agree it looks tricky to get it right (and even trickier to prove/test for it) :-). I think the lower fruits are in looking at making the test suite of the few common offenders more robust (using libfaketime or the likes) to prevent (re)occurrences of time bombs in the future. -- Thanks, Maxim
Re: Turning off tests leads to a different store item
Hi, On Wed, 8 Nov 2023 at 20:20, Saku Laesvuori wrote: > There is another way: simply preventing the tests from changing the > resulting store item. For example, the package could first be built > without tests and then that build tree could be copied to the build tree > of the build with tests enabled. Somehow, the store would need to keep all the build intermediary artifacts produced, right? For instance, consider the extreme case where the build phase produces .o artifact files and the tests for whatever reasons relies on these temporary artifacts. Well, we had a quick chat with Josselin and Andreas about separating the tests at https://hpc.guix.info/events/2023/workshop/program/ And my understanding of the rough conclusion we had: it is not easy and the evil are in all the details. For example autotools: somehow "make check" is connected in one way or the other to "make" and/or "make install". Somehow, the complete build tree (with intermediary artifacts as .o) should keep in the store. >From a pragmatical point of view, there is packages where the tests cannot be totally separated from from the temporary build, therefore the question seems: how do these cases compare to the other regular cases? What is the ratio? Is the rule about many corner cases without a clear "regular"? Or are they just few corner cases? Cheers, simon
Re: Turning off tests leads to a different store item
On Wed, Nov 08, 2023 at 10:18:40AM -0800, Vagrant Cascadian wrote: > On 2023-11-08, Felix Lechner via wrote: > > On Wed, Nov 08 2023, Maxim Cournoyer wrote: > >> A source tree doesn't produce a derivation. A derivation is the > >> complete build recipe that captures the source and the package > >> definition, that when built by the daemon produces a store item. > > > > Okay, thanks! Now I'm going to get it right: > > > > The store item that is produced should not change whether build-time > > tests run or not. > > > > It does not make sense (and wastes resources) to rebuild a consuming > > package because build-time tests were enabled or disabled in an input. > > I do not really think people are misunderstanding you, more that your > *should* does not align with reality in a way that guix can depend on; > build-time tests *do* affect the build result in some cases. > > [...] > > The only way to ensure they do not change the build results is even more > resource-intensive; systematically run the build without tests and then > run the build with tests and compare the store items... assuming the > build is reproducible in the first place, which is not yet 100% reliable > either, unfortunately. There is another way: simply preventing the tests from changing the resulting store item. For example, the package could first be built without tests and then that build tree could be copied to the build tree of the build with tests enabled. The result of that build could then just be copied from the testless build, ignoring any changes the test suite has made to the build tree. I'm not confident enough in my understanding of how Guix builds things to say for sure that this specific method would work, but I am quite sure that the general idea is implementable. signature.asc Description: PGP signature
Re: Turning off tests leads to a different store item
On 2023-11-08, Felix Lechner via wrote: > On Wed, Nov 08 2023, Maxim Cournoyer wrote: >> A source tree doesn't produce a derivation. A derivation is the >> complete build recipe that captures the source and the package >> definition, that when built by the daemon produces a store item. > > Okay, thanks! Now I'm going to get it right: > > The store item that is produced should not change whether build-time > tests run or not. > > It does not make sense (and wastes resources) to rebuild a consuming > package because build-time tests were enabled or disabled in an input. > > The historical version of openssl gave rise to this thread. It did not > build anymore because the tests no longer worked with the certificates > shipped in that release (a common problem in TLS libraries). Rebuilding > openssl without running the tests rendered the rebuild useless because > it produced a different store item. That should not happen. > > Does that make more sense? I do not really think people are misunderstanding you, more that your *should* does not align with reality in a way that guix can depend on; build-time tests *do* affect the build result in some cases. You can write tests that can be run outside the build environment, like Debian's autopkgtest, but those may need to actually be different tests and most upstreams do not provide them yet... The only way to ensure they do not change the build results is even more resource-intensive; systematically run the build without tests and then run the build with tests and compare the store items... assuming the build is reproducible in the first place, which is not yet 100% reliable either, unfortunately. This does seem well outside the context of help-guix (drop from CC in replies?) at this point, and should perhaps be moved to guix-devel(added to CC), as it may require significant changes to guix; it is not a simple matter of helping someone figure out how to do something with guix. live well, vagrant signature.asc Description: PGP signature
