bug#43843: git-annex is not Reproducible
On Tue, 13 Oct 2020 at 15:06, Ludovic Courtès wrote: > You can always build with ‘--cores=1’, which will ensure both the build > and test will run sequentially. Yes but in this case, "guix challenge" would still complain, isn't it?
bug#43843: git-annex is not Reproducible
zimoun skribis: > On Mon, 12 Oct 2020 at 12:34, Ricardo Wurmus wrote: > >> > Does the parallel build save a lot of time and CPU? If yes, maybe we >> > could to provide a transformation for the expert, something like >> > "haskell-build-system-with-parellel-build" which tweaks >> > "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT? >> >> We shouldn’t compromise reproducibility for parallel builds. Ideally we >> would figure out what exactly causes the differences and fix that >> instead of disabling parallel builds, but if that turns out to be too >> difficult I think we should just revert this until we have a good fix. >> >> Perhaps something can be done by fixing the order of files somewhere. > > My proposal is for the expert and not at the CLI level. Something > like "(funky-name ghc-foo)" which returns a new package with > PARALLEL-BUILD? turned to #t. And the default should be #f. WDYT? You can always build with ‘--cores=1’, which will ensure both the build and test will run sequentially. Ludo’.
bug#43843: git-annex is not Reproducible
On Mon, 12 Oct 2020 at 12:34, Ricardo Wurmus wrote: > > Does the parallel build save a lot of time and CPU? If yes, maybe we > > could to provide a transformation for the expert, something like > > "haskell-build-system-with-parellel-build" which tweaks > > "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT? > > We shouldn’t compromise reproducibility for parallel builds. Ideally we > would figure out what exactly causes the differences and fix that > instead of disabling parallel builds, but if that turns out to be too > difficult I think we should just revert this until we have a good fix. > > Perhaps something can be done by fixing the order of files somewhere. My proposal is for the expert and not at the CLI level. Something like "(funky-name ghc-foo)" which returns a new package with PARALLEL-BUILD? turned to #t. And the default should be #f. WDYT? Well, I am going to try to rebuild the packages with #f and see if it fixes the ~300 unreproducible packages. Cheers, simon
bug#43843: git-annex is not Reproducible
zimoun writes: > On Sat, 10 Oct 2020 at 11:34, Ricardo Wurmus wrote: > >> > --8<---cut here---start->8--- >> > 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit >> > commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 >> > Author: Ricardo Wurmus >> > Date: Tue Jun 16 22:25:48 2020 +0200 >> > >> > build-system/haskell: Support parallel builds. >> > >> > * guix/build-system/haskell.scm (haskell-build): Add keyword >> > PARALLEL-BUILD? >> > and pass it on to the builder. >> > * guix/build/haskell-build-system.scm (build): Accept keyword >> > PARALLEL-BUILD? >> > and pass the number of parallel jobs to GHC. >> > >> > guix/build-system/haskell.scm | 2 ++ >> > guix/build/haskell-build-system.scm | 7 +-- >> > 2 files changed, 7 insertions(+), 2 deletions(-) >> > bisect run success >> > --8<---cut here---end--->8--- >> >> Shall we revert it? Previously the build system also supported parallel >> builds, but didn’t seem to pass the right arguments to actually make it >> happen. > > What do the Haskellers think about that? Let me answer as an ex-Haskeller :) > Does the parallel build save a lot of time and CPU? If yes, maybe we > could to provide a transformation for the expert, something like > "haskell-build-system-with-parellel-build" which tweaks > "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT? We shouldn’t compromise reproducibility for parallel builds. Ideally we would figure out what exactly causes the differences and fix that instead of disabling parallel builds, but if that turns out to be too difficult I think we should just revert this until we have a good fix. Perhaps something can be done by fixing the order of files somewhere. -- Ricardo
bug#43843: git-annex is not Reproducible
On Sat, 10 Oct 2020 at 11:34, Ricardo Wurmus wrote: > > --8<---cut here---start->8--- > > 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit > > commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 > > Author: Ricardo Wurmus > > Date: Tue Jun 16 22:25:48 2020 +0200 > > > > build-system/haskell: Support parallel builds. > > > > * guix/build-system/haskell.scm (haskell-build): Add keyword > > PARALLEL-BUILD? > > and pass it on to the builder. > > * guix/build/haskell-build-system.scm (build): Accept keyword > > PARALLEL-BUILD? > > and pass the number of parallel jobs to GHC. > > > > guix/build-system/haskell.scm | 2 ++ > > guix/build/haskell-build-system.scm | 7 +-- > > 2 files changed, 7 insertions(+), 2 deletions(-) > > bisect run success > > --8<---cut here---end--->8--- > > Shall we revert it? Previously the build system also supported parallel > builds, but didn’t seem to pass the right arguments to actually make it > happen. What do the Haskellers think about that? Does the parallel build save a lot of time and CPU? If yes, maybe we could to provide a transformation for the expert, something like "haskell-build-system-with-parellel-build" which tweaks "PARALLEL-BUILD?", similarly to the recent "no tests". WDYT? All the best, simon
bug#43843: git-annex is not Reproducible
zimoun writes: > Dear, > > On Thu, 08 Oct 2020 at 09:17, Efraim Flashner wrote: > >>> $ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 -- >>> build --no-grafts --check git-annex >>> guix build: error: derivation >>> `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may >>> not be deterministic: output >>> `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs >>> >>> However, going back one more to 35e24fab73 (gnu: git-annex: Update to >>> 8.20200810, 2020-08-11), the check was successful: >>> >>> $ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 -- >>> build --no-grafts --check git-annex >>> [...] >>> /gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810 > >> In between those two commits is also the change to the haskell build >> system update, where we use the dynamic libraries. The best bet is to >> make sure the dependencies are built reproducibly. > > Yep, it is what “git bisect” says. > > --8<---cut here---start->8--- > 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit > commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 > Author: Ricardo Wurmus > Date: Tue Jun 16 22:25:48 2020 +0200 > > build-system/haskell: Support parallel builds. > > * guix/build-system/haskell.scm (haskell-build): Add keyword > PARALLEL-BUILD? > and pass it on to the builder. > * guix/build/haskell-build-system.scm (build): Accept keyword > PARALLEL-BUILD? > and pass the number of parallel jobs to GHC. > > guix/build-system/haskell.scm | 2 ++ > guix/build/haskell-build-system.scm | 7 +-- > 2 files changed, 7 insertions(+), 2 deletions(-) > bisect run success > --8<---cut here---end--->8--- Shall we revert it? Previously the build system also supported parallel builds, but didn’t seem to pass the right arguments to actually make it happen. -- Ricardo
bug#43843: git-annex is not Reproducible
Dear, On Thu, 08 Oct 2020 at 09:17, Efraim Flashner wrote: >> $ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 -- >> build --no-grafts --check git-annex >> guix build: error: derivation >> `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may >> not be deterministic: output >> `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs >> >> However, going back one more to 35e24fab73 (gnu: git-annex: Update to >> 8.20200810, 2020-08-11), the check was successful: >> >> $ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 -- >> build --no-grafts --check git-annex >> [...] >> /gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810 > In between those two commits is also the change to the haskell build > system update, where we use the dynamic libraries. The best bet is to > make sure the dependencies are built reproducibly. Yep, it is what “git bisect” says. --8<---cut here---start->8--- 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 is the first bad commit commit 67cb9fa2357026ee42ec5bb0923ec4dc4a43abe2 Author: Ricardo Wurmus Date: Tue Jun 16 22:25:48 2020 +0200 build-system/haskell: Support parallel builds. * guix/build-system/haskell.scm (haskell-build): Add keyword PARALLEL-BUILD? and pass it on to the builder. * guix/build/haskell-build-system.scm (build): Accept keyword PARALLEL-BUILD? and pass the number of parallel jobs to GHC. guix/build-system/haskell.scm | 2 ++ guix/build/haskell-build-system.scm | 7 +-- 2 files changed, 7 insertions(+), 2 deletions(-) bisect run success --8<---cut here---end--->8--- I did: git bisect start e13fefbc 35e24fab git bisest run ./check.sh where check.sh reads: --8<---cut here---start->8--- #!/bin/sh PKG=git-annex git log -1 --format='%h %s [%an]' COMMIT=$(git log -1 --format='%H') echo "Start: $COMMIT" guix time-machine \ --commit=$COMMIT \ -- build $PKG\ --no-grafts guix time-machine \ --commit=$COMMIT \ -- build $PKG\ --no-grafts --check -K status=$? echo "End: [$status] $COMMIT" exit $status --8<---cut here---end--->8--- And I do not know if it is related, but these packages using the ’haskell-build-system’ are not reproducible neither (304/~640): agda cabal-install cpphs darcs dhall elm-compiler ghc-abstract-deque ghc-abstract-par ghc-adjunctions ghc-aeson ghc-alex ghc-asn1-encoding ghc-asn1-types ghc-assoc ghc-atomic-primops ghc-attoparsec ghc-auto-update ghc-aws ghc-base-compat ghc-basement ghc-base-unicode-symbols ghc-bifunctors ghc-bitarray ghc-blaze-html ghc-blaze-markup ghc-bloomfilter ghc-bytes ghc-bytestring-handle ghc-c2hs ghc-cairo ghc-cborg ghc-cereal ghc-cgi ghc-charset ghc-chart ghc-chasingbottoms ghc-cheapskate ghc-chunked-data ghc-clientsession ghc-cmdargs ghc-colour ghc-comonad ghc-concurrent-extra ghc-conduit ghc-conduit-algorithms ghc-conduit-extra ghc-configurator ghc-constraints ghc-convertible ghc-crypto-api ghc-crypto-api-tests ghc-crypto-cipher-tests ghc-crypto-cipher-types ghc-cryptohash ghc-cryptonite ghc-crypto-random ghc-curl ghc-data-accessor ghc-dbus ghc-descriptive ghc-diagrams-core ghc-diagrams-lib ghc-diagrams-solve ghc-diff ghc-digest ghc-doctest ghc-edisonapi ghc-edisoncore ghc-edit-distance ghc-either ghc-errors ghc-esqueleto ghc-exactprint ghc-extra ghc-fast-logger ghc-feed ghc-fgl ghc-filemanip ghc-filepath-bytestring ghc-fingertree ghc-foldl ghc-foundation ghc-free ghc-fsnotify ghc-generic-deriving ghc-generics-sop ghc-genvalidity-property ghc-getopt-generics ghc-gluraw ghc-glut ghc-gnuplot ghc-graphviz ghc-gtk2hs-buildtools ghc-hackage-security ghc-haddock-api ghc-haddock-library ghc-happstack-server ghc-happy ghc-hashtables ghc-haskeline ghc-haskell-lexer ghc-haskell-src ghc-haskell-src-exts ghc-haskell-src-exts-util ghc-hedgehog ghc-highlighting-kate ghc-hindent ghc-hmatrix ghc-hmatrix-gsl ghc-hmatrix-gsl-stats ghc-hmatrix-special ghc-hourglass ghc-hpack ghc-hslogger ghc-hslua ghc-hsopenssl ghc-hspec ghc-hspec-contrib ghc-hspec-core ghc-hspec-meta ghc-hsyaml ghc-http ghc-http2 ghc-http-client ghc-http-date ghc-http-types ghc-hunit ghc-hxt ghc-hxt-regex-xmlschema ghc-hxt-unicode ghcid ghc-inline-c ghc-inspection-testing ghc-integer-logarithms ghc-interpolate ghc-intervalmap ghc-intervals ghc-io-streams ghc-jira-wiki-markup ghc-json ghc-juicypixels ghc-kan-extensions ghc-language-c ghc-language-glsl ghc-language-javascript ghc-lens ghc-lens-family-core ghc-libmpd ghc-lib-parser ghc-libxml ghc-libyaml ghc-lifted-base ghc-linear ghc-listlike ghc-llvm-hs ghc-llvm-hs-pure ghc-logging-facade ghc-math-functions
bug#43843: git-annex is not Reproducible
On Wed, Oct 07, 2020 at 11:04:13PM -0400, Kyle Meyer wrote:
> On 10/07/20 12:35:51 +0200, zimoun wrote:
> > Dear,
> >
> > Using Guix 04a459a, the package ’git-annex’ is not reproducible:
> >
> > guix build git-annex
> > guix build git-annex --no-grafts --check -K
> >
> > return:
> >
> > --8<---cut here---start->8---
> > guix build: error: derivation
> > `/gnu/store/70qkv5hi5yq20m1h9jfm17m5lsc235mk-git-annex-8.20200908.drv' may
> > not be deterministic: output
> > `/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908' differs
> > from
> > ‘/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check’
> > --8<---cut here---end--->8---
> >
> > then:
> >
> > --8<---cut here---start->8---
> > $ diff -r --no-dereference
> > /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908{,-check}
> > Binary files
> > /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908/bin/git-annex
> > and
> > /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check/bin/git-annex
> > differ
> > --8<---cut here---end--->8---
> >
> > Is it expected? I have not tried ’diffoscope’ yet to investigate.
>
> Thanks for noticing. I see the same on my end (and also with the
> git-annex update [1] that I submitted before I saw this message).
>
> I suspect this is a relatively new issue, as I've run `build --check'
> successfully on git-annex in the past. The git-annex definition prior
> to the one above was introduced with e13fefbcff (gnu: git-annex: Create
> fake static output, 2020-08-17). On that commit, the check also failed
> for me:
>
> $ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 --
> build --no-grafts --check git-annex
> guix build: error: derivation
> `/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may
> not be deterministic: output
> `/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs
>
> However, going back one more to 35e24fab73 (gnu: git-annex: Update to
> 8.20200810, 2020-08-11), the check was successful:
>
> $ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 --
> build --no-grafts --check git-annex
> [...]
> /gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810
>
>
> [1]: bug#43854: <20201007204946.22878-1-k...@kyleam.com>
>
In between those two commits is also the change to the haskell build
system update, where we use the dynamic libraries. The best bet is to
make sure the dependencies are built reproducibly.
--
Efraim Flashner אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
signature.asc
Description: PGP signature
bug#43843: git-annex is not Reproducible
On 10/07/20 12:35:51 +0200, zimoun wrote:
> Dear,
>
> Using Guix 04a459a, the package ’git-annex’ is not reproducible:
>
> guix build git-annex
> guix build git-annex --no-grafts --check -K
>
> return:
>
> --8<---cut here---start->8---
> guix build: error: derivation
> `/gnu/store/70qkv5hi5yq20m1h9jfm17m5lsc235mk-git-annex-8.20200908.drv' may
> not be deterministic: output
> `/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908' differs
> from ‘/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check’
> --8<---cut here---end--->8---
>
> then:
>
> --8<---cut here---start->8---
> $ diff -r --no-dereference
> /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908{,-check}
> Binary files
> /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908/bin/git-annex
> and
> /gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check/bin/git-annex
> differ
> --8<---cut here---end--->8---
>
> Is it expected? I have not tried ’diffoscope’ yet to investigate.
Thanks for noticing. I see the same on my end (and also with the
git-annex update [1] that I submitted before I saw this message).
I suspect this is a relatively new issue, as I've run `build --check'
successfully on git-annex in the past. The git-annex definition prior
to the one above was introduced with e13fefbcff (gnu: git-annex: Create
fake static output, 2020-08-17). On that commit, the check also failed
for me:
$ guix time-machine --commit=e13fefbcffa50144d9301d67b61928ac3c964fb3 --
build --no-grafts --check git-annex
guix build: error: derivation
`/gnu/store/5illriaaijqrhl17aymzgiw6wzaq666s-git-annex-8.20200810.drv' may not
be deterministic: output
`/gnu/store/32h0xbqdwkm22naqba10fk9124cdgw9q-git-annex-8.20200810' differs
However, going back one more to 35e24fab73 (gnu: git-annex: Update to
8.20200810, 2020-08-11), the check was successful:
$ guix time-machine --commit=35e24fab73bc450c6e0d3b9a5bbdfa88d2c425f4 --
build --no-grafts --check git-annex
[...]
/gnu/store/m79ri0g2s3d8pk52ysjkdvallk2zqbwb-git-annex-8.20200810
[1]: bug#43854: <20201007204946.22878-1-k...@kyleam.com>
bug#43843: git-annex is not Reproducible
Dear,
Using Guix 04a459a, the package ’git-annex’ is not reproducible:
guix build git-annex
guix build git-annex --no-grafts --check -K
return:
--8<---cut here---start->8---
guix build: error: derivation
`/gnu/store/70qkv5hi5yq20m1h9jfm17m5lsc235mk-git-annex-8.20200908.drv' may not
be deterministic: output
`/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908' differs from
‘/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check’
--8<---cut here---end--->8---
then:
--8<---cut here---start->8---
$ diff -r --no-dereference
/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908{,-check}
Binary files
/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908/bin/git-annex
and
/gnu/store/cbrwrlwx6kg7l5q6ifl7q08icm17704g-git-annex-8.20200908-check/bin/git-annex
differ
--8<---cut here---end--->8---
Is it expected? I have not tried ’diffoscope’ yet to investigate.
Note that:
speaks about GCC patch… I do not know if it is relevant.
All the best,
simon
