Hi, Ricardo Wurmus <[email protected]> writes:
> [...] > > It seems to me that this is a more general problem affecting all of our > Haskell packages. The configure phase that you didn’t paste should show > that modules are provided by slightly different packages. > > The haskell-build-system suffers from non-determinism. It might just be > limited to the package database files that are generated by ghc-pkg > (where readdir is used and the result isn’t sorted). This is exactly the problem. I’ve attached a patch that should fix this. Unfortunately, I kind of rediscovered this from scratch, so for the sake of completeness, here’s all the details of what I found. I don’t recall the exact chain of dependencies that led me from git-annex to ghc-exceptions, but the latter is a nice and small example of the general problem. The ghc-exceptions package depends on ghc-stm. Even though Hydra and Berlin built the exact same ghc-stm (save for the “package.cache” file), the way that ghc-exceptions depends on it differs between build servers. Hydra uses “stm-[version]” in the ghc-exceptions package database, and Berlin uses “stm-[version]-[hash]”. For more complicated packages, these differences get sufficiently jumbled up and GHC gets grumpy because we are trying to use two “different” versions of ghc-stm. In this case, they both have the same name, ID, ABI, and are even bit-for-bit identical (again, except for “package.cache”), but GHC doesn’t care because it is worried about the different reference (with the hash and without it). Another concrete example is ghc-aeson. It needs (among other things) ghc-uuid-types and ghc-hashable. These both need ghc-text. Right now, I cannot build ghc-aeson because it has conflicting requirements for ghc-text. It looks like I got a ghc-uuid-types substitute from Hydra that doesn’t include the hash, and a ghc-hashable substitute from Berlin that does. Again, ghc-text is bit-for-bit identical between the two up to “package.cache”. What’s great is that both build servers are internally consistent, so they never run into trouble. It’s just us poor users that suffer. :p Obviously the “package.cache” file looks pretty guilty, but it is also pretty inscrutable. I pulled code out of GHC to dump a textual representation of it, and found out that the only differences are in the order of packages. Everything else is the same. I modified 'ghc-pkg' to write a sorted binary cache, and it seems to solve all the issues mentioned above. That is, I built ghc-text, ghc-hashable, and ghc-uuid-types on two different computers, and got bit-for-bit identical results for all of them. (I also built ghc-text on both computers without the patch, and saw the same differing “package.cache” problem that I observed between Hydra and Berlin.) I think this patch will solve the git-annex problem, as well improve Haskell reproducibility. Thoughts? -- Tim
>From bb29ee8ccc656b86039127b31fd8b79533927053 Mon Sep 17 00:00:00 2001 From: Timothy Sample <[email protected]> Date: Wed, 2 Jan 2019 16:40:48 -0500 Subject: [PATCH] gnu: ghc: Sort packages before writing binary cache. This improves the reproducibility of packages built with the Haskell build system. * gnu/packages/haskell.scm (ghc)[arguments]: Add a phase that patches 'ghc-pkg' so that it sorts packages before generating a binary cache. --- gnu/packages/haskell.scm | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/gnu/packages/haskell.scm b/gnu/packages/haskell.scm index 8d0e2aef6..1a751a5b4 100644 --- a/gnu/packages/haskell.scm +++ b/gnu/packages/haskell.scm @@ -14,7 +14,7 @@ ;;; Copyright © 2017 rsiddharth <[email protected]> ;;; Copyright © 2017, 2018 Tobias Geerinckx-Rice <[email protected]> ;;; Copyright © 2018 Tonton <[email protected]> -;;; Copyright © 2018 Timothy Sample <[email protected]> +;;; Copyright © 2018, 2019 Timothy Sample <[email protected]> ;;; Copyright © 2018 Arun Isaac <[email protected]> ;;; ;;; This file is part of GNU Guix. @@ -508,6 +508,18 @@ interactive environment for the functional language Haskell.") (assoc-ref inputs "ghc-testsuite") "--strip-components=1") #t)) + ;; This phase patches the 'ghc-pkg' command so that it sorts + ;; the list of packages in the binary cache it generates. + (add-after 'unpack 'patch-ghc-pkg + (lambda _ + (substitute* "utils/ghc-pkg/Main.hs" + (("import Data.List") + (string-append "import Data.List\n" + "import Data.Ord (comparing)")) + (("pkgsCabalFormat = packages db") + (string-append "pkgsCabalFormat = sortBy" + " (comparing (display . installedUnitId))" + " (packages db)"))))) (add-after 'unpack-testsuite 'fix-shell-wrappers (lambda _ (substitute* '("driver/ghci/ghc.mk" -- 2.20.1
