I also remember this discussion.. I think it was in the context of optimising hadrian.
On Wed, 27 Mar 2019, 15:56 Alp Mestanogullari, <a...@well-typed.com> wrote: > > https://gitlab.haskell.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&search=%22ghc+-M%22 > seems to suggest this never made it into a ticket. I searched in the > Hadrian github repo as well, no luck there either. It certainly came up in > various discussions I've had though. > On 27/03/2019 16:05, Andrey Mokhov wrote: > > Hi David, > > > > We had a discussion about this with Neil some time ago, and I think we had > the following list of progressively more complex invariants for different > types of build systems: > > > > · Non-cloud build systems: **all direct inputs must be declared**. > If you miss a direct input dependency then a build may complete > successfully but with an incorrect result. > > > > · Cloud build systems: **all direct inputs and direct outputs must > be declared**. If you miss a direct output then a build may fail because > the cloud will not be able to restore the corresponding output. > > > > · Cloud build systems with shallow (deferred) materialisation of > build artefacts: **all transitive inputs and direct outputs must be > declared**. Let’s say you’d like to download the resulting GHC binary > directly, without materialising any intermediate artefacts. Then you’ll > need to know GHC’s ultimate transitive inputs. > > > > I think for now we are really keen to make Hadrian a cloud build system, > but whether shallow builds are valuable enough is not clear. Maybe not. > Therefore, I’d say we don’t need to track transitive inputs right now. > Furthermore, if we were to track all transitive inputs, we would lose the > desirable early cutoff property, which prevents rebuilding after adding a > comment in a file on which a lot of other files transitively depend on. > > > > Having said that, if we really access a file during compilation, then I > think it is **not** a transitive dependency by definition! Any file which > is accessed during a build rule is a direct dependency. > > > > > GHC is reading *.hi files that are not reported as dependencies by > > > `ghc -M -include-pkg-deps`. This is because they are not direct, but > transitive > > > dependencies! > > > > So, here I’m confused. If we read a file A when compiling a file B, then > it’s by definition a direct dependency. Perhaps we just read too much? > Maybe the solution is to switch to fine-grained `ghc -M` mode, to analyse > import dependencies for a single module instead of doing it transitively, > which I believe was discussed in a ticket some time ago? I can’t find this > ticket, but I think Alp was looking into it at some point. Alp: do you > remember it? > > > > Thank you for all your work on Hadrian! > > > > Cheers, > > Andrey > > > > *From:* David Eichmann [mailto:dav...@well-typed.com > <dav...@well-typed.com>] > *Sent:* 27 March 2019 12:54 > *To:* Neil Mitchell <ndmitch...@gmail.com> <ndmitch...@gmail.com>; Andrey > Mokhov <andrey.mok...@newcastle.ac.uk> <andrey.mok...@newcastle.ac.uk>; > GHC developers <ghc-devs@haskell.org> <ghc-devs@haskell.org> > *Subject:* Hadrian Transitive Dependencies > > > > Hello Shake/Hadrian contributors and the like, > > Recently I've been putting Hadrian's fsatrace linting feature to good use, > tracking down missing dependencies in Hadrian. Ultimately, we want to use > shake's cloud build / shared cache feature and ensure it works across CI > builds. Unfortunately the feature isn't working smoothly with Hadrian: see > #16295 <https://gitlab.haskell.org/ghc/ghc/issues/16295>. This is very > desirable to improve CI build times. It is my understanding that in order > to get caching to work: > > 1. All accessed files must declared with `need` AND > 2. All created files must be declared with `produces` (or be the target of > the build rule) > > Is my understanding correct? Or is there a weaker condition (perhaps only > 2 is necessary)? > > If I'm correct, this amounts to fixing all fsatrace lint errors. See here > <https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901> for a > breakdown of lint errors / missing dependencies. A large portion of these > are Haskell interface files (i.e. *.hi files). Before building a Haskell > object file, dependencies are discovered via `ghc` using the `-M > -include-pkg-deps` options. Unfortunately, shake's fsatrace linting > complains about other *.hi files being accessed! For example when building > `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the following > dependencies from ghc: > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > libraries/mtl/Control/Monad/RWS/Class.hs > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi > > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : > _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi > > And shake complains of the following missing deps: > > _build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static > -hide-all-packages -no-user-package-db '-package-db > _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id > base-4.13.0.0' '-package-id transformers-0.5.5.0' -i > -i_build/stage1/libraries/mtl/build > -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes > -I_build/generated -I_build/stage1/libraries/mtl/build > -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include > > -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include > > -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include > -I_build/generated -optc-I_build/generated -optP-include > -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir > _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances > -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c > libraries/mtl/Control/Monad/RWS/Class.hs -o > _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall > -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat > -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances > -XHaskell2010 -XSafe > -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h > -Wno-deprecated-flags > > Lint checking error - > _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 > values were used but not depended upon: > > Used: _build/HEAD_default/stage0/lib/settings > > Used: _build/HEAD_default/stage0/lib/platformConstants > > Used: _build/HEAD_default/stage0/lib/llvm-targets > > Used: _build/HEAD_default/stage0/lib/llvm-passes > > Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache > > Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi > > Used: > _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi > > GHC is reading *.hi files that are not reported as dependencies by `ghc > -M -include-pkg-deps`. This is because they are not direct, but > *transitive* dependencies! How do we fix these lint errors (again with > the goal of using shakes shared cache feature)? Some ideas: > > * Wildly over approximate dependencies. This may be easier to implement > but cause unneeded recompilation (when a false dependency changes). Either: > * `need` all dependent packages' interface files recursively as well > as transitive dependencies reported by `ghc -M -include-pkg-deps` within > the current package. OR > * OR `need` all transitive dependencies reported by `ghc -M > -include-pkg-deps`. This will likely result in fewer dependencies but > requires a bit more work in recovering dependent packages' dependency > graphs. > * Perhaps transitive dependencies are not important for shared caching to > work. Change shakes linting feature to allow (untracked?) transitive > dependencies to be accessed. > > Feed back would be greatly appreciated. > > David Eichmann > > > > -- > > David Eichmann, Haskell Consultant > > Well-Typed LLP, http://www.well-typed.com > > > > Registered in England & Wales, OC335890 > > 118 Wymering Mansions, Wymering Road, London W9 2NF, England > > -- > Alp Mestanogullari, Haskell Consultant > Well-Typed LLP, https://www.well-typed.com/ > > Registered in England and Wales, OC335890 > 118 Wymering Mansions, Wymering Road, London, W9 2NF, England > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
