Let me know if I'm talking nonsense, but I believe that we are building both stages for each architecture and flavour. Do we need to build two stages everywhere? What stops us from building a single stage? And if anything, what can we change to get into a situation where we can?
Quite better than reusing build incrementally, is not building at all. On Mon, Feb 22, 2021 at 10:09 AM Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org> wrote: > Incremental CI can cut multiple hours to < mere minutes, especially with > the test suite being embarrassingly parallel. There simply no way > optimizations to the compiler independent from sharing a cache between CI > runs can get anywhere close to that return on investment. > > I rather agree with this. I don’t think there is much low-hanging fruit > on compile times, aside from coercion-zapping which we are working on > anyway. If we got a 10% reduction in compile time we’d be over the moon, > but our users would barely notice. > > > > To get truly substantial improvements (a factor of 2 or 10) I think we > need to do less compiling – hence incremental CI. > > > Simon > > > > *From:* ghc-devs <ghc-devs-boun...@haskell.org> *On Behalf Of *John > Ericson > *Sent:* 22 February 2021 05:53 > *To:* ghc-devs <ghc-devs@haskell.org> > *Subject:* Re: On CI > > > > I'm not opposed to some effort going into this, but I would strongly > opposite putting all our effort there. Incremental CI can cut multiple > hours to < mere minutes, especially with the test suite being > embarrassingly parallel. There simply no way optimizations to the compiler > independent from sharing a cache between CI runs can get anywhere close to > that return on investment. > > (FWIW, I'm also skeptical that the people complaining about GHC > performance know what's hurting them most. For example, after > non-incrementality, the next slowest thing is linking, which is...not done > by GHC! But all that is a separate conversation.) > > John > > On 2/19/21 2:42 PM, Richard Eisenberg wrote: > > There are some good ideas here, but I want to throw out another one: put > all our effort into reducing compile times. There is a loud plea to do this > on Discourse > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdiscourse.haskell.org%2Ft%2Fcall-for-ideas-forming-a-technical-agenda%2F1901%2F24&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691120329%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1CV0MEVUZpbAbmKAWTIiqLgjft7IbN%2BCSnvB3W3iX%2FU%3D&reserved=0>, > and it would both solve these CI problems and also help everyone else. > > > > This isn't to say to stop exploring the ideas here. But since time is > mostly fixed, tackling compilation times in general may be the best way out > of this. Ben's survey of other projects (thanks!) shows that we're way, way > behind in how long our CI takes to run. > > > > Richard > > > > On Feb 19, 2021, at 7:20 AM, Sebastian Graf <sgraf1...@gmail.com> wrote: > > > > Recompilation avoidance > > > > I think in order to cache more in CI, we first have to invest some time in > fixing recompilation avoidance in our bootstrapped build system. > > > > I just tested on a hadrian perf ticky build: Adding one line of *comment* > in the compiler causes > > - a (pretty slow, yet negligible) rebuild of the stage1 compiler > - 2 minutes of RTS rebuilding (Why do we have to rebuild the RTS? It > doesn't depend in any way on the change I made) > - apparent full rebuild the libraries > - apparent full rebuild of the stage2 compiler > > That took 17 minutes, a full build takes ~45minutes. So there definitely > is some caching going on, but not nearly as much as there could be. > > I know there have been great and boring efforts on compiler determinism in > the past, but either it's not good enough or our build system needs fixing. > > I think a good first step to assert would be to make sure that the hash of > the stage1 compiler executable doesn't change if I only change a comment. > > I'm aware there probably is stuff going on, like embedding configure dates > in interface files and executables, that would need to go, but if possible > this would be a huge improvement. > > > > On the other hand, we can simply tack on a [skip ci] to the commit > message, as I did for > https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fmerge_requests%2F4975&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691130329%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bgT0LeZXjF%2BMklzctvZL6WaVpaddN7%2FSpojcEXGXv7Q%3D&reserved=0>. > Variants like [skip tests] or [frontend] could help to identify which tests > to run by default. > > > > Lean > > > > I had a chat with a colleague about how they do CI for Lean. Apparently, > CI turnaround time including tests is generally 25 minutes (~15 minutes for > the build) for a complete pipeline, testing 6 different OSes and > configurations in parallel: > https://github.com/leanprover/lean4/actions/workflows/ci.yml > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fleanprover%2Flean4%2Factions%2Fworkflows%2Fci.yml&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691140326%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9MEWPlRhO2xZK2iu5OqzXS9RZqc9pKNJcGDv7Nj3hyA%3D&reserved=0> > > They utilise ccache to cache the clang-based C++-backend, so that they > only have to re-run the front- and middle-end. In effect, they take > advantage of the fact that the "function" clang, in contrast to the > "function" stage1 compiler, stays the same. > > It's hard to achieve that for GHC, where a complete compiler pipeline > comes as one big, fused "function": An external tool can never be certain > that a change to Parser.y could not affect the CodeGen phase. > > > > Inspired by Lean, the following is a bit inconcrete and imaginary, but > maybe we could make it so that compiler phases "sign" parts of the > interface file with the binary hash of the respective subcomponents of the > phase? > > E.g., if all the object files that influence CodeGen (that will later be > linked into the stage1 compiler) result in a hash of 0xdeadbeef before and > after the change to Parser.y, we know we can stop recompiling Data.List > with the stage1 compiler when we see that the IR passed to CodeGen didn't > change, because the last compile did CodeGen with a stage1 compiler with > the same hash 0xdeadbeef. The 0xdeadbeef hash is a proxy for saying "the > function CodeGen stayed the same", so we can reuse its cached outputs. > > Of course, that is utopic without a tool that does the "taint analysis" of > which modules in GHC influence CodeGen. Probably just including all the > transitive dependencies of GHC.CmmToAsm suffices, but probably that's too > crude already. For another example, a change to GHC.Utils.Unique would > probably entail a full rebuild of the compiler because it basically affects > all compiler phases. > > There are probably parallels with recompilation avoidance in a language > with staged meta-programming. > > > > Am Fr., 19. Feb. 2021 um 11:42 Uhr schrieb Josef Svenningsson via ghc-devs > <ghc-devs@haskell.org>: > > Doing "optimistic caching" like you suggest sounds very promising. A way > to regain more robustness would be as follows. > > If the build fails while building the libraries or the stage2 compiler, > this might be a false negative due to the optimistic caching. Therefore, > evict the "optimistic caches" and restart building the libraries. That way > we can validate that the build failure was a true build failure and not > just due to the aggressive caching scheme. > > > > Just my 2p > > > > Josef > > > ------------------------------ > > *From:* ghc-devs <ghc-devs-boun...@haskell.org> on behalf of Simon Peyton > Jones via ghc-devs <ghc-devs@haskell.org> > *Sent:* Friday, February 19, 2021 8:57 AM > *To:* John Ericson <john.ericson@obsidian.systems>; ghc-devs < > ghc-devs@haskell.org> > *Subject:* RE: On CI > > > > 1. Building and testing happen together. When tests failure > spuriously, we also have to rebuild GHC in addition to re-running the > tests. That's pure waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fissues%2F13897&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691140326%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Nm6vfgGLLlJpiGa8XKxI6kNkBetp8ZZLPZS8hF%2BydrM%3D&reserved=0> > tracks this more or less. > > I don’t get this. We have to build GHC before we can test it, don’t we? > > 2 . We don't cache between jobs. > > This is, I think, the big one. We endlessly build the exact same > binaries. > > There is a problem, though. If we make **any** change in GHC, even a > trivial refactoring, its binary will change slightly. So now any caching > build system will assume that anything built by that GHC must be rebuilt – > we can’t use the cached version. That includes all the libraries and the > stage2 compiler. So caching can save all the preliminaries (building the > initial Cabal, and large chunk of stage1, since they are built with the > same bootstrap compiler) but after that we are dead. > > I don’t know any robust way out of this. That small change in the source > code of GHC might be trivial refactoring, or it might introduce a critical > mis-compilation which we really want to see in its build products. > > However, for smoke-testing MRs, on every architecture, we could perhaps > cut corners. (Leaving Marge to do full diligence.) For example, we could > declare that if we have the result of compiling library module X.hs with > the stage1 GHC in the last full commit in master, then we can re-use that > build product rather than compiling X.hs with the MR’s slightly modified > stage1 GHC. That **might** be wrong; but it’s usually right. > > Anyway, there are big wins to be had here. > > Simon > > > > > > > > *From:* ghc-devs <ghc-devs-boun...@haskell.org> *On Behalf Of *John > Ericson > *Sent:* 19 February 2021 03:19 > *To:* ghc-devs <ghc-devs@haskell.org> > *Subject:* Re: On CI > > > > I am also wary of us to deferring checking whole platforms and what not. I > think that's just kicking the can down the road, and will result in more > variance and uncertainty. It might be alright for those authoring PRs, but > it will make Ben's job keeping the system running even more grueling. > > Before getting into these complex trade-offs, I think we should focus on > the cornerstone issue that CI isn't incremental. > > 1. Building and testing happen together. When tests failure > spuriously, we also have to rebuild GHC in addition to re-running the > tests. That's pure waste. > https://gitlab.haskell.org/ghc/ghc/-/issues/13897 > > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fissues%2F13897&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691150320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=KlQGW1amK%2BtlRTGl4cDgMyl%2Bfz17fuUAHFNAaNXbzZI%3D&reserved=0> > tracks this more or less. > 2. We don't cache between jobs. Shake and Make do not enforce > dependency soundness, nor cache-correctness when the build plan itself > changes, and this had made this hard/impossible to do safely. Naively this > only helps with stage 1 and not stage 2, but if we have separate stage 1 > and --freeze1 stage 2 builds, both can be incremental. Yes, this is also > lossy, but I only see it leading to false failures not false acceptances > (if we can also test the stage 1 one), so I consider it safe. MRs that only > work with a slow full build because ABI can so indicate. > > The second, main part is quite hard to tackle, but I strongly believe > incrementality is what we need most, and what we should remain focused on. > > John > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691160313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uE1IOblLTYJ2j3H2vkFKgQyVZs5sehXd1Tl70X0kUqE%3D&reserved=0> > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691160313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uE1IOblLTYJ2j3H2vkFKgQyVZs5sehXd1Tl70X0kUqE%3D&reserved=0> > > > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs@haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691170308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Yrob9grqAWOxZnFXcM%2BZ60VNsrhIejcmwkSIR3Wq0gA%3D&reserved=0> > > _______________________________________________ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >
_______________________________________________ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
