After much pondering, I've decided the best way to work with language-specific build managers is for them to come up with the end-to-end build plan, and us to build each package in that build plan.
Our current method with Haskell of trying to replicate each package and all the possible ways it can be configured just doesn't scale very well with respect to configuaration options. Things like `if impl(ghcjs):` have long remained unimplemented, and implementing them bakes more and more build-plan resolution machinery into machine until we approach re-implementing cabal-install. [Peti has discussed this in depth in https://github.com/NixOS/cabal2nix/blob/master/doc/03-map-cabal-files-to-nix-without-information-loss.md .] At the same time, or current method isn't great for developing multiple packages simultaneously. The default thing to do is make a package set with each one, but if one doesn't handle the source very carefully, a change to any of them will cause them all to be rebuilt. Furthermore, one needs switch shells per each package---which is a pain in itself---and that prevents the use of neat things like `stack repl` which interprets multiple packages at once for seamless concurrent development. [I discussed part of this in https://github.com/NixOS/nixpkgs/issues/10902 a bit.] `stack --nix` at first glance seems to solve both of these areas, but in fact comes with it's own set of downsides. Yes, since Stack manages the build plan less knowledge is needed on the nix side, but stack also *executes* the build plan. This means that both Nix's superior caching is not utilized, and per-package patches (like foreign dependencies) are unavailable. [See https://github.com/NixOS/nixpkgs/issues/15995.] What I'm proposing should give us the best of both worlds in that language package manager makes most of the decisions, but we still have an opportunity to cache and patch, and development and deployment workflows overlap as much as possible. This methodology could in principle work with any language and its package manager, but I single out Cabal+Haskell because the new Nix-style workflow ( http://cabal.readthedocs.io/en/latest/nix-local-build.html) makes it the most ready. For both deployment and development, the idea is we first run cabal-install to compute a build plan. During this stage it in principle only needs access to .cabal files / the Hackage index and a project config---certainly not any already built binaries like today. Next we instruct cabal-install to decompose the plan into "sub plans" for each dependency/would-be system package---this is the biggest change needed for cabal-install, but one that any system package manager could leverage, not just us. After that, using a reworked cabal2nix, we convert each of those plans to a Nix derivation---this may include turning hackage deps into local deps with their source fetched by nix. Finally, each of those derivations can use `cabal new-install` to install their root package, and the caches/package DBs can be passed downstream and combined just like today. There are two important concepts to note here: First, naively, the use of entire sub plans in each Nix derivation would cause cabal-install to rebuild each dependency for package in the plan that depends that needs it, but since the Nix derivations will also depend on each other, `cabal new-build`'s own caching should kick in so each Nix derivation does in fact correspond to each Haskell package being built. Second, while cabal-install can be instructed to build any package in a plan, the decomposition of build plans is necessary so build plans that differ in ways that don't affect common dependencies still end up sharing nix-executed builds. In both cases, we rely on cabal-install's imitation of our methodology. For development, we leverage the new nix-style build's preexisting notion of a local vs external package. For convenience, it's probably best to let cabal-install devise the build plan impurely (this is analogous to non-restricted mode for Nix expr evaluation), and not copy any local sources to the Nix store. Then, only external packages (and only those which are dependencies of the current goal) should be built by Nix in accordance with the plan above. In a shell with those built, cabal-install can be run again building local packages normally and locally using its more fine-grained per-file hashing and enabling tricks analogous to `stack-repl`. While the default shell derivation will need tweaking if any local packages have special requirements like foreign dependencies, this is better than with `stack --nix` where the shell needs tweaking if *any* dependency has such special requirements. like today. For deployment we don't need to worry about local vs external packages, and on the cabal-install side `cabal new-install` should treat them all the same IIUC. However, we do need to make sure that the build plan resolution can be done in a pure environment: this means making sure all local packages locations are in the Nix store and the Hackage index is pre-downloaded by Nix. After that however no further complications arise. There's a lot of work with nixpkgs and cabal2nix required for all this, but comparatively little upstream for cabal-install. And the end result I believe is hugely worthwhile, solving just about every pain point with our existing infrastructure. John N.B. not quite sure on the division of labor between Cabal and cabal-install with the new system, and erring on the side of cabal-install.
_______________________________________________ nix-dev mailing list [email protected] http://lists.science.uu.nl/mailman/listinfo/nix-dev
