Hi, I'm working at Serokell and I was suggested to solve the problem of incremental recompilation generic framework absence as (a part of?) my coursework in this term (university) that would be helpful for our company as well.
Actually, it's not a real problem for most users, but for those who build packages without hydra it might be helpful. It's also a great thing for those who develop with nix and need to build the project frequently (another approaches like stack-nix for haskell are of course available, but pure nix solution has its own advantages and may be preferable). So while researching I discovered that: 1. Nobody has really succeeded in making stable generic framework for this task -- all solutions are ad-hoc and language/build-tool specific 2. Ad-hoc solutions are mostly based on exploiting the nix itself -- writing a package that somehow captures its previous build states and uses them. My proposal is very raw and may appear completely nonviable. I'm writing this to hear any comments about the idea, other approaches and get any help that could be provided. So, here the sketch: 1. Allow adding some extra field (let's call it `cache`) to derivation that would be resemble out fields but with specific target -- storing a folder in nixstore with builds (both succeeded and not) for this derivation. This mechanism should be optional and so some hackery around derivation is supposed to be done. Adding extra flags to store shouldn't be hard too. 2. Write a smart framework in ~stdenv/generic/*~ for specifying behavior on using last caches. Framework isn't supposed to know anything about the state of builds in the folder generated by `cache` (from user point of view), so one should just specify how to "reuse" caches from previous build. 3. All failing and successful builds are going into `cache` folder in nixstore. Some other piece of code (I'm not sure whether it should be in nix itself or in nixpkgs) will, if requested, give those caches to builder and he will reuse it somehow (with generic builder in most cases obviously). Caches will have some kind of versions that would allow us to build a tree from them -- given cache A and builder X it would possibly modify A to state B. So we then put B as a child of B into `cache` folder. So when tree of builds is constructed we can adopt the following build strategy: choose some subset of leaves (based on some heuristics I suppose), then just run builds, if any failed, put its parent into set, and so do it until you reach a root -- the most raw cache. Basically a leaves-first bfs-like traversal. If root fails to build, build the derivation from scratch and put its result into `cache` dir as another root. 4. We can also add some flag permitting to build project from source directly when caches are available. Consider a situation when you're building a project with estimated time 10h and you know that most of the time it can fail because, for example, you applied some patch and not sure if it will work. Then just fixing it after build fail and running build from the previous cache would be a great idea. 5. Derivations in a particular cache tree could be stored by specifying the diff with derivation's parent to save space. I'm not considering the lack of space in nixstore as a problem though (for simplicity). -- Volkhov Mikhail M3#38 ITMO study group 17' Computer Technologies Department _______________________________________________ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev