Re: [Nix-dev] Incremental recompilation

[stewart mackenzie]

Isn't the simplest solution this:

declare an output

`outputs = ["out" "cache"];`

now the word cache is hardcoded to create a directory
/tmp/${derivation-name}-cache/ this name doesn't change.
Now, in your derivation you can set the build tool (via env var if it
supports that) to use this cache folder as a place to store the build
artifacts.
You may then copy the final build product from `$cache` to `$out` at
some in the `installPhase`.

The last thing about `cache` is that it is _not_ deleted after the
derivation builds.

kr
/sjm
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Incremental recompilation

[Ericson, John]

Nicolas is correct. One complication however one is that one has to be
careful when files contain relative paths. The cleanest treatment of the
recursive hashing would normalize any nix store path within the source
location to hash of just that subtree, but that breaks the relative paths.
Note that its not always possible to resolve all relative paths while
hashing either, as they may form cycles.

Back to Volkhov's original point, I think that compilers should work like
Nix itself -- explicit graph of computation all data is content-addressed
and all function application is cached on the hashes of the arguments (and
function itself). Its a lot of work to refactor compilers to work this way
of course, but practicalities aside I think that's the cleanest way to go.

To cache across derivations, let's say, the compiler and Nix should be able
to use the same content-address store---that's half the battle. The other
half is the compiler (also like Nix) needs to map function applications to
their evaluations. As I said above the arguments are already
content-addressed, but the functions need to be given some sort of abstract
names. Perhaps indices prefixed with the derivation path would do, and then
Nix and the compiler can share the same store for their evaluation maps too.

As it turns out, Rust is exploring incremental compilation
https://github.com/rust-lang/rfcs/blob/master/text/1298-incremental-compilation.md
and their plan is, well, kinda a first approximation of the above. I'd be
really interested in trying to put this plan into action with rustc---this
is our best shot given the feasibility of refactoring GCC, LLVM, GHC, etc
on our own.

John


On Sat, Apr 30, 2016 at 8:39 AM, Nicolas Pierron <
nicolas.b.pier...@gmail.com> wrote:

> On Fri, Apr 29, 2016 at 7:56 PM, stewart mackenzie 
> wrote:
> > IMHO, incremental compilation is a strong boon for nix adoption. It is
> the
> > only serious issue I can see which prevents Nix being a Make replacement.
>
> I agree with you on this point, Nix has the potential of been a better
> ccache & distcc in one shot.
>
> Note, this is not the same issue as mention in the Volkhov email.  The
> email suggest caching artifacts of the compilation spawn by Nix and
> independently of the build system, while doing a make replacement
> involves replacing/wrapping existing build system and using Nix for
> caching artifacts of the compilation spawned in user-land.  But, I
> guess the second problem's solution could be a mean for solving the
> first problem.
>
> Also, I admit I thought on how to solve this issue in the first place,
> but the main problem I got is not even building, this is only about
> making a copy of the source directory into the nix store.
> When your project sources are larger than 1 GB (without the VCS), you
> don't want to be reading all the bytes before being able to start any
> compilation.
>
> Thus, none of the approaches proposed before are practical for larger
> projects.
> So, how can we make it such that it will scale?
>
> The only approach I can think of would be to:
>  - Upload each individual file to the Nix store, when they would be needed.
>  - Make a symlink fram to re-build fake source directories.
>  - Use hash-by-content to prevent recompilation.  (when using  "gcc
> -M"  to filter out directory content, you want to make sure you get
> the same set of headers, and only cause a recompilation if some of the
> header changed)
>
> --
> Nicolas Pierron
> http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Incremental recompilation

[Nicolas Pierron]

On Fri, Apr 29, 2016 at 7:56 PM, stewart mackenzie  wrote:
> IMHO, incremental compilation is a strong boon for nix adoption. It is the
> only serious issue I can see which prevents Nix being a Make replacement.

I agree with you on this point, Nix has the potential of been a better
ccache & distcc in one shot.

Note, this is not the same issue as mention in the Volkhov email.  The
email suggest caching artifacts of the compilation spawn by Nix and
independently of the build system, while doing a make replacement
involves replacing/wrapping existing build system and using Nix for
caching artifacts of the compilation spawned in user-land.  But, I
guess the second problem's solution could be a mean for solving the
first problem.

Also, I admit I thought on how to solve this issue in the first place,
but the main problem I got is not even building, this is only about
making a copy of the source directory into the nix store.
When your project sources are larger than 1 GB (without the VCS), you
don't want to be reading all the bytes before being able to start any
compilation.

Thus, none of the approaches proposed before are practical for larger projects.
So, how can we make it such that it will scale?

The only approach I can think of would be to:
 - Upload each individual file to the Nix store, when they would be needed.
 - Make a symlink fram to re-build fake source directories.
 - Use hash-by-content to prevent recompilation.  (when using  "gcc
-M"  to filter out directory content, you want to make sure you get
the same set of headers, and only cause a recompilation if some of the
header changed)

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Incremental recompilation

[stewart mackenzie]

It would be great to deploy the build product without the caches.

Preferably expose the API as part of mkDerivation.
This way I could pass in a 'develop' flag and if true, switch the cache on,
else leave it off. Now I'll be able to deploy without caches.

i.e.: if develop == true then cache = true else "";

IMHO, incremental compilation is a strong boon for nix adoption. It is the
only serious issue I can see which prevents Nix being a Make replacement.
Many projects start with a Make file, right away there is a dependency. Why
not just install Nix instead of Make as your project dependency,
immediately your project gets access to needed deps and do a whole bunch of
really powerful things to your source code.

We use Nix as a Make replacement, and I'm hooked to the point where I'm
able to endure no IR. So, this IR feature will upgrade Nix to a class A
drug, without the hazardous effects, from where I'm sitting.

/sjm
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev