Re: [Nix-dev] Introducing Nixpkgs Overlays
+1 from me, we need to plan out the details such as deprecation timeline
(involving dates and docs).
I'll review the PR.
On Tue, Dec 27, 2016 at 8:43 PM, Benno Fünfstück <
benno.fuenfstu...@gmail.com> wrote:
> Hi Nicolas,
>
> I like the idea of Nixpkgs overlays. I have not thought fully about the
> whole proposal (and I need to catch up with how security updates work
> first), but I think there is small mistake when you say:
>
> As part of this modification, I intend to remove the
> `pkgs.overridePackages` function, as this one can literally be
> replaced by the following Nix expression:
>
> ```nix
> let pkgs = import {}; in
> import pkgs.path { overlay = [ (self: super: {
> foo = super.foo.override { enableBar = true; };
> bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
> }) ]; }
> ```
>
>
> The `overridePackages` function is useful when you want to chain multiple
> functions that modify the package set, like
>
> ```nix
> let
> pkgs = import {};
> pkgsA = functionA pkgs
> pkgsB = pkgsA.overridePackages (super: self: {
> });
> ```
>
> In this code example. I do not need to know how functionA is implemented
> internally. It seems to me that, if I instead used overlays for this, it
> would be hard to achieve the same effect: I would have to manually take
> care to re-apply the overrides that `functionA` put in place.
>
> This does not mean that `overridePackages` needs to stay as-is. But
> perhaps it would be possible to implement it using overlays, while still
> providing the same interface to the user?
>
>
> Regards,
>
> Benno Fünfstück
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Introducing Nixpkgs Overlays
Hi Nicolas,
I like the idea of Nixpkgs overlays. I have not thought fully about the
whole proposal (and I need to catch up with how security updates work
first), but I think there is small mistake when you say:
As part of this modification, I intend to remove the
`pkgs.overridePackages` function, as this one can literally be
replaced by the following Nix expression:
```nix
let pkgs = import {}; in
import pkgs.path { overlay = [ (self: super: {
foo = super.foo.override { enableBar = true; };
bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
}) ]; }
```
The `overridePackages` function is useful when you want to chain multiple
functions that modify the package set, like
```nix
let
pkgs = import {};
pkgsA = functionA pkgs
pkgsB = pkgsA.overridePackages (super: self: {
});
```
In this code example. I do not need to know how functionA is implemented
internally. It seems to me that, if I instead used overlays for this, it
would be hard to achieve the same effect: I would have to manually take
care to re-apply the overrides that `functionA` put in place.
This does not mean that `overridePackages` needs to stay as-is. But perhaps
it would be possible to implement it using overlays, while still providing
the same interface to the user?
Regards,
Benno Fünfstück
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Introducing Nixpkgs Overlays
I am using `overridePackages` for a similar purpose already.
(Simplified for ease of presentation.)
One top-level file ties the overlays together
and serves as a replacement for ``.
`mynixpkgs.nix`:
``` nix
{ machine }:
let
pkgs = import { };
machineOverlay =
{ machineA = import ./machineA.nix;
machineB = import ./machineB.nix;
}.${machine} or (throw "unknown machine: ${machine}!")
;
extends' =
lhs: rhs: self: super:
let super' = rhs self super; in
super' // lhs self (super // super');
inherit (pkgs.lib) foldl' flip;
in
pkgs.overridePackages (
foldl' (flip extends') (_:_:{}) [
(import ./extraPkgs.nix)
machineOverlay
# ...
]
)
```
I can add my own private packages in their own overlay layer.
`extraPkgs.nix`:
``` nix
self: super: with self; {
foo = callPackage ./foo { };
bar = callPackage ./bar { };
}
```
And I can add machine specific settings
in another overlay layer.
`machineA.nix`:
``` nix
self: super: with self; {
boost = super.boost.override { mpi = openmpi; };
fftw = super.fftw.overrideDerivation (attrs: {
configureFlags = attrs.configureFlags ++ [ "--enable-avx" ];
});
foo = (super.foo.override {
# Some machine specific parameters
}).overrideDerivation (attrs: {
# Some machine specific attributes
});
}
```
`machineB.nix`:
``` nix
self: super: with self; {
# Different machine specific settings.
}
```
Having such an overlay mechanism built into nixpkgs
would simplify this pattern.
I would welcome such a change.
Best, Andreas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Introducing Nixpkgs Overlays
Hello Nicolas, I support this work. It would make layering of fractalide on nixpkgs easier, also the layering of a developer's set of agents on top of fractalide would be easier and cleaner. I really hope this lands. kudos kr/sjm ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Introducing Nixpkgs Overlays
Hello Nixpkgs users,
One of the issues of Nixpkgs is the lack of modularity. By
modularity, I mean that someone else can provide a complementary
source of packages which works side-by-side with Nixpkgs, and that as
a user, you can combine multiple of these sources.
# Extending Nixpkgs Today
Today, we have a lot of way to extend Nixpkgs:
1. We can import nixpkgs, and add packages on top of it.
```nix
let pkgs = import {}; in
pkgs // rec {
foo = pkgs.foo.override { enableBar = true; };
bar = import ./pkgs/bar { inherit (pkgs) stdenv fetchurl foo; };
}
```
2. We have `packageOverrides` from ~/.nixpkgs/config.nix.
```nix
{
packageOverrides = pkgs: rec {
foo = pkgs.foo.override { enableBar = true; };
bar = import ./pkgs/bar { inherit (pkgs) stdenv fetchurl foo; };
};
}
```
3. We have `pkgs.overridePackages`.
```nix
let pkgs = import {}; in
pkgs.overridePackages (self: super: {
foo = super.foo.override { enableBar = true; };
bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
})
```
But none of these approaches can easily be composed with other sources
doing similar things.
# Extending Nixpkgs with Overlays
Fortunately, the recent improvements of Nixpkgs inner structure
[2,3,4,5], highlights that we have a list of layers which are stacked
one on top of the others.
These layers are literally what composes Nixpkgs, we could technically
rewrite the entire Nixpkgs collection into thousands of these layers,
where each one handles a single package at a time, but this would be
inefficient.
On the other side, we could expose this layering mechanism to external
sources of Nixpkgs, and gain the modularity that Nixpkgs never had.
This is what I did with the `overlays` pull-request [1].
An overlay is literally an additional layer added on top of Nixpkgs,
within the fix-point of Nixpkgs.
The same principles was already used by `pkgs.overridePackages`
without adding the composition ability of overlays.
An overlay would look something like [6]:
```nix
self: super:
{
foo = super.foo.override { enableBar = true; };
bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
}
```
Which is essentially the minimal part of all the previous examples,
and more over the exact same way we represents the different layers
within Nixpkgs.
The modularity aspect of overlays comes from the fact that overlays
are listed, either in the directory, or as argument of nixpkgs. By
default the ~.nixpkgs/overlays/ directory is considered unless
NIXPKGS_OVERLAYS specify otherwise.
This directory should only contain Nix expression files (or directory
with a default.nix) which are overlays, with the same layout as the
example above. One can also use a symbolic link to where each of
these overlays are installed on your file system.
The directory is converted in a list of elements, which are ordered
based on the alphabetical order of the elements within the directory.
[1] https://github.com/NixOS/nixpkgs/pull/21243
[2] https://github.com/NixOS/nixpkgs/pull/9400
[3] https://github.com/NixOS/nixpkgs/pull/14000
[4] https://github.com/NixOS/nixpkgs/pull/15043
[5] With the help of Peter Simons' extend function.
[6] https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix
# Planning for the future.
As part of this modification, I intend to remove the
`pkgs.overridePackages` function, as this one can literally be
replaced by the following Nix expression:
```nix
let pkgs = import {}; in
import pkgs.path { overlay = [ (self: super: {
foo = super.foo.override { enableBar = true; };
bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
}) ]; }
```
One additional reason to remove `pkgs.overridePackages` function, is
that the creation of this function also stands as the function which
adds Nixpkgs fix-point. The `pkgs.overridePackages` function does not
play nicely with the `security-updates` branch, as for the applying
patches to packages, we have peels the fix-point out, as well as this
function. Thus, there would be no way to benefit from the security
updates while using `pkgs.overridePackages`, without making it much
more complex.
On the other hand, the overlays system play nicely with the
`security-updates` branch, as it add layers within the existing
fix-point, and thus peeling the fix-point does not change the
overlays.
# If you are not yet convinced
A while ago, I suggested to add modularity to NixOS [7], and I
personally consider it as a great success for one reason. This reason
is allow any user to extend NixOS without changing syntax, and without
changing NixOS sources.
Adding overlays would add the same benefits to Nixpkgs, such as giving
the opportunities to multiple companies to distribute software [6
(above)] without providing a full clone of Nixpkgs them-self.
[7] http://lists.science.uu.nl/pipermail/nix-dev/2008-November/001467.html
--
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
_
