Re: [Nix-dev] Questions about the all-packages fixpoint
On Sat, Mar 18, 2017 at 7:40 PM, Benno Fünfstück
wrote:
> Ok, thanks again!
>
>> - 0 hop: means that you are linking statically. Any change of such
> dependency would cause your package to be rebuilt.
>
> Does this mean that when I link statically, I cannot go through self? so the
> following would be wrong:
>
> foo = import foo.nix { bar = self.bar; }
>
> if `foo` links statically against `bar`?
Yes, this used to be the case, the problem is that we cannot override
statically linked libraries, so at the end I think I would split the
self-fixpoint into one fix-point for statically linking, and a second
fix-point of dynamically-linking.
Thus at the end, I expect to create an interface such as:
{dynamicLink, staticLink}: super:
But then we would have to revise the way callPackage works in order to
distinguish statically linked dependencies versus dynamically linked
dependencies.
This would become a non-problem as soon as I manage time to answer and
push S.O.S. [1] forward, in order to get rid of callPackage.
[1] https://github.com/zimbatm/rfcs/pull/3
--
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Questions about the all-packages fixpoint
Ok, thanks again!
> - 0 hop: means that you are linking statically. Any change of such
dependency would cause your package to be rebuilt.
Does this mean that when I link statically, I cannot go through self? so
the following would be wrong:
foo = import foo.nix { bar = self.bar; }
if `foo` links statically against `bar`?
Nicolas Pierron schrieb am Sa., 18. März 2017
um 17:59 Uhr:
> On Sat, Mar 18, 2017 at 3:27 PM, Benno Fünfstück
> wrote:
> > Nicolas Pierron schrieb am Fr., 17. März
> 2017
> > um 22:36 Uhr:
> >>
> >> On Fri, Mar 17, 2017 at 9:38 PM, Benno Fünfstück
> >> wrote:
> >> > One thing that is nicer about `self.callPackage` though is that you
> can
> >> > follow the rule "whenever taking something from self does not lead
> >> > infinite
> >> > recursion, take it from self" when writing overrides.
> >>
> >> And we should not advertise that, because this would lead to packages
> >> which are not patched, under the rules of the future security-update
> >> work.
> >> So whatever you think this is simpler or not, this is incorrect, I
> >> guess we could nullify these functions in the latest layer, preventing
> >> callPackage to ever be used through `self`.
> >
> >
> > Oh, why would that lead to not applying security updates? I'm not very
> > familar with "future security-update work", but that sounds unexpected to
> > me. What's so deeply magical about security updates?
>
> The security-update branch needs to peel off the fix-point of Nixpkgs.
> Then the patching behaviour is added in-between every package by doing
> one more step of the Nixpkgs function.
> This has the consequences of giving the following semantic to the
> numbers of hops that you use to fetch the dependencies:
> - 0 hop: means that you are linking statically. Any change of such
> dependency would cause your package to be rebuilt.
> - 1 hop: means that you are linking dynamically. Any change of such
> dependency would cause your package to be patched.
> - >1 hops: means nothing. Any change of such dependency would leave
> your package to be unchanged.
>
> Going through `self` is adding hops. If you were to call
> `self.callPackage` instead of `super.callPackage`, you would have 2
> hops instead of 1 hop, because callPackage generate attribute set out
> of `self`.
>
> --
> Nicolas Pierron
> http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Questions about the all-packages fixpoint
On Sat, Mar 18, 2017 at 3:27 PM, Benno Fünfstück wrote: > Nicolas Pierron schrieb am Fr., 17. März 2017 > um 22:36 Uhr: >> >> On Fri, Mar 17, 2017 at 9:38 PM, Benno Fünfstück >> wrote: >> > One thing that is nicer about `self.callPackage` though is that you can >> > follow the rule "whenever taking something from self does not lead >> > infinite >> > recursion, take it from self" when writing overrides. >> >> And we should not advertise that, because this would lead to packages >> which are not patched, under the rules of the future security-update >> work. >> So whatever you think this is simpler or not, this is incorrect, I >> guess we could nullify these functions in the latest layer, preventing >> callPackage to ever be used through `self`. > > > Oh, why would that lead to not applying security updates? I'm not very > familar with "future security-update work", but that sounds unexpected to > me. What's so deeply magical about security updates? The security-update branch needs to peel off the fix-point of Nixpkgs. Then the patching behaviour is added in-between every package by doing one more step of the Nixpkgs function. This has the consequences of giving the following semantic to the numbers of hops that you use to fetch the dependencies: - 0 hop: means that you are linking statically. Any change of such dependency would cause your package to be rebuilt. - 1 hop: means that you are linking dynamically. Any change of such dependency would cause your package to be patched. - >1 hops: means nothing. Any change of such dependency would leave your package to be unchanged. Going through `self` is adding hops. If you were to call `self.callPackage` instead of `super.callPackage`, you would have 2 hops instead of 1 hop, because callPackage generate attribute set out of `self`. -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Questions about the all-packages fixpoint
Nicolas Pierron schrieb am Fr., 17. März 2017 um 22:36 Uhr: > On Fri, Mar 17, 2017 at 9:38 PM, Benno Fünfstück > wrote: > > One thing that is nicer about `self.callPackage` though is that you can > > follow the rule "whenever taking something from self does not lead > infinite > > recursion, take it from self" when writing overrides. > > And we should not advertise that, because this would lead to packages > which are not patched, under the rules of the future security-update > work. > So whatever you think this is simpler or not, this is incorrect, I > guess we could nullify these functions in the latest layer, preventing > callPackage to ever be used through `self`. > Oh, why would that lead to not applying security updates? I'm not very familar with "future security-update work", but that sounds unexpected to me. What's so deeply magical about security updates? ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Questions about the all-packages fixpoint
On Fri, Mar 17, 2017 at 9:38 PM, Benno Fünfstück wrote: > One thing that is nicer about `self.callPackage` though is that you can > follow the rule "whenever taking something from self does not lead infinite > recursion, take it from self" when writing overrides. And we should not advertise that, because this would lead to packages which are not patched, under the rules of the future security-update work. So whatever you think this is simpler or not, this is incorrect, I guess we could nullify these functions in the latest layer, preventing callPackage to ever be used through `self`. -- Nicolas Pierron http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Questions about the all-packages fixpoint
Hi Nicolas,
thank you for the answer, that clears up must of my questions. I do think
that `self` is pretty standard throughout nixpkgs (at least the parts that
I have worked with) for exactly this fixpoint construction, thus the name
conflict is a little unfortuntate, but seems we cannot do much about that
right now.
However, I am still unsure what you mean with
If `super.callPackage` were taking all its arguments from `super`,
then this would imply that you could not easily override the argument
of packages which are inside the dependencies of another one.
By taking the packages from `self`, we can easily replace it for all its
uses.
Well, that's why you should almost never use `super.callPackage`. I would
imagine that inside of `all-packages.nix`, at the top, we would have
something like:
`let callPackage = self.callPackage in ... `
so that all occurences of `callPackage` refer to `self.callPackage`, yet
`pkgs.callPackage` only looks inside the scope of `pkgs`. There's only one
problem I see with this, when packages take what is called `self` in
all-packages.nix directly as an argument, but I think they should take
`pkgs` instead anyway.
For me, it feels like `super.callPackage` calls something in the scope of
`super`, but perhaps that only makes more sense to me because I'm used to
it. Now that I think of it, there's an argument for `super.callPackage` to
take the deps from `self` as well: that way, `foo = super.callPackage
path/to/foo {}; ` is equavilent to `super.foo` if foo is defined like that
in `super`.
One thing that is nicer about `self.callPackage` though is that you can
follow the rule "whenever taking something from self does not lead infinite
recursion, take it from self" when writing overrides.
--
Benno
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Questions about the all-packages fixpoint
Hi Benno,
Before answering, let me give you a brief intro to the fix-extend
function combo.
The extend function is similar to the update operator of Nix `//`, the
main difference is that it give its left-hand side (`super`) as
argument to its right hand side.
But this extend function give an extra argument (`self`) to all its
arguments. When chaining these extend function, the `self` argument is
the same for the all layers.
When a fix point is applied to this chain of extend function, `self`
corresponds to the last set produced by the extend function chain.
Note that using `self` can easily cause infinite recursion, for
example if the addition of an attribute to the set depends on the
presence of an attribute in `self`.
On Fri, Mar 17, 2017 at 5:04 PM, Benno Fünfstück
wrote:
> 2. In `stage.nix`, why do we construct a *local* fixpoint just for
> all-packages? The current code is:
>
> allPackages = self: super:
> let res = import ./all-packages.nix
> { inherit lib nixpkgsFun noSysDirs config; }
> res self;
> in res;
>
> Even more confusingly, if we look at `all-packages.nix`, it is a function
> defined like this:
>
> { lib, nixpkgsFun, noSysDirs, config}:
> self: pkgs:
>
> So the variable that is called `self` in `stage.nix` is bound to what is
> called `pkgs` in `all-packages.nix`, and `self` in `all-packages.nix`
> actually refers to what is called `res` in `stage.nix`
This is because this code is still legacy before the introduction of
the extend function.
We should not use this `self` argument under all-packages.nix, but
last time I tried to avoid changing any hashes.
Another reason why I did not address this, such as renaming it because
`self` is quite an overloaded name, and a naive search and replace
would not work as expected.
Note, there is also a package named `self`, for interpreting the Self language.
So `pkgs` of all-packages.nix is literaly the result of the fix-point,
given to all overlays under the name `self`.
> 1. An override is given the arguments `self` and `super`, as expected. But
> why does `super.callPackage` resolve dependencies from `self`?
`callPackage` take a function as argument, or a path to a function,
and list the arguments of it. It then create a set with the requested
argument and a set given as argument.
In `top-level/splice.nix`, the callPackage function is given an
argument which is constructed from the result of the fix-point.
Thus `super.callPackage` will fill the arguments of the functions with
the value taken within the result of the fix-point.
> I would have
> expected it to not know about `self`, and resolve dependencies in the scope
> of `super`. If I wanted this behaviour, I would have used
> `self.callPackage`, but the current behaviour makes that unnecessary. Why is
> this implemented like that?
The history of all-packages.nix used to be that we had a "rec" keyword
on top of the attribute set.
Then we added the `self` reference that you complained about in (2) in
order to be able to override packages.
If `super.callPackage` were taking all its arguments from `super`,
then this would imply that you could not easily override the argument
of packages which are inside the dependencies of another one.
By taking the packages from `self`, we can easily replace it for all its uses.
> This is confusing to me. Also, why do we need `self` and `pkgs`? Wouldn't
> one of them be enough?
Yes, but again, renaming these variables is hard without the proper
tooling to guarantee that we are not making any mistake.
--
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Questions about the all-packages fixpoint
Hello all,
I am (again) wrapping my head around the current construction of the
package set. I think I understand how it works, but there's a few subtle
details that I feel like they are confusing, bur probably necessary. I
would like to understand why:
1. An override is given the arguments `self` and `super`, as expected. But
why does `super.callPackage` resolve dependencies from `self`? I would have
expected it to not know about `self`, and resolve dependencies in the scope
of `super`. If I wanted this behaviour, I would have used
`self.callPackage`, but the current behaviour makes that unnecessary. Why
is this implemented like that?
2. In `stage.nix`, why do we construct a *local* fixpoint just for
all-packages? The current code is:
allPackages = self: super:
let res = import ./all-packages.nix
{ inherit lib nixpkgsFun noSysDirs config; }
res self;
in res;
Even more confusingly, if we look at `all-packages.nix`, it is a function
defined like this:
{ lib, nixpkgsFun, noSysDirs, config}:
self: pkgs:
So the variable that is called `self` in `stage.nix` is bound to what is
called `pkgs` in `all-packages.nix`, and `self` in `all-packages.nix`
actually refers to what is called `res` in `stage.nix`
This is confusing to me. Also, why do we need `self` and `pkgs`? Wouldn't
one of them be enough?
Greetings,
Benno
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
