Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Bjørn Forsman]

On 9 May 2016 at 20:21, Eelco Dolstra  wrote:

> * The mirroring script only copies tarballs of packages in the Nixpkgs Hydra
> jobset. Since moreutils is not part of the jobset, it's not mirrored. This can
> be fixed by adding a meta.platforms attribute to moreutils.

Thank you! (Completely forgot about meta.platforms...)

- Bjørn
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Eelco Dolstra]

Hi,

On 08/05/16 02:19, J. R. Haigh wrote:

> Dear Nix project leaders, I don't find NixOS to be very deterministic at all,
> and it's nearly always for the same reason: source files on random servers
> scattered across the Internet going walkabouts.

We have a content-addressed tarball mirror at http://tarballs.nixos.org for this
purpose. "fetchurl" will automatically use this mirror to obtain files by hash.
However:

* The mirroring script was not running lately. I've revived it so 16.03 tarballs
are mirrored now
(https://github.com/NixOS/nixos-org-configurations/commit/a17ccf87deae4fb86639c8d34ab5938edd68d8c4).

* The mirroring script only copies tarballs of packages in the Nixpkgs Hydra
jobset. Since moreutils is not part of the jobset, it's not mirrored. This can
be fixed by adding a meta.platforms attribute to moreutils.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Arnold Krille]

On Sun, 8 May 2016 18:11:09 +0300 Игорь Пашев 
wrote:
> Can't be nix cache used for this? Source tarballs are kind of
> derivation too. 

Somehow I was under the impression that the output of
"fetchFromUrl" or "fetchFromGit" was also a result worth storing in the
nix-store. And thus also part of the 'binary'-cache.
Or is that only when configuring a private binary-cache/hydra?

- Arnold


signature.asc
Description: PGP signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage - please can we improve the determinism

[Brad Ediger]

We have also run into this issue with spotify and similar unfree /
non-redistributable packages. The consensus last time we had upstream URLs
disappear was that it is not permissible to cache and redistribute even the
original source.

There may be a partial solution for this problem, but it will not solve the
issue for nonfree packages.

On Sun, May 8, 2016 at 10:11 AM Игорь Пашев  wrote:

> Can't be nix cache used for this? Source tarballs are kind of derivation
> too.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Игорь Пашев]

Can't be nix cache used for this? Source tarballs are kind of derivation too.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Bjørn Forsman]

On 8 May 2016 at 02:19, J. R. Haigh  wrote:
> Dear Nix project leaders,
> I don't find NixOS to be very deterministic at all, and it's nearly 
> always for the same reason: source files on random servers scattered across 
> the Internet going walkabouts.
> Please can we have all source files hosted by the Nix project and all 
> source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by the 
> Nix-hosted copies.

+1. Except I don't think we should change the URLs in nixpkgs, those
should still point to upstream. IMO, Nix fetchers should be
configurable to try Nix mirrors before (or after) the given URLs in
Nix package files.

> Ideally there should be a way to specify multiple URLs for each source file, 
> so both the original and the Nix-hosted locations can be specified, as well 
> as any other mirrors if there are any.

That is already possible. For instance, fetchurl can take a list of
urls in the "urls" attribute (note the 's'). (But again, I'd prefer to
not have to specify these Nix provided backup URLs manually for each
package, better do that globally.)

> In this specific instance, please could someone with commit access resolve 
> this breakage a.s.a.p.?: 
> https://github.com/NixOS/nixpkgs-channels/blob/nixos-15.09/pkgs/tools/misc/moreutils/default.nix

That was fixed just a couple of days ago. But unfortunately it takes a
while for the change to hit the channel :-( (Until then, build from
git.)

Best regards,
Bjørn Forsman
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Oliver Charles]

On Sun, 8 May 2016, 1:19 a.m. J. R. Haigh,  wrote:

> Dear Nix project leaders,
> I don't find NixOS to be very deterministic at all, and it's
> nearly always for the same reason: source files on random servers scattered
> across the Internet going walkabouts.
>

While I agree this is a problem, this is not a problem with determinism. If
the source code can't be found as it is expected, then the build fails -
that *is* deterministic.

Please can we have all source files hosted by the Nix project and
> all source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by
> the Nix-hosted copies. Ideally there should be a way to specify multiple
> URLs for each source file, so both the original and the Nix-hosted
> locations can be specified, as well as any other mirrors if there are any.
>

We do have the mirror:// scheme which knows to look in many locations, but
I imagine the source you're referring to is not mirrored?

Ollie
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Source URL breakage – please can we improve the determinism

[Roger Qiu]

Its definitely annoying.

There's discussion about it involving ipfs.

One stopgap solution is to rollback just for only that package. You can do
that with nix-env -f on a tar.gz archive of nixpkgs from a commit hash on
github.
On 08/05/2016 10:19 AM, "J. R. Haigh"  wrote:

> Dear Nix project leaders,
> I don't find NixOS to be very deterministic at all, and it's
> nearly always for the same reason: source files on random servers scattered
> across the Internet going walkabouts.
> Please can we have all source files hosted by the Nix project and
> all source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by
> the Nix-hosted copies. Ideally there should be a way to specify multiple
> URLs for each source file, so both the original and the Nix-hosted
> locations can be specified, as well as any other mirrors if there are any.
> As for the duration of the hosting, I think that at least every
> release build that goes through Hydra to build live images and such should
> keep hosted files (source or binary alike) indefinitely. Anything that was
> not one of these Hydra builds I don't mind too much if they go missing
> because I nearly always stick to the release builds, but it would still be
> appreciated if all source files could be hosted indefinitely to aid with
> binary searches of regressions or other changes.
> Better still, if magnet URIs could be specified and Nix would
> support downloading from torrents then it would be very easy to avoid
> breakage due to missing source files.
> 
> In this specific instance, please could someone with commit access
> resolve this breakage a.s.a.p.?:
> https://github.com/NixOS/nixpkgs-channels/blob/nixos-15.09/pkgs/tools/misc/moreutils/default.nix
> Perhaps it could be done by updating it from 0.57 to 0.58 or by
> finding it on another mirror somewhere. The Debian source archive at
> http://ftp.de.debian.org/debian/pool/main/m/moreutils/ lists 0.47, 0.52,
> and 0.58, but not 0.57. Moreutils is depended on by Mercurial and Mercurial
> is a build dependency of many things, so that one source breakage has
> broken a lot of things from building in the head of the NixOS 15.09 release
> channel branch.
>
> Best regards,
> James R. Haigh.
> --
> Sent from Claws Mail on NixOS.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Source URL breakage – please can we improve the determinism

[J. R. Haigh]

Dear Nix project leaders,
I don't find NixOS to be very deterministic at all, and it's nearly 
always for the same reason: source files on random servers scattered across the 
Internet going walkabouts.
Please can we have all source files hosted by the Nix project and all 
source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by the 
Nix-hosted copies. Ideally there should be a way to specify multiple URLs for 
each source file, so both the original and the Nix-hosted locations can be 
specified, as well as any other mirrors if there are any.
As for the duration of the hosting, I think that at least every release 
build that goes through Hydra to build live images and such should keep hosted 
files (source or binary alike) indefinitely. Anything that was not one of these 
Hydra builds I don't mind too much if they go missing because I nearly always 
stick to the release builds, but it would still be appreciated if all source 
files could be hosted indefinitely to aid with binary searches of regressions 
or other changes.
Better still, if magnet URIs could be specified and Nix would support 
downloading from torrents then it would be very easy to avoid breakage due to 
missing source files.

In this specific instance, please could someone with commit access 
resolve this breakage a.s.a.p.?: 
https://github.com/NixOS/nixpkgs-channels/blob/nixos-15.09/pkgs/tools/misc/moreutils/default.nix
Perhaps it could be done by updating it from 0.57 to 0.58 or by finding 
it on another mirror somewhere. The Debian source archive at 
http://ftp.de.debian.org/debian/pool/main/m/moreutils/ lists 0.47, 0.52, and 
0.58, but not 0.57. Moreutils is depended on by Mercurial and Mercurial is a 
build dependency of many things, so that one source breakage has broken a lot 
of things from building in the head of the NixOS 15.09 release channel branch.

Best regards,
James R. Haigh.
-- 
Sent from Claws Mail on NixOS.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev