Re: [Nix-dev] nix-daemon and private git repos

2017-07-06 Thread Harmen via nix-dev
On Tue, Jul 04, 2017 at 08:10:09PM +, zimbatm wrote:

Thanks for the suggestions.

I took away from this that it's best to not have Nix deal with the checkouts.
For now I'll keep things as separate repos, to keep things easier.

Eventually I would like to go to one pkgs tree for all repos, but I'll think
about how to do that best. It also ties into integration and system testing,
which is not just Nix related.

thanks!
Harmen
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-04 Thread zimbatm
Yes, the source is part of the build input and is uploaded to the worker to
run the build.

On Tue, 4 Jul 2017, 17:46 Tomas Hlavaty, 
wrote:

> On Tue 04 Jul 2017 at 13:49, "Alexander V. Nikolaev" 
> wrote:
> > On Mon, Jul 03, 2017 at 03:19:31PM +0200, Harmen wrote:
> >
> > I have `fetchgitCustom` expression, which can use pre-seeded "deploy"
> > keys (but with some security implications -- because key is
> > world-readable). It works with sandbox builds, and should work with
> > hydra as well.
>
> And does it work if the package is build by remote build slaves?
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-04 Thread Tomas Hlavaty
On Tue 04 Jul 2017 at 13:49, "Alexander V. Nikolaev"  wrote:
> On Mon, Jul 03, 2017 at 03:19:31PM +0200, Harmen wrote:
>
> I have `fetchgitCustom` expression, which can use pre-seeded "deploy"
> keys (but with some security implications -- because key is
> world-readable). It works with sandbox builds, and should work with
> hydra as well.

And does it work if the package is build by remote build slaves?
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-04 Thread Alexander V. Nikolaev
On Mon, Jul 03, 2017 at 03:19:31PM +0200, Harmen wrote:

I have `fetchgitCustom` expression, which can use pre-seeded "deploy"
keys (but with some security implications -- because key is
world-readable). It works with sandbox builds, and should work with
hydra as well.

If anyone interesting in this solution, I'll prepare PR soon.

> I'm struggling to get fetchgitPrivate to work on nix-daemon installation (no
> NixOS, these are Ubuntu machines with nix).
> I can make it work on my dev machine, with is a non-daemon install, by setting
> 
> NIX_PATH=ssh-config-file=/my/ssh/config:$NIX_PATH
> 
> But that doesn't work in sandboxed daemon mode, because the nixbld* users 
> can't
> read that file (both because of access rights, and because of the sandbox).
> 
> Nix has this warning in fetchgitPrivate:
> > Note that the config file and any keys it points to must be readable
> > by the build user, which depending on your nix configuration means making it
> > readable by the build-users-group, the user of the running nix-daemon, or 
> > the
> > user calling the nix command which started the build. Similarly, if using an
> > ssh agent ssh-auth-sock must point to a socket the build user can access.
> > You may need StrictHostKeyChecking=no in the config file. Since ssh
> > will refuse to use a group-readable private key, if using build-users you 
> > will
> > likely want to use something like IdentityFile /some/directory/%u/key and 
> > have
> > a directory for each build user accessible to that user.
> from
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgit/private.nix
> 
> which sounds reasonable, but it I don't see how to do that since I don't know
> exactly which user will run the build. Also because of the sandbox the build
> can't read the ssh config file at all.
> 
> 
> So next option is to generate the configfile in my expression, a-la
> https://www.mpscholten.de/nixos/2016/07/07/private-github-repositories-and-nixos.html
> but I don't know how to set nix.path from inside an expression. It would also
> require bundling the key with the expression, but if that works...
> 
> 
> I can't be the first to want to use fetchgitPrivate with a sandboxed
> nix-daemon. Any experiences or tips?
> Thanks!
> Harmen
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-04 Thread Tomas Hlavaty
Hi Harmen,

On Mon 03 Jul 2017 at 15:19, Harmen  wrote:
> I can't be the first to want to use fetchgitPrivate with a sandboxed
> nix-daemon. Any experiences or tips?

I had it working but there are several cases which needs extra setup
that I recommend to avoid fetchgitPrivate and use source variables,
e.g. src = ; This works the same in any context, e.g. in
hydra, without hydra, etc.  You just need to set the proprietarySrc
accordingly.

Tomas
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-04 Thread Tomas Hlavaty
On Mon 03 Jul 2017 at 15:38, Harmen  wrote:

> On Mon, Jul 03, 2017 at 03:27:34PM +0200, Tomas Hlavaty wrote:
>> Hi Harmen,
>> 
>> On Mon 03 Jul 2017 at 15:19, Harmen  wrote:
>> > I can't be the first to want to use fetchgitPrivate with a sandboxed
>> > nix-daemon. Any experiences or tips?
>> 
>> I had it working but there are several cases which needs extra setup
>> that I recommend to avoid fetchgitPrivate and use source variables,
>> e.g. src = ; This works the same in any context, e.g. in
>> hydra, without hydra, etc.  You just need to set the proprietarySrc
>> accordingly.
>
> could you be a bit more specific what you mean?

Have a look at NIX_PATH: https://nixos.org/nix/manual/#env-NIX_PATH

You can configure the NIX_PATH locally, e.g.

export NIX_PATH=proprietarySrc=$HOME/src/proprietary:$NIX_PATH

or explicitly via command-line arg to nix-build, nix-shell etc.

Then instead of writing your package as:

stdenv.mkDerivation rec {
  src = fetchgitPrivate ...;
  ...
}

write

stdenv.mkDerivation rec {
  src = ;
  ...
}

If used in the context of hydra, hydra configuration says, where and how
to obtain that software, then hydra downloads it and passes it further
for build.  E.g. in
, there
are paths nix and nixpkgs, which are then written in nix as  and
 respectively.  In this example, no fetchgitPrivate is used but
you can see those paths.  In your case of proprietary package, you don't
have to do anything special except allowing hydra to download those
proprietary repos.  We have a read-only access to our git repo with ssh
key for hydra.

If you go with the fetchgitPrivate option, you have to grant access in
too many places, i.e. in every context where your package can be
evaluated.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-03 Thread Harmen
On Mon, Jul 03, 2017 at 03:27:34PM +0200, Tomas Hlavaty wrote:
> Hi Harmen,
> 
> On Mon 03 Jul 2017 at 15:19, Harmen  wrote:
> > I can't be the first to want to use fetchgitPrivate with a sandboxed
> > nix-daemon. Any experiences or tips?
> 
> I had it working but there are several cases which needs extra setup
> that I recommend to avoid fetchgitPrivate and use source variables,
> e.g. src = ; This works the same in any context, e.g. in
> hydra, without hydra, etc.  You just need to set the proprietarySrc
> accordingly.

Thanks for the help Tomas,

could you be a bit more specific what you mean?

Thanks!
Harmen
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] nix-daemon and private git repos

2017-07-03 Thread Tomasz Czyż
You don't need to know the user itself I think, you could share it with
nixbld group (probably).
Also, I think this way recommends to use ssh-agent as far as I remember
looking at that (but could changed).

What I did for one project was:

 337   fetchgitPrivate = (args: derivation ((nixpkgs.fetchgit
> args).drvAttrs // {
>  338 # This function is wrapper around nix fetch git function
> to use SSH key.
>  339 # It can be useful when fetching from private
> repositories.
>  340 #
>
>  341 # Original function is in
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgit/default.nix
>  342 #
>
>  343 GIT_SSH = rr.lib.writeShellScript "git-ssh" ''#!
> ${nixpkgs.bash}/bin/bash -eu
>  344   if ! [[ -r '${repositories-ssh-private-key-path}'
> ]];then
>  345 echo "ERROR:" >&2
>
>  346 echo "ERROR:" >&2
>
>  347 echo "ERROR: ssh key
> '${repositories-ssh-private-key-path}' is not readable by build user
> ($(id))" >&2
>  348 echo "ERROR:" >&2
>
>  349 echo "ERROR:" >&2
>
>  350 exit 5
>
>  351   fi
>
>  352   exec ${nixpkgs.openssh}/bin/ssh -o
> StrictHostKeyChecking=no -i '${repositories-ssh-private-key-path}' "$@"
>  353 '';

But probably is way better to go with agent.

2017-07-03 14:19 GMT+01:00 Harmen :

> Hi all,
>
> I'm struggling to get fetchgitPrivate to work on nix-daemon installation
> (no
> NixOS, these are Ubuntu machines with nix).
> I can make it work on my dev machine, with is a non-daemon install, by
> setting
>
> NIX_PATH=ssh-config-file=/my/ssh/config:$NIX_PATH
>
> But that doesn't work in sandboxed daemon mode, because the nixbld* users
> can't
> read that file (both because of access rights, and because of the sandbox).
>
> Nix has this warning in fetchgitPrivate:
> > Note that the config file and any keys it points to must be readable
> > by the build user, which depending on your nix configuration means
> making it
> > readable by the build-users-group, the user of the running nix-daemon,
> or the
> > user calling the nix command which started the build. Similarly, if
> using an
> > ssh agent ssh-auth-sock must point to a socket the build user can access.
> > You may need StrictHostKeyChecking=no in the config file. Since ssh
> > will refuse to use a group-readable private key, if using build-users
> you will
> > likely want to use something like IdentityFile /some/directory/%u/key
> and have
> > a directory for each build user accessible to that user.
> from
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgit/
> private.nix
>
> which sounds reasonable, but it I don't see how to do that since I don't
> know
> exactly which user will run the build. Also because of the sandbox the
> build
> can't read the ssh config file at all.
>
>
> So next option is to generate the configfile in my expression, a-la
> https://www.mpscholten.de/nixos/2016/07/07/private-
> github-repositories-and-nixos.html
> but I don't know how to set nix.path from inside an expression. It would
> also
> require bundling the key with the expression, but if that works...
>
>
> I can't be the first to want to use fetchgitPrivate with a sandboxed
> nix-daemon. Any experiences or tips?
> Thanks!
> Harmen
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>



-- 
Tomasz Czyż
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev