Re: [Nix-dev] fetchgit
Is bcupkgs a fork of Nixpkgs? If so then you probably want to override Nixpkgs in your NIX_PATH by setting NIX_PATH=nixpkgs=~/local/bcupkgs. On 9 Mar 2017 11:53 AM, "Sébastien Petitdemange" < sebastien.petitdema...@esrf.fr> wrote: Hi, I've modify fetchgit module to add an optional argument to filter submodule. And I would like to use this optional argument to build my local package. Unfortunately, nix-build still get fetchgit from the main channel instead of ~/local/bcupckgs: -> nix-build --show-trace ~/local/bcupkgs --cores 8 -A lima-core error: while evaluating the attribute ‘src’ of the derivation ‘lima-core-1.7’ at /users/blissadm/local/bcupkgs/pkgs/development/libraries/ lima-core/default.nix:10:3: anonymous function at /nix/store/apyslcfxrq2s4hsbbidk04jgjlw5vl6n-nixpkgs-17.09pre102884. b94e253/nixpkgs/pkgs/build-support/fetchgit/default.nix:14:1 called with unexpected argument ‘submodulesList’, at /users/blissadm/local/bcupkgs/pkgs/development/libraries/ lima-core/default.nix:12:9 it didn't get the "good one" which is in ~/local/bcupkgs/pkgs/build-support/fetchgit. I've tried to install nix-prefetch-git locally but it doesn't change anything. What is the good way to archive this? Regards, SEB ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit
Hi Profpatsch,
Here is the tree of my private package:
./local/bcupkgs/
├── default.nix
└── pkgs
├── build-support
│ └── fetchgit
│ ├── builder.sh
│ ├── default.nix
│ └── nix-prefetch-git
└── development
└── libraries
└── lima-core
├── 0001-removed-Python-modules-from-install.inc.patch
├── config.patch
└── default.nix
I've attached all the nix script I've wrote so fare.
Thanks for your help,
SEB
On 03/09/2017 06:02 PM, Profpatsch wrote:
> I cannot tell you anything without a patch or at least
> a code sample.
diff --git a/pkgs/build-support/fetchgit/builder.sh b/pkgs/build-support/fetchgit/builder.sh
index 4bbef1d..da46106 100644
--- a/pkgs/build-support/fetchgit/builder.sh
+++ b/pkgs/build-support/fetchgit/builder.sh
@@ -10,6 +10,7 @@ $SHELL $fetcher --builder --url "$url" --out "$out" --rev "$rev" \
${leaveDotGit:+--leave-dotGit} \
${deepClone:+--deepClone} \
${fetchSubmodules:+--fetch-submodules} \
- ${branchName:+--branch-name "$branchName"}
+ ${branchName:+--branch-name "$branchName"} \
+ ${submodulesList:+--submodules-list "$submodulesList"}
stopNest
diff --git a/pkgs/build-support/fetchgit/default.nix b/pkgs/build-support/fetchgit/default.nix
index 7f98c97..01f7da2 100644
--- a/pkgs/build-support/fetchgit/default.nix
+++ b/pkgs/build-support/fetchgit/default.nix
@@ -12,7 +12,8 @@
in "${if matched == null then base else builtins.head matched}${appendShort}";
in
{ url, rev ? "HEAD", md5 ? "", sha256 ? "", leaveDotGit ? deepClone
-, fetchSubmodules ? true, deepClone ? false
+, fetchSubmodules ? true, submodulesList ? ""
+, deepClone ? false
, branchName ? null
, name ? urlToName url rev
}:
@@ -52,7 +53,7 @@ stdenv.mkDerivation {
outputHashMode = "recursive";
outputHash = if sha256 == "" then md5 else sha256;
- inherit url rev leaveDotGit fetchSubmodules deepClone branchName;
+ inherit url rev leaveDotGit fetchSubmodules submodulesList deepClone branchName;
GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt";
diff --git a/pkgs/build-support/fetchgit/nix-prefetch-git b/pkgs/build-support/fetchgit/nix-prefetch-git
index 705d84c..52f6c4d 100755
--- a/pkgs/build-support/fetchgit/nix-prefetch-git
+++ b/pkgs/build-support/fetchgit/nix-prefetch-git
@@ -9,6 +9,7 @@ hashType=$NIX_HASH_ALGO
deepClone=$NIX_PREFETCH_GIT_DEEP_CLONE
leaveDotGit=$NIX_PREFETCH_GIT_LEAVE_DOT_GIT
fetchSubmodules=
+submodulesList=
builder=
branchName=$NIX_PREFETCH_GIT_BRANCH_NAME
@@ -46,6 +47,7 @@ Options:
--no-deepClone Make a shallow clone of just the required ref.
--leave-dotGit Keep the .git directories.
--fetch-submodules Fetch submodules.
+ --submodules-list list of submodules to fetch (empty means all).
--builder Clone as fetchgit does, but url, rev, and out option are mandatory.
"
exit 1
@@ -66,6 +68,7 @@ for arg; do
--no-deepClone) deepClone=false;;
--leave-dotGit) leaveDotGit=true;;
--fetch-submodules) fetchSubmodules=true;;
+ --submodules-list) argfun=submodulesList;;
--builder) builder=true;;
--help) usage; exit;;
*)
@@ -84,6 +87,9 @@ for arg; do
var=${argfun#set_}
eval $var=$arg
;;
+ submodulesList)
+ submodulesList="$arg"
+ ;;
esac
argfun=""
fi
@@ -186,12 +192,15 @@ init_submodules(){
# checkout each submodule
hash=$(echo "$l" | awk '{print substr($1,2)}')
dir=$(echo "$l" | awk '{print $2}')
-name=$(
-git config -f .gitmodules --get-regexp submodule\..*\.path |
-sed -n "s,^\(.*\)\.path $dir\$,\\1,p")
-url=$(git config --get "${name}.url")
-
-clone "$dir" "$url" "$hash" ""
+ match=$(echo $submodulesList|grep $dir || true)
+ if [ -z "$submodulesList" ] || [ -n "$match" ] ; then
+ name=$(
+ git config -f .gitmodules --get-regexp submodule\..*\.path |
+ sed -n "s,^\(.*\)\.path $dir\$,\\1,p")
+ url=$(git config --get "${name}.url")
+
+ clone "$dir" "$url" "$hash" ""
+ fi
done
}
{ system ? builtins.currentSystem}:
let
pkgs = import { inherit system; };
callPackage = pkgs.lib.callPackageWith (pkgs // bcuPackages);
callPackages = pkgs.lib.callPackagesWith (pkgs // bcuPackages);
python27Packages = pkgs.recurseIntoAttrs (
callPackage ./python_packages.nix {
python = pkgs.python27;
self = combinedPythonPackages;
});
pythonPackages = python27Packages;
combinedPythonPackages = pkgs.python27Packages // python27Packages //
bcuPackages;
bcuPackages = {
tango = callPackage ./pkgs/development/libraries/tango {};
pytango = pythonPackages.pytango;
inherit (callPackages
./pkgs/tools/package-management/nix-prefetch-scripts { })
nix-prefetch-git
Re: [Nix-dev] fetchgit
On 17-03-09 04:31pm, Sébastien Petitdemange wrote: > Hi Profpatsch, > > Yes I'm in the folder where I changed fetchgit but it doesn't call my > modification version. > How can I make sure that my package (lima-core) call my modify version? > Is there a way to do that? I cannot tell you anything without a patch or at least a code sample. -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit
Hi Profpatsch, Yes I'm in the folder where I changed fetchgit but it doesn't call my modification version. How can I make sure that my package (lima-core) call my modify version? Is there a way to do that? SEB On 03/09/2017 12:08 PM, Profpatsch wrote: > If you’re in the folder of the nixpkgs where you changed fetchgit, > it should work. If it doesn’t, maybe your changes are not calling > your modified version of fetchgit. <> signature.asc Description: OpenPGP digital signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit
Hi Teo, No, it's not a fork. It contain only few private package + fetchgit with the modification. SEB On 03/09/2017 11:57 AM, Teo Klestrup wrote: > Is bcupkgs a fork of Nixpkgs? If so then you probably want to override > Nixpkgs in your NIX_PATH by setting NIX_PATH=nixpkgs=~/local/bcupkgs. <> signature.asc Description: OpenPGP digital signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit
On 17-03-09 11:52am, Sébastien Petitdemange wrote: > Hi, > > I've modify fetchgit module to add an optional argument to filter > submodule. And I would like to use this optional argument to build my > local package. Unfortunately, nix-build still get fetchgit from the main > channel instead of ~/local/bcupckgs: > > -> nix-build --show-trace ~/local/bcupkgs --cores 8 -A lima-core If you’re in the folder of the nixpkgs where you changed fetchgit, it should work. If it doesn’t, maybe your changes are not calling your modified version of fetchgit. -- Proudly written in Mutt with Vim on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es May take up to five days to read your message. If it’s urgent, call me. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] fetchgit
Hi, I've modify fetchgit module to add an optional argument to filter submodule. And I would like to use this optional argument to build my local package. Unfortunately, nix-build still get fetchgit from the main channel instead of ~/local/bcupckgs: -> nix-build --show-trace ~/local/bcupkgs --cores 8 -A lima-core error: while evaluating the attribute ‘src’ of the derivation ‘lima-core-1.7’ at /users/blissadm/local/bcupkgs/pkgs/development/libraries/lima-core/default.nix:10:3: anonymous function at /nix/store/apyslcfxrq2s4hsbbidk04jgjlw5vl6n-nixpkgs-17.09pre102884.b94e253/nixpkgs/pkgs/build-support/fetchgit/default.nix:14:1 called with unexpected argument ‘submodulesList’, at /users/blissadm/local/bcupkgs/pkgs/development/libraries/lima-core/default.nix:12:9 it didn't get the "good one" which is in ~/local/bcupkgs/pkgs/build-support/fetchgit. I've tried to install nix-prefetch-git locally but it doesn't change anything. What is the good way to archive this? Regards, SEB <> signature.asc Description: OpenPGP digital signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
On Mon, Jan 13, 2014 at 01:40:55PM +0800, S??nke Hahn wrote: I also tried out https-urls. The problem is that not all repos that I need to access are public. So I have to provide some authentication. For ssh I can just put keys in .ssh, but for https in this setting, I'm not sure if it's even possible. If it's GitHub-specific and you want to have access to private repositories, please have a look at this: https://github.com/blog/1270-easier-builds-and-deployments-using-git-over-https-and-oauth We're using this internally for deployment and the following is the quite ugly approach we were using before GH introduced OAuth for HTTPS URLs: https://gist.github.com/aszlig/8397726 Hope this helps. a! -- aszlig Universal dilettante RedMoon Studios GmbH Co KG signature.asc Description: Digital signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
Thomas Bereknyei wrote:
I'm not positive this is the problem, but take a look at another
example. No quotes around the url and different format.
src = fetchgit {
url = git://git.gnupg.org/libgcrypt.git;
rev = 99b18aa53;
sha256 = 1rhbpxqrkfszlv8jvw8s4apwklal07k8zxv5q555l7binc1j1j3z;
};
Thanks for the suggestion. I tried:
url = git://g...@github.com/jekor/haskell-sscgi.git;
This was the result:
fatal: Unable to look up g...@github.com (port 9418) (Name or service not
known)
So I tried
url = git://github.com/jekor/haskell-sscgi.git;
, which works, great! (I get another error message based on a missing
Setup.hs, but that's an easy fix.)
Omitting the username does not seem to work for bitbucket, though. Nixops
just hangs after initializing the git repo. I guess, github has 'git' set as
the default user.
On Mon, Jan 13, 2014 at 12:40 AM, Sönke Hahn soenkeh...@gmail.com wrote:
Hi,
I am trying to use fetchgit to package code that is available in
git-repos. I am using for example this expression:
pkgs.fetchgit {
url = g...@github.com:jekor/haskell-sscgi.git;
rev = 1547156d7199bdf40a837040a905d9b6b771dd26;
}
This does not work, the error message being:
error: cannot run ssh: No such file or directory
fatal: unable to fork
error: cannot run ssh: No such file or directory
fatal: unable to fork
Is this not supported by fetchgit? Is there anything I can do about it?
I tried to bring ssh in the PATH somehow, but I didn't manage. Can I
include the openssh package to make it visible to fetchgit?
I also tried out https-urls. The problem is that not all repos that I
need to access are public. So I have to provide some authentication. For
ssh I can just put keys in .ssh, but for https in this setting, I'm not
sure if it's even possible.
Any advice appreciated.
Cheers,
Sönke
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
On 01/13/2014 04:09 PM, Sönke Hahn wrote:
Omitting the username does not seem to work for bitbucket, though. Nixops
just hangs after initializing the git repo. I guess, github has 'git' set as
the default user.
Different providers have different URI schemes, example from nixpkgs:
src = fetchgit {
url = https://bitbucket.org/equalsraf/vim-qt.git ;
rev = 4160bfd5c1380e899d2f426b494fc4f1cf6ae85e;
sha256 = 1qa3xl1b9gqw66p71h53l7ibs4y3zfyj553jss70ybxaxchbhi5b;
};
Vlada
smime.p7s
Description: S/MIME Cryptographic Signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
Vladimír Čunát wrote: Maybe it's easy to add the support, I'll look again at it this evening (I hope), if noone else solves it beforehand. The main problem is the purity of ~/.ssh -- this directory just shouldn't be considered, as the behavior inherently impure. Adding key/password to parameters of fetchgit also doesn't seem very comfortable. That's very interesting. So even if fetchgit would work with ssh I still wouldn't know how to provide keys in an elegant way. So under these circumstances, I wouldn't be that interested in fixing fetchgit anymore. (I generally wonder how to deal with secrets when using nix/nixos/nixops, but that's stuff for another thread, I guess.) Thanks for your insight. Sönke ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
Aristid Breitkreuz wrote:
A noteworthy workaround would be that you can just have a manual checkout
of your private repos and then do
src = ./path/to/repo;
Thanks. Yes, that's what we're doing right now. (And it's not even that
inconvenient, but I thought fetchgit could be better.)
Am 13.01.2014 08:49 schrieb Vladimír Čunát vcu...@gmail.com:
Hi.
On 01/13/2014 07:52 AM, Thomas Bereknyei wrote:
I'm not positive this is the problem, but take a look at another
example. No quotes around the url and different format.
src = fetchgit {
url = git://git.gnupg.org/libgcrypt.git;
rev = 99b18aa53;
sha256 = 1rhbpxqrkfszlv8jvw8s4apwklal07k8zxv5q555l7binc1j1j3z;
};
I'm quite certain the quotes wouldn't break this case. Here you have just
plain git protocol without SSH.
The current implementation of fetchgit doesn't support SSH, judging from
a brief look at its code. (BTW, fetchgit runs in a separate derivation,
so it isn't affected by adding stuff to buildInputs, or even to $PATH.)
Maybe it's easy to add the support, I'll look again at it this evening (I
hope), if noone else solves it beforehand. The main problem is the purity
of ~/.ssh -- this directory just shouldn't be considered, as the behavior
inherently impure. Adding key/password to parameters of fetchgit also
doesn't seem very comfortable.
Vlada
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
Yes, that helps, thanks! It's actually not github, we are using bitbucket for internal repos, but bitbucket seems to have similar possibilities: https://confluence.atlassian.com/display/BITBUCKET/OAuth+on+Bitbucket I'm going to look into this tomorrow. Cheers, Sönke aszlig wrote: On Mon, Jan 13, 2014 at 01:40:55PM +0800, S??nke Hahn wrote: I also tried out https-urls. The problem is that not all repos that I need to access are public. So I have to provide some authentication. For ssh I can just put keys in .ssh, but for https in this setting, I'm not sure if it's even possible. If it's GitHub-specific and you want to have access to private repositories, please have a look at this: https://github.com/blog/1270-easier-builds-and-deployments-using-git-over-https-and-oauth We're using this internally for deployment and the following is the quite ugly approach we were using before GH introduced OAuth for HTTPS URLs: https://gist.github.com/aszlig/8397726 Hope this helps. a! ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
Previously I've had success accessing private git repos via ssh using: 1. a ssh:// URL. EG: ssh://g...@git.corp.ooyala.com/qa/tools/jenkins-ci.git 2. Patching fetchgit to pass the GIT_SSH env variable and configuring GIT_SSH to propagate the keys as expected. Part #2 is much like Azlig implemented. Cheers, Corey -Corey O'Connor coreyocon...@gmail.com http://corebotllc.com/ On Mon, Jan 13, 2014 at 7:22 AM, Sönke Hahn soenkeh...@gmail.com wrote: Yes, that helps, thanks! It's actually not github, we are using bitbucket for internal repos, but bitbucket seems to have similar possibilities: https://confluence.atlassian.com/display/BITBUCKET/OAuth+on+Bitbucket I'm going to look into this tomorrow. Cheers, Sönke aszlig wrote: On Mon, Jan 13, 2014 at 01:40:55PM +0800, S??nke Hahn wrote: I also tried out https-urls. The problem is that not all repos that I need to access are public. So I have to provide some authentication. For ssh I can just put keys in .ssh, but for https in this setting, I'm not sure if it's even possible. If it's GitHub-specific and you want to have access to private repositories, please have a look at this: https://github.com/blog/1270-easier-builds-and-deployments-using-git-over-https-and-oauth We're using this internally for deployment and the following is the quite ugly approach we were using before GH introduced OAuth for HTTPS URLs: https://gist.github.com/aszlig/8397726 Hope this helps. a! ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] fetchgit with ssh
Hi,
I am trying to use fetchgit to package code that is available in git-repos.
I am using for example this expression:
pkgs.fetchgit {
url = g...@github.com:jekor/haskell-sscgi.git;
rev = 1547156d7199bdf40a837040a905d9b6b771dd26;
}
This does not work, the error message being:
error: cannot run ssh: No such file or directory
fatal: unable to fork
error: cannot run ssh: No such file or directory
fatal: unable to fork
Is this not supported by fetchgit? Is there anything I can do about it?
I tried to bring ssh in the PATH somehow, but I didn't manage. Can I include
the openssh package to make it visible to fetchgit?
I also tried out https-urls. The problem is that not all repos that I need
to access are public. So I have to provide some authentication. For ssh I
can just put keys in .ssh, but for https in this setting, I'm not sure if
it's even possible.
Any advice appreciated.
Cheers,
Sönke
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
I'm not positive this is the problem, but take a look at another
example. No quotes around the url and different format.
src = fetchgit {
url = git://git.gnupg.org/libgcrypt.git;
rev = 99b18aa53;
sha256 = 1rhbpxqrkfszlv8jvw8s4apwklal07k8zxv5q555l7binc1j1j3z;
};
On Mon, Jan 13, 2014 at 12:40 AM, Sönke Hahn soenkeh...@gmail.com wrote:
Hi,
I am trying to use fetchgit to package code that is available in git-repos.
I am using for example this expression:
pkgs.fetchgit {
url = g...@github.com:jekor/haskell-sscgi.git;
rev = 1547156d7199bdf40a837040a905d9b6b771dd26;
}
This does not work, the error message being:
error: cannot run ssh: No such file or directory
fatal: unable to fork
error: cannot run ssh: No such file or directory
fatal: unable to fork
Is this not supported by fetchgit? Is there anything I can do about it?
I tried to bring ssh in the PATH somehow, but I didn't manage. Can I include
the openssh package to make it visible to fetchgit?
I also tried out https-urls. The problem is that not all repos that I need
to access are public. So I have to provide some authentication. For ssh I
can just put keys in .ssh, but for https in this setting, I'm not sure if
it's even possible.
Any advice appreciated.
Cheers,
Sönke
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
Hi.
On 01/13/2014 07:52 AM, Thomas Bereknyei wrote:
I'm not positive this is the problem, but take a look at another
example. No quotes around the url and different format.
src = fetchgit {
url = git://git.gnupg.org/libgcrypt.git;
rev = 99b18aa53;
sha256 = 1rhbpxqrkfszlv8jvw8s4apwklal07k8zxv5q555l7binc1j1j3z;
};
I'm quite certain the quotes wouldn't break this case. Here you have
just plain git protocol without SSH.
The current implementation of fetchgit doesn't support SSH, judging from
a brief look at its code. (BTW, fetchgit runs in a separate derivation,
so it isn't affected by adding stuff to buildInputs, or even to $PATH.)
Maybe it's easy to add the support, I'll look again at it this evening
(I hope), if noone else solves it beforehand. The main problem is the
purity of ~/.ssh -- this directory just shouldn't be considered, as the
behavior inherently impure. Adding key/password to parameters of
fetchgit also doesn't seem very comfortable.
Vlada
smime.p7s
Description: S/MIME Cryptographic Signature
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit with ssh
A noteworthy workaround would be that you can just have a manual checkout
of your private repos and then do
src = ./path/to/repo;
Am 13.01.2014 08:49 schrieb Vladimír Čunát vcu...@gmail.com:
Hi.
On 01/13/2014 07:52 AM, Thomas Bereknyei wrote:
I'm not positive this is the problem, but take a look at another
example. No quotes around the url and different format.
src = fetchgit {
url = git://git.gnupg.org/libgcrypt.git;
rev = 99b18aa53;
sha256 = 1rhbpxqrkfszlv8jvw8s4apwklal07k8zxv5q555l7binc1j1j3z;
};
I'm quite certain the quotes wouldn't break this case. Here you have just
plain git protocol without SSH.
The current implementation of fetchgit doesn't support SSH, judging from a
brief look at its code. (BTW, fetchgit runs in a separate derivation, so it
isn't affected by adding stuff to buildInputs, or even to $PATH.)
Maybe it's easy to add the support, I'll look again at it this evening (I
hope), if noone else solves it beforehand. The main problem is the purity
of ~/.ssh -- this directory just shouldn't be considered, as the behavior
inherently impure. Adding key/password to parameters of fetchgit also
doesn't seem very comfortable.
Vlada
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Hi, On 19/11/12 07:11, Marc Weber wrote: Isn't it enough to depend on the git's hash value, No, because Nix's fixed-output derivation feature requires a md5/sha1/sha256 hash of the expected contents. -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Excerpts from Eelco Dolstra's message of Mon Nov 19 11:01:39 +0100 2012: No, because Nix's fixed-output derivation feature requires a md5/sha1/sha256 hash of the expected contents. I know what the current implementation requires. Just wondering whether this should be relaxed for git (like) VCS sources, because they naturally have a hash. I mean why run nix-prefetch git if using url and git commit hash could be enough? If you don't trust builders, fetching git sources is that common that it could even be built into the nix tool. My goal is to simplify installing packages from other sub universes such as ruby. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Hi, On 19/11/12 11:25, Marc Weber wrote: Excerpts from Eelco Dolstra's message of Mon Nov 19 11:01:39 +0100 2012: No, because Nix's fixed-output derivation feature requires a md5/sha1/sha256 hash of the expected contents. I know what the current implementation requires. Just wondering whether this should be relaxed for git (like) VCS sources, because they naturally have a hash. No. fetchgit won't work if it's not a fixed-output derivation, because it won't necessarily have network access (it might run in a chroot). -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Excerpts from Eelco Dolstra's message of Mon Nov 19 11:36:00 +0100 2012: No. fetchgit won't work if it's not a fixed-output derivation, because it won't necessarily have network access (it might run in a chroot). Again: I'm not talking about the current state. I'm aware about how it works. I'm talking about: Does it make sense to introduce a special fixed hash for git repos or what about implementing git checkouts natively so that passing the git's hash is enough? git sources are very common today. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
- Original message - Excerpts from Eelco Dolstra's message of Mon Nov 19 11:36:00 +0100 2012: No. fetchgit won't work if it's not a fixed-output derivation, because it won't necessarily have network access (it might run in a chroot). Again: I'm not talking about the current state. I'm aware about how it works. I'm talking about: Does it make sense to introduce a special fixed hash for git repos or what about implementing git checkouts natively so that passing the git's hash is enough? git sources are very common today. True, simplification sounds like a good idea. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Is it terribly difficult to run nix-prefetch-git? Built-in vcs-specific support doesn't strike me as simplification. On Nov 19, 2012, at 7:10 AM, Joachim Schiele j...@lastlog.de wrote: - Original message - Excerpts from Eelco Dolstra's message of Mon Nov 19 11:36:00 +0100 2012: No. fetchgit won't work if it's not a fixed-output derivation, because it won't necessarily have network access (it might run in a chroot). Again: I'm not talking about the current state. I'm aware about how it works. I'm talking about: Does it make sense to introduce a special fixed hash for git repos or what about implementing git checkouts natively so that passing the git's hash is enough? git sources are very common today. True, simplification sounds like a good idea. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Could fetchgit handle that on its own though? Also, at least for github, if you want to install a specific tag, which isn't always the case, you can link to the .zip copy of it from the github page. /M Shea Levy s...@shealevy.com writes: Is it terribly difficult to run nix-prefetch-git? Built-in vcs-specific support doesn't strike me as simplification. On Nov 19, 2012, at 7:10 AM, Joachim Schiele j...@lastlog.de wrote: - Original message - Excerpts from Eelco Dolstra's message of Mon Nov 19 11:36:00 +0100 2012: No. fetchgit won't work if it's not a fixed-output derivation, because it won't necessarily have network access (it might run in a chroot). Again: I'm not talking about the current state. I'm aware about how it works. I'm talking about: Does it make sense to introduce a special fixed hash for git repos or what about implementing git checkouts natively so that passing the git's hash is enough? git sources are very common today. True, simplification sounds like a good idea. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Excerpts from Shea Levy's message of Mon Nov 19 13:38:37 +0100 2012: Is it terribly difficult to run nix-prefetch-git? YES: I'm talking about such configurations: http://gembundler.com/ And here you have git repo and hash. Trying to semi automatically package such things requires much overhead if you have to prefetch everything to get a sha256 hash. I'm not talking about the one project you do package once in a year. I'm talking about 20 small ruby gem packages you need to get some bleeding edge code working. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Marc Weber marco-owe...@gmx.de writes: Excerpts from Shea Levy's message of Mon Nov 19 13:38:37 +0100 2012: Is it terribly difficult to run nix-prefetch-git? YES: I'm talking about such configurations: http://gembundler.com/ And here you have git repo and hash. Trying to semi automatically package such things requires much overhead if you have to prefetch everything to get a sha256 hash. I'm not talking about the one project you do package once in a year. I'm talking about 20 small ruby gem packages you need to get some bleeding edge code working. Have you looked at Shea's npm2nix utility for node.js packages? It's really not that big/scary. Just give it the name of an npm package and it outputs a nix expression (including sha256) for that package, including its dependencies. A similar solution for rubygems would probably not be too hard. As rubygems itself is written in ruby, you can probably plug in to its dependency resolution and downloading capabilities so you can focus on generating the sha256 and the nix expression. Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
A similar solution for rubygems would probably not be too hard. As rubygems itself is written in ruby, you can probably plug in to its dependency resolution and downloading capabilities so you can focus on generating the sha256 and the nix expression. If you still haven't got it: I worte nixpkgs-ruby-overlay which already does it. I also wrote hack-nix packaging all hackage - and I did so after having disregarder a 80% working attempt doing it the nodejs way. I'm looking for packaging fast changing dev versions of packages. And then I don't want to wait for any double fetches. I want to give code a try. I know what I want and why. I accept that the nix community eventually things differently about this. So this may just end up being another patch in my github repos. Maybe I have to use standard ubuntu distribution cause cause I may not have time to finish all this in time (yet) Marc Weber ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Excerpts from Eelco Dolstra's message of Mon Nov 19 16:31:26 +0100 2012:
Why would you need a double fetch? After running fetchgit, the Git tree is
in
the Nix store and shouldn't be downloaded again unless you do a garbage
collect
in between.
You're right about this.
I want to make bundler (which dynamically fetches updates for
dependencies of ruby packages) use the nix store to share git sources
and gem install results.
nixpkgs-ruby-overlay gets the job done, and I could manually package all
git sources additionally to the packages found on rubyforge. It just
takes too long.
I want to work like other ruby using people do:
bundle update (fetch all dependencies, and if this was done previously reuse
store paths)
Of course running nix-prefetch-git is an option, however checking
whether a store path representing { url = ..; hash = .. } already exists
is harder. If you run nix-prefetch-git twice it will fetch twice
(waste). I haven't looked for options.
If nix could handle this, I could just create a .nix file and I'd always
get what I want: the source - if it exists I would not have to bother at
all.
About changeroot builds: You're right. So mabye a hacky
mkDerivation {
allownetwork = true;
}
would do. It could be used for such cases. Why should it be allowed?
If a programmer wants to shoot himself into the food, you can't prevent
him doing so. Thus the goal should be making it hard to do it by
accident. And this property still holds if allownetwork = true or such
existed.
So comment on whether you see huge security risks using git url and
git's hash only.
Also mind that I don't say that sha256 checks for fetchgit should no
longer be used. I just think its not worth bothering for use cases where
other tools neither do (such as bundler for ruby) - they don't even
bother to use the full git hash length (which is bad IMHO).
Marc Weber
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
Of course running nix-prefetch-git is an option, however checking
whether a store path representing { url = ..; hash = .. } already exists
is harder. If you run nix-prefetch-git twice it will fetch twice
(waste). I haven't looked for options.
nix-store --check-validity $(nix-store -q --outputs $(nix-instantiate
expression.nix -A src))
?
Also, I do use fresh checkouts as src for various Nix expressions. I
just added a repository set to chroot-accessible locations and do what
you say (telling only git hashes to Nix).
If nix could handle this, I could just create a .nix file and I'd always
get what I want: the source - if it exists I would not have to bother at
all.
So comment on whether you see huge security risks using git url and
git's hash only.
It is not so much security risks as it is about special case being a
separate source of bugs.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] fetchgit - why sha256 protection?
Isn't it enough to depend on the git's hash value, eg
fetchgit { git_hash = xxx; url = yyy; }
Is compromising a git repository (even using shallow clones) that much
easier than compromising a .tar.* file protected by sha256?
In anyway you have to find a hash collision.
A lot of foreign tools (eg bundler for ruby) just store the url and the
hash. Packaging github like projects would be a lot easier if passing a
hash would be enough.
Marc Weber
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
On Sun, Nov 18, 2012 at 10:11 PM, Marc Weber marco-owe...@gmx.de wrote:
Isn't it enough to depend on the git's hash value, eg
fetchgit { git_hash = xxx; url = yyy; }
Is compromising a git repository (even using shallow clones) that much
easier than compromising a .tar.* file protected by sha256?
That would be better because there is no trivial way to check the
sha256 when making the Nix expression.
How does git distinguish a branchnamed after a revision?
--
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] fetchgit - why sha256 protection?
On Sun, Nov 18, 2012 at 10:24 PM, Nicolas Pierron
nicolas.b.pier...@gmail.com wrote:
On Sun, Nov 18, 2012 at 10:11 PM, Marc Weber marco-owe...@gmx.de wrote:
Isn't it enough to depend on the git's hash value, eg
fetchgit { git_hash = xxx; url = yyy; }
Is compromising a git repository (even using shallow clones) that much
easier than compromising a .tar.* file protected by sha256?
That would be better because there is no trivial way to check the
sha256 when making the Nix expression.
How does git distinguish a branchnamed after a revision?
We should also enforce that provided hashes have all digits, to
prevent easier attack.
--
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] fetchgit vs tarball creation, privately maintained infrastructure
I messed up Subject and Cc m( Now, with Marc's permission publicly. My original mail: Hi Marc, there are currently several expressions pointing to your server resulting in 404s. Why do you prefer to put sources on your private server instead of public infrastructure? Maybe we can keep the benefits and still profit from assumed higher availability of public infrastructure. regards florian On Fri, 23 Sep 2011 23:19:55 +0200, Marc Weber marco-owe...@gmx.de wrote: (..) I haven't had time to recover. Tell me the target packages and I'll upload everything required to build them. see the attached file Why don't I use fetchgit etc? Cause its too slow. My internet is slow. I need incremental updates. That's why I wrote nix-repository-manager to serve my needs. To solve that I would rather teach fetchgit to keep a cache of its repositories, instead of creating tarballs. Many things I install via fetchgit, I anyway want to have on my laptop for development. Having one common cache from which fetchgit for nix and you for development can clone. Anyway, I think nix expressions should not depend on anybody's private infrastructure, but instead use public infrastructure. Why do you prefer to put sources on your private server instead of public infrastructure? Maybe we can keep the benefits and still profit from assumed higher availability of public infrastructure. I'm planing to put my stuff on amazon. I can't afford huge build farms. But being able to launch some nice CPU powered instances on Amazon will allow to me provide binaries for my branches. I don't think its necessary that you provide your own hosting infrastructure for that. Currently my tool supports pushing tars by SSH. It should be doable to make it cope with any public infrastructure. Which one do you suggest? I would not like to see packages use that approach without agreement that we in nix are generating tarballs for things available via git/svn/... I think the approach of caching is more fruitful. If we agree to create tarballs instead of fetching git, there is some good reason for it and those tarballs should be hosted on nix infrastructure, i.e. making it a solution used by every package maintainer. If you want to join tell me. A lot of my patches did never get any review. And committing them without ok was no good according to ludo either. That's why I had to fork (also because SVN is too slow for me). I'd happy to review isolated patches implementing functionality I'm interested in. mawercer.de Description: Binary data -- Florian Friesdorf f...@chaoflow.net GPG FPR: 7A13 5EEE 1421 9FC2 108D BAAF 38F8 99A3 0C45 F083 Jabber/XMPP: f...@chaoflow.net IRC: chaoflow on freenode,ircnet,blafasel,OFTC pgpXdp2Arf1CJ.pgp Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
