Re: [Nix-dev] hydra's nix-expression works in nix-shell, but fails with nix-build
Have you tried it with `nix-shell --pure`? -- Кирилл Елагин On Sun, Jun 15, 2014 at 5:01 AM, Thomas Strobel wrote: > Hi! > > I'm trying to write a nix-expression for current hydra, see below. It > works within nix-shell, but fails with nix-build, both on 14.04. > Could someone maybe help me with that, please? > > Many thanks, > Thomas > > > > hydra.nix > > > { stdenv, fetchgit, pkgs }: > with pkgs; > let > nixpkgs = pkgs; > version = "0.17"; > > nix = pkgs.nixUnstable; > > perlDeps = pkgs.buildEnv { > name = "hydra-perl-deps"; > paths = with perlPackages; > [ ModulePluggable > CatalystAuthenticationStoreDBIxClass > CatalystDispatchTypeRegex > CatalystPluginAccessLog > CatalystPluginAuthorizationRoles > CatalystPluginCaptcha > CatalystPluginSessionStateCookie > CatalystPluginSessionStoreFastMmap > CatalystPluginStackTrace > CatalystPluginUnicodeEncoding > CatalystTraitForRequestProxyBase > CatalystViewDownload > CatalystViewJSON > CatalystViewTT > CatalystXScriptServerStarman > CatalystActionREST > CryptRandPasswd > DBDPg > DBDSQLite > DataDump > DateTime > DigestSHA1 > EmailSender > FileSlurp > LWP > LWPProtocolHttps > IOCompress > IPCRun > JSONXS > PadWalker > CatalystDevel > Readonly > SetScalar > SQLSplitStatement > Starman > SysHostnameLong > TestMore > TextDiff > TextTable > XMLSimple > NetAmazonS3 > nix git > ]; > }; > in > > stdenv.mkDerivation rec { > name = "hydra-${version}"; > > src = fetchgit { > url = "https://github.com/NixOS/hydra.git";; > rev = "91f895b3d660560e40278b1c4e8c50172f3ac065"; > sha256 = > "71954b6a3f6037b75698f6c7262cf0f97442bb5af8c5d31e543479ae8eee8f7a"; > }; > > > preConfigure = '' > # TeX needs a writable font cache. > export VARTEXFONTS=$TMPDIR/texfonts > > addToSearchPath PATH $(pwd)/src/script > addToSearchPath PATH $(pwd)/src/c > addToSearchPath PERL5LIB $(pwd)/src/lib > > ./bootstrap > ''; > > configureFlags = ["--with-nix=${nix}" > "--with-docbook-xsl=${docbook_xsl}/xml/xsl/docbook" ]; > > buildInputs = > [ makeWrapper libtool unzip nukeReferences pkgconfig boehmgc sqlite > gitAndTools.topGit mercurial darcs subversion bazaar openssl bzip2 > guile perlDeps perl > autoconf automake libtool libxslt dblatex tetex > ]; > > hydraPath = lib.makeSearchPath "bin" ( > [ libxslt sqlite subversion openssh nix coreutils findutils > gzip bzip2 lzma gnutar unzip git gitAndTools.topGit mercurial > darcs gnused graphviz bazaar > ] ++ lib.optionals stdenv.isLinux [ rpm dpkg cdrkit ] ); > > preCheck = '' > patchShebangs . > export LOGNAME=${LOGNAME:-foo} > ''; > > postInstall = '' > mkdir -p $out/nix-support > nuke-refs $out/share/doc/hydra/manual/manual.pdf > > for i in $out/bin/*; do > wrapProgram $i \ > --prefix PERL5LIB ':' $out/libexec/hydra/lib:$PERL5LIB \ > --prefix PATH ':' $out/bin:$hydraPath \ > --set HYDRA_RELEASE ${version} \ > --set HYDRA_HOME $out/libexec/hydra \ > --set NIX_RELEASE ${nix.name} > done > ''; # */ > > meta.description = "Build of Hydra on ${system}"; > passthru.perlDeps = perlDeps; > } > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] hydra's nix-expression works in nix-shell, but fails with nix-build
Hi! I'm trying to write a nix-expression for current hydra, see below. It works within nix-shell, but fails with nix-build, both on 14.04. Could someone maybe help me with that, please? Many thanks, Thomas hydra.nix { stdenv, fetchgit, pkgs }: with pkgs; let nixpkgs = pkgs; version = "0.17"; nix = pkgs.nixUnstable; perlDeps = pkgs.buildEnv { name = "hydra-perl-deps"; paths = with perlPackages; [ ModulePluggable CatalystAuthenticationStoreDBIxClass CatalystDispatchTypeRegex CatalystPluginAccessLog CatalystPluginAuthorizationRoles CatalystPluginCaptcha CatalystPluginSessionStateCookie CatalystPluginSessionStoreFastMmap CatalystPluginStackTrace CatalystPluginUnicodeEncoding CatalystTraitForRequestProxyBase CatalystViewDownload CatalystViewJSON CatalystViewTT CatalystXScriptServerStarman CatalystActionREST CryptRandPasswd DBDPg DBDSQLite DataDump DateTime DigestSHA1 EmailSender FileSlurp LWP LWPProtocolHttps IOCompress IPCRun JSONXS PadWalker CatalystDevel Readonly SetScalar SQLSplitStatement Starman SysHostnameLong TestMore TextDiff TextTable XMLSimple NetAmazonS3 nix git ]; }; in stdenv.mkDerivation rec { name = "hydra-${version}"; src = fetchgit { url = "https://github.com/NixOS/hydra.git";; rev = "91f895b3d660560e40278b1c4e8c50172f3ac065"; sha256 = "71954b6a3f6037b75698f6c7262cf0f97442bb5af8c5d31e543479ae8eee8f7a"; }; preConfigure = '' # TeX needs a writable font cache. export VARTEXFONTS=$TMPDIR/texfonts addToSearchPath PATH $(pwd)/src/script addToSearchPath PATH $(pwd)/src/c addToSearchPath PERL5LIB $(pwd)/src/lib ./bootstrap ''; configureFlags = ["--with-nix=${nix}" "--with-docbook-xsl=${docbook_xsl}/xml/xsl/docbook" ]; buildInputs = [ makeWrapper libtool unzip nukeReferences pkgconfig boehmgc sqlite gitAndTools.topGit mercurial darcs subversion bazaar openssl bzip2 guile perlDeps perl autoconf automake libtool libxslt dblatex tetex ]; hydraPath = lib.makeSearchPath "bin" ( [ libxslt sqlite subversion openssh nix coreutils findutils gzip bzip2 lzma gnutar unzip git gitAndTools.topGit mercurial darcs gnused graphviz bazaar ] ++ lib.optionals stdenv.isLinux [ rpm dpkg cdrkit ] ); preCheck = '' patchShebangs . export LOGNAME=${LOGNAME:-foo} ''; postInstall = '' mkdir -p $out/nix-support nuke-refs $out/share/doc/hydra/manual/manual.pdf for i in $out/bin/*; do wrapProgram $i \ --prefix PERL5LIB ':' $out/libexec/hydra/lib:$PERL5LIB \ --prefix PATH ':' $out/bin:$hydraPath \ --set HYDRA_RELEASE ${version} \ --set HYDRA_HOME $out/libexec/hydra \ --set NIX_RELEASE ${nix.name} done ''; # */ meta.description = "Build of Hydra on ${system}"; passthru.perlDeps = perlDeps; } ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] pytables, numexpr, and pandas
On 06/14/2014 11:17 PM, Andreas Herrmann wrote: > Hi, > > I've created two new nix-expressions, that I need to use the Pandas package. Great. Disclaimer: I'm not a person ‘in power’ of nixpkgs and simply speak from experience. > * PyTables is a python library for HDF5 storage. It's a different approach > than h5py in that it is not a pythonic wrapper of the C library, but rather > implements relational db alike features. Pandas depends on it to read/write > HDF5 files. > http://www.pytables.org/moin > > * numexpr speeds up numpy ndarray expressions. PyTables depends on it. > https://github.com/pydata/numexpr > > However, I have a few questions before I go ahead and send a pull-request. > > A) Is there some kind of test suite that I need to run on the expressions > before I can send a pull request? (I can build and use them...) Simply make sure it builds and runs on top of the current master. For Python, check multiple Python versions. nix-build can help you build the packages. > B) Which branch should I send the pull-request to? Is master fine? master is fine. > C1) Both packages have a test-suite. However, it's not integrated into ` > setup.py`. I managed to get it running within the nix build process anyway. > The corresponding `checkPhase` is attached below. Is that an acceptable way > of doing this, or should I just set `doCheck = false;`? If it has a test-suite then I see no reason not to include it. Make sure that when tests actually do fail, this is caught as a failure rather than accepted anyway. Make a test fail and see if the whole thing does. > C2) The test-suite of pytables takes quite some time and has a number of > failures of the "not-implemented" type. Is it worth including that > test-suite or should I just leave that one out? I would include any steps necessary to run the tests but after ensuring that the failures aren't NixOS/nix specific, I'd set doCheck = false: this makes it easier for you or someone else to turn on the tests on any future, fixed versions. Make sure to let the maintainer know that there are failing tests. > D1) Pandas works without pytables. However, without it, it cannot > import/export hdf5 archives. Should I add PyTables as a dependency to > Pandas, or should every user individually add both to his environment if he > needs both? Or is there a way to make it an optional dependency? Optional dependencies are possible but I am unsure how they are done for Python packages. While I don't know how to specify such optional dependencies in python-packages.nix, you can simply create a nix-expression outside of it. See pkgs/games/anki/default.nix for an example of a Python package with optional dependencies. Provide a sensible default because only the default way will be built by Hydra. If you need Hydra to build the package in more ways then you should make two separate entries in all-packages.nix with the different flag setups but if there's only one common case then one should suffice. > D2) The latest release of Pandas is 0.14.0, the nix-pkgs version is 0.12.0. > Should I use this opportunity to bump that version number, or should this > go through a different channel? I would advise you to bump the Pandas version while you're working around it. > I've read the contributing guides on the wiki (at least those that I could > find). I learned a lot from them, but the above questions remained. > > Best, > > Andreas > > > The numexpr `checkPhase` within a call to `buildPythonPackage`: > -- > > # Run the test suite. > # It requires the build path to be in the python module search path. > checkPhase = '' > ${python}/bin/${python.executable} < import sysconfig > import sys > import os > f = "lib.{platform}-{version[0]}.{version[1]}" > lib = f.format(platform=sysconfig.get_platform(), > version=sys.version_info) > build = os.path.join(os.getcwd(), 'build', lib) > sys.path.insert(0, build) > import numexpr > numexpr.test() > EOF > ''; > > > > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > I hope that somehow helps. If anything is incorrect, I'm sure someone will correct me. -- Mateusz K. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
>> I think that any proposal would be seriously considered. Any already >> existing application creating such a problem would increase your chances >> to make someone else propose a solution… Otherwise — I think the problem >> can surface, but I have no idea ho to fix it sanely… I think many other >> people think the same. > >Well, let's not try to decide the undecidable. Would it be so bad to >have explicit dependencies? That seems to be the only sound solution. Nobody would use it… except in the cases where there is no other solution. I think it is a nice solution. >A straightforward way is to add a small dotfile via the builder script >with paths to the output. The only problem at this point is that there >is pretty much no support for doing that, so it's error-prone. I could >totally see something like this happening: > >stdenv.mkDerivation { >... >requisites = [cacert dataFiles1 dataFiles2]; >... >} > >All it does is to register the given derivations as requisites >unconditionally, perhaps simply by writing the aforementioned dotfile >at $out/.nixdeps. I think it should be $out/nix-support/forced-runtime-deps What do other people think? Is it something the should be added to fixupPhase in stdenv or is it worth adding in a slightly more complicated way without a rebuild? (Yes, situations like this are exactly why I think composing the entire builder in Nix has its merits) ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] pytables, numexpr, and pandas
Hi, I've created two new nix-expressions, that I need to use the Pandas package. * PyTables is a python library for HDF5 storage. It's a different approach than h5py in that it is not a pythonic wrapper of the C library, but rather implements relational db alike features. Pandas depends on it to read/write HDF5 files. http://www.pytables.org/moin * numexpr speeds up numpy ndarray expressions. PyTables depends on it. https://github.com/pydata/numexpr However, I have a few questions before I go ahead and send a pull-request. A) Is there some kind of test suite that I need to run on the expressions before I can send a pull request? (I can build and use them...) B) Which branch should I send the pull-request to? Is master fine? C1) Both packages have a test-suite. However, it's not integrated into ` setup.py`. I managed to get it running within the nix build process anyway. The corresponding `checkPhase` is attached below. Is that an acceptable way of doing this, or should I just set `doCheck = false;`? C2) The test-suite of pytables takes quite some time and has a number of failures of the "not-implemented" type. Is it worth including that test-suite or should I just leave that one out? D1) Pandas works without pytables. However, without it, it cannot import/export hdf5 archives. Should I add PyTables as a dependency to Pandas, or should every user individually add both to his environment if he needs both? Or is there a way to make it an optional dependency? D2) The latest release of Pandas is 0.14.0, the nix-pkgs version is 0.12.0. Should I use this opportunity to bump that version number, or should this go through a different channel? I've read the contributing guides on the wiki (at least those that I could find). I learned a lot from them, but the above questions remained. Best, Andreas The numexpr `checkPhase` within a call to `buildPythonPackage`: -- # Run the test suite. # It requires the build path to be in the python module search path. checkPhase = '' ${python}/bin/${python.executable} <___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Sat, 14 Jun 2014 22:09:21 +0400 Michael Raskin <7c6f4...@mail.ru> wrote: > >It's probably not a big deal. On the other hand consider that it only > >happens in a rather special scenario and may go unnoticed for quite > >some time. > > > >Consider an application (like a game) with a bunch of separate sets of > >data files, each optional and referring to each other, compressed to > >save space. This is an actual design that I'm following in one of my > >projects. I'll know to make it Nix-friendly, but non-NixOS developers > >may fail to do so. > > I think that any proposal would be seriously considered. Any already > existing application creating such a problem would increase your chances > to make someone else propose a solution… Otherwise — I think the problem > can surface, but I have no idea ho to fix it sanely… I think many other > people think the same. Well, let's not try to decide the undecidable. Would it be so bad to have explicit dependencies? That seems to be the only sound solution. A straightforward way is to add a small dotfile via the builder script with paths to the output. The only problem at this point is that there is pretty much no support for doing that, so it's error-prone. I could totally see something like this happening: stdenv.mkDerivation { ... requisites = [cacert dataFiles1 dataFiles2]; ... } All it does is to register the given derivations as requisites unconditionally, perhaps simply by writing the aforementioned dotfile at $out/.nixdeps. Greets, Ertugrul -- Ertugrul Söylemez ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
>It's probably not a big deal. On the other hand consider that it only >happens in a rather special scenario and may go unnoticed for quite >some time. > >Consider an application (like a game) with a bunch of separate sets of >data files, each optional and referring to each other, compressed to >save space. This is an actual design that I'm following in one of my >projects. I'll know to make it Nix-friendly, but non-NixOS developers >may fail to do so. I think that any proposal would be seriously considered. Any already existing application creating such a problem would increase your chances to make someone else propose a solution… Otherwise — I think the problem can surface, but I have no idea ho to fix it sanely… I think many other people think the same. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Sat, 14 Jun 2014 13:43:24 -0400 Shea Levy wrote: > > I can't. I'm just pointing out a potential problem here. If you > > disagree, I'll be happy to construct a proof of concept. > > Of course it could happen, but we have a huge number of packages in nixpkgs, > including a significant subset of hackage, and it hasn't actually happened > yet as far as anyone knows. > > No one claims hash scanning/hash rewriting is 100% sound. It is, however, > extremely convenient and has worked remarkably well up until now. If we start > finding real world use cases where it fails, then we can start thinking about > alternatives, but until then why is this a big deal? It's probably not a big deal. On the other hand consider that it only happens in a rather special scenario and may go unnoticed for quite some time. Consider an application (like a game) with a bunch of separate sets of data files, each optional and referring to each other, compressed to save space. This is an actual design that I'm following in one of my projects. I'll know to make it Nix-friendly, but non-NixOS developers may fail to do so. Greets, Ertugrul -- Ertugrul Söylemez ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
> On Jun 14, 2014, at 13:29, Ertugrul Söylemez wrote: > > On Sat, 14 Jun 2014 09:36:21 -0400 > Shea Levy wrote: > >> Please point to a specific example of a package with an unretained >> dependency due to UTF-16 storage > > I can't. I'm just pointing out a potential problem here. If you > disagree, I'll be happy to construct a proof of concept. > Of course it could happen, but we have a huge number of packages in nixpkgs, including a significant subset of hackage, and it hasn't actually happened yet as far as anyone knows. No one claims hash scanning/hash rewriting is 100% sound. It is, however, extremely convenient and has worked remarkably well up until now. If we start finding real world use cases where it fails, then we can start thinking about alternatives, but until then why is this a big deal? > > Greets, > Ertugrul > > -- > Ertugrul Söylemez > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Sat, 14 Jun 2014 09:36:21 -0400 Shea Levy wrote: > Please point to a specific example of a package with an unretained dependency > due to UTF-16 storage I can't. I'm just pointing out a potential problem here. If you disagree, I'll be happy to construct a proof of concept. Greets, Ertugrul -- Ertugrul Söylemez ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
> On Jun 14, 2014, at 4:28, Ertugrul Söylemez wrote: > > On Fri, 13 Jun 2014 18:54:53 +0200 > Eelco Dolstra wrote: > >>> The path-rewriting proposal is a very bad idea and will cause a lot of >>> breakage. For many/enough applications rewriting will not work at all, >>> because they might encode paths in data structures or be using a >>> non-UTF8 multi-byte encoding. >> >> Packages that store paths in UTF-16 don't work with Nix anyway, independent >> from >> hash rewriting, because Nix finds runtime dependencies by scanning for plain >> ASCII hashes. To my knowledge we've never encountered such a package in >> Nixpkgs. >> >> This paper has a small evaluation of hash rewriting: >> http://nixos.org/~eelco/pubs/secsharing-ase2005-final.pdf (section 6.1) > > We certainly have. Please point to a specific example of a package with an unretained dependency due to UTF-16 storage > As said, Haskell works that way: while hardcoded > paths are commonly stored in UTF-8, serialized text is more likely in > UTF-16, because that's the internal format of the Text type. Also > continuous path strings are very uncommon, because we use > platform-independent combinators to construct paths. > > Nix' dependency scanning still works most of the time, because paths to > other Haskell packages are stored at least in a GHC registration file as > continuous strings. It could fail for other requisites like data files > from non-Haskell packages. For example the CA certificates are stored > in their own derivation. If not half of the system would depend on that > derivation, it would most likely be GCed away and Haskell programs > would be unable to find them. > > Rewriting is even worse, because now you are interpreting a > structurally rich file as a mere byte-string and then changing things > based on vague assumptions. This is dangerous, in particular for ELF > files, which are in fact Turing-complete (ELF itself, not the code it > contains). Just as a theoretical point consider that the question > whether your rewriting algorithm is correct is undecidable in general. > > More practically you would break half of Hackage. The breakage would > range from being as gentle as a Haskell program failing to find > certificate files to as severe as a Haskell plugin-based web > application (like clckwrks) fatal-error-crashing. > > As a Haskell developer I must warn you about rewriting. This /will/ > cause trouble at least for Haskell. Even the current dependency > scanning mechanism is a ticking time bomb for Haskell users, though it > did not yet explode for me. > > >> Or even better, ask the admin to use pam_namespace: >> >> http://www.linux-pam.org/Linux-PAM-html/sag-pam_namespace.html >> >> And hopefully, one day users won't need to be root to do bind mounts. > > That would be very useful, yeah. It's probably already possible anyway > given the right capability, but I don't know. > > > Greets, > Ertugrul > > -- > Ertugrul Söylemez > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Binary packages for custom store path installs
On Fri, 13 Jun 2014 18:54:53 +0200 Eelco Dolstra wrote: > > The path-rewriting proposal is a very bad idea and will cause a lot of > > breakage. For many/enough applications rewriting will not work at all, > > because they might encode paths in data structures or be using a > > non-UTF8 multi-byte encoding. > > Packages that store paths in UTF-16 don't work with Nix anyway, independent > from > hash rewriting, because Nix finds runtime dependencies by scanning for plain > ASCII hashes. To my knowledge we've never encountered such a package in > Nixpkgs. > > This paper has a small evaluation of hash rewriting: > http://nixos.org/~eelco/pubs/secsharing-ase2005-final.pdf (section 6.1) We certainly have. As said, Haskell works that way: while hardcoded paths are commonly stored in UTF-8, serialized text is more likely in UTF-16, because that's the internal format of the Text type. Also continuous path strings are very uncommon, because we use platform-independent combinators to construct paths. Nix' dependency scanning still works most of the time, because paths to other Haskell packages are stored at least in a GHC registration file as continuous strings. It could fail for other requisites like data files from non-Haskell packages. For example the CA certificates are stored in their own derivation. If not half of the system would depend on that derivation, it would most likely be GCed away and Haskell programs would be unable to find them. Rewriting is even worse, because now you are interpreting a structurally rich file as a mere byte-string and then changing things based on vague assumptions. This is dangerous, in particular for ELF files, which are in fact Turing-complete (ELF itself, not the code it contains). Just as a theoretical point consider that the question whether your rewriting algorithm is correct is undecidable in general. More practically you would break half of Hackage. The breakage would range from being as gentle as a Haskell program failing to find certificate files to as severe as a Haskell plugin-based web application (like clckwrks) fatal-error-crashing. As a Haskell developer I must warn you about rewriting. This /will/ cause trouble at least for Haskell. Even the current dependency scanning mechanism is a ticking time bomb for Haskell users, though it did not yet explode for me. > Or even better, ask the admin to use pam_namespace: > > http://www.linux-pam.org/Linux-PAM-html/sag-pam_namespace.html > > And hopefully, one day users won't need to be root to do bind mounts. That would be very useful, yeah. It's probably already possible anyway given the right capability, but I don't know. Greets, Ertugrul -- Ertugrul Söylemez ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev