Re: [Nix-dev] hydra's nix-expression works in nix-shell, but fails with nix-build

2014-06-14 Thread Kirill Elagin
Have you tried it with `nix-shell --pure`?


--
Кирилл Елагин


On Sun, Jun 15, 2014 at 5:01 AM, Thomas Strobel  wrote:

> Hi!
>
> I'm trying to write a nix-expression for current hydra, see below. It
> works within nix-shell, but fails with nix-build, both on 14.04.
> Could someone maybe help me with that, please?
>
> Many thanks,
> Thomas
>
>
>
> hydra.nix
> 
>
> { stdenv, fetchgit, pkgs }:
> with pkgs;
> let
>   nixpkgs = pkgs;
>   version = "0.17";
>
>   nix = pkgs.nixUnstable;
>
>   perlDeps = pkgs.buildEnv {
> name = "hydra-perl-deps";
> paths = with perlPackages;
>   [ ModulePluggable
> CatalystAuthenticationStoreDBIxClass
> CatalystDispatchTypeRegex
> CatalystPluginAccessLog
> CatalystPluginAuthorizationRoles
> CatalystPluginCaptcha
> CatalystPluginSessionStateCookie
> CatalystPluginSessionStoreFastMmap
> CatalystPluginStackTrace
> CatalystPluginUnicodeEncoding
> CatalystTraitForRequestProxyBase
> CatalystViewDownload
> CatalystViewJSON
> CatalystViewTT
> CatalystXScriptServerStarman
> CatalystActionREST
> CryptRandPasswd
> DBDPg
> DBDSQLite
> DataDump
> DateTime
> DigestSHA1
> EmailSender
> FileSlurp
> LWP
> LWPProtocolHttps
> IOCompress
> IPCRun
> JSONXS
> PadWalker
> CatalystDevel
> Readonly
> SetScalar
> SQLSplitStatement
> Starman
> SysHostnameLong
> TestMore
> TextDiff
> TextTable
> XMLSimple
> NetAmazonS3
> nix git
>   ];
>   };
> in
>
> stdenv.mkDerivation rec {
>   name = "hydra-${version}";
>
>   src = fetchgit {
> url = "https://github.com/NixOS/hydra.git";;
> rev = "91f895b3d660560e40278b1c4e8c50172f3ac065";
> sha256 =
> "71954b6a3f6037b75698f6c7262cf0f97442bb5af8c5d31e543479ae8eee8f7a";
>   };
>
>
>   preConfigure = ''
> # TeX needs a writable font cache.
> export VARTEXFONTS=$TMPDIR/texfonts
>
> addToSearchPath PATH $(pwd)/src/script
> addToSearchPath PATH $(pwd)/src/c
> addToSearchPath PERL5LIB $(pwd)/src/lib
>
> ./bootstrap
> '';
>
>   configureFlags = ["--with-nix=${nix}"
> "--with-docbook-xsl=${docbook_xsl}/xml/xsl/docbook" ];
>
>   buildInputs =
> [ makeWrapper libtool unzip nukeReferences pkgconfig boehmgc sqlite
>   gitAndTools.topGit mercurial darcs subversion bazaar openssl bzip2
>   guile perlDeps perl
>   autoconf automake libtool libxslt dblatex tetex
> ];
>
>   hydraPath = lib.makeSearchPath "bin" (
> [ libxslt sqlite subversion openssh nix coreutils findutils
>   gzip bzip2 lzma gnutar unzip git gitAndTools.topGit mercurial
> darcs gnused graphviz bazaar
> ] ++ lib.optionals stdenv.isLinux [ rpm dpkg cdrkit ] );
>
>   preCheck = ''
> patchShebangs .
> export LOGNAME=${LOGNAME:-foo}
>   '';
>
>   postInstall = ''
> mkdir -p $out/nix-support
> nuke-refs $out/share/doc/hydra/manual/manual.pdf
>
> for i in $out/bin/*; do
> wrapProgram $i \
> --prefix PERL5LIB ':' $out/libexec/hydra/lib:$PERL5LIB \
> --prefix PATH ':' $out/bin:$hydraPath \
> --set HYDRA_RELEASE ${version} \
> --set HYDRA_HOME $out/libexec/hydra \
> --set NIX_RELEASE ${nix.name}
> done
>   ''; # */
>
>   meta.description = "Build of Hydra on ${system}";
>   passthru.perlDeps = perlDeps;
> }
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


[Nix-dev] hydra's nix-expression works in nix-shell, but fails with nix-build

2014-06-14 Thread Thomas Strobel
Hi!

I'm trying to write a nix-expression for current hydra, see below. It
works within nix-shell, but fails with nix-build, both on 14.04.
Could someone maybe help me with that, please?

Many thanks,
Thomas



hydra.nix


{ stdenv, fetchgit, pkgs }:
with pkgs;
let
  nixpkgs = pkgs;
  version = "0.17";

  nix = pkgs.nixUnstable;

  perlDeps = pkgs.buildEnv {
name = "hydra-perl-deps";
paths = with perlPackages;
  [ ModulePluggable
CatalystAuthenticationStoreDBIxClass
CatalystDispatchTypeRegex
CatalystPluginAccessLog
CatalystPluginAuthorizationRoles
CatalystPluginCaptcha
CatalystPluginSessionStateCookie
CatalystPluginSessionStoreFastMmap
CatalystPluginStackTrace
CatalystPluginUnicodeEncoding
CatalystTraitForRequestProxyBase
CatalystViewDownload
CatalystViewJSON
CatalystViewTT
CatalystXScriptServerStarman
CatalystActionREST
CryptRandPasswd
DBDPg
DBDSQLite
DataDump
DateTime
DigestSHA1
EmailSender
FileSlurp
LWP
LWPProtocolHttps
IOCompress
IPCRun
JSONXS
PadWalker
CatalystDevel
Readonly
SetScalar
SQLSplitStatement
Starman
SysHostnameLong
TestMore
TextDiff
TextTable
XMLSimple
NetAmazonS3
nix git
  ];
  };
in

stdenv.mkDerivation rec {
  name = "hydra-${version}";

  src = fetchgit {
url = "https://github.com/NixOS/hydra.git";;
rev = "91f895b3d660560e40278b1c4e8c50172f3ac065";
sha256 =
"71954b6a3f6037b75698f6c7262cf0f97442bb5af8c5d31e543479ae8eee8f7a";
  };


  preConfigure = ''
# TeX needs a writable font cache.
export VARTEXFONTS=$TMPDIR/texfonts

addToSearchPath PATH $(pwd)/src/script
addToSearchPath PATH $(pwd)/src/c
addToSearchPath PERL5LIB $(pwd)/src/lib

./bootstrap
'';

  configureFlags = ["--with-nix=${nix}"
"--with-docbook-xsl=${docbook_xsl}/xml/xsl/docbook" ];

  buildInputs =
[ makeWrapper libtool unzip nukeReferences pkgconfig boehmgc sqlite
  gitAndTools.topGit mercurial darcs subversion bazaar openssl bzip2
  guile perlDeps perl
  autoconf automake libtool libxslt dblatex tetex
];

  hydraPath = lib.makeSearchPath "bin" (
[ libxslt sqlite subversion openssh nix coreutils findutils
  gzip bzip2 lzma gnutar unzip git gitAndTools.topGit mercurial
darcs gnused graphviz bazaar
] ++ lib.optionals stdenv.isLinux [ rpm dpkg cdrkit ] );

  preCheck = ''
patchShebangs .
export LOGNAME=${LOGNAME:-foo}
  '';

  postInstall = ''
mkdir -p $out/nix-support
nuke-refs $out/share/doc/hydra/manual/manual.pdf

for i in $out/bin/*; do
wrapProgram $i \
--prefix PERL5LIB ':' $out/libexec/hydra/lib:$PERL5LIB \
--prefix PATH ':' $out/bin:$hydraPath \
--set HYDRA_RELEASE ${version} \
--set HYDRA_HOME $out/libexec/hydra \
--set NIX_RELEASE ${nix.name}
done
  ''; # */

  meta.description = "Build of Hydra on ${system}";
  passthru.perlDeps = perlDeps;
}
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] pytables, numexpr, and pandas

2014-06-14 Thread Mateusz Kowalczyk
On 06/14/2014 11:17 PM, Andreas Herrmann wrote:
> Hi,
> 
> I've created two new nix-expressions, that I need to use the Pandas package.

Great.

Disclaimer: I'm not a person ‘in power’ of nixpkgs and simply speak from
experience.

> * PyTables is a python library for HDF5 storage. It's a different approach
> than h5py in that it is not a pythonic wrapper of the C library, but rather
> implements relational db alike features. Pandas depends on it to read/write
> HDF5 files.
> http://www.pytables.org/moin
> 
> * numexpr speeds up numpy ndarray expressions. PyTables depends on it.
> https://github.com/pydata/numexpr
> 
> However, I have a few questions before I go ahead and send a pull-request.
> 
> A) Is there some kind of test suite that I need to run on the expressions
> before I can send a pull request? (I can build and use them...)

Simply make sure it builds and runs on top of the current master. For
Python, check multiple Python versions. nix-build can help you build the
packages.

> B) Which branch should I send the pull-request to? Is master fine?

master is fine.

> C1) Both packages have a test-suite. However, it's not integrated into `
> setup.py`. I managed to get it running within the nix build process anyway.
> The corresponding `checkPhase` is attached below. Is that an acceptable way
> of doing this, or should I just set `doCheck = false;`?

If it has a test-suite then I see no reason not to include it. Make sure
that when tests actually do fail, this is caught as a failure rather
than accepted anyway. Make a test fail and see if the whole thing does.

> C2) The test-suite of pytables takes quite some time and has a number of
> failures of the "not-implemented" type. Is it worth including that
> test-suite or should I just leave that one out?

I would include any steps necessary to run the tests but after ensuring
that the failures aren't NixOS/nix specific, I'd set doCheck = false:
this makes it easier for you or someone else to turn on the tests on any
future, fixed versions. Make sure to let the maintainer know that there
are failing tests.

> D1) Pandas works without pytables. However, without it, it cannot
> import/export hdf5 archives. Should I add PyTables as a dependency to
> Pandas, or should every user individually add both to his environment if he
> needs both? Or is there a way to make it an optional dependency?

Optional dependencies are possible but I am unsure how they are done for
Python packages. While I don't know how to specify such optional
dependencies in python-packages.nix, you can simply create a
nix-expression outside of it. See pkgs/games/anki/default.nix for an
example of a Python package with optional dependencies. Provide a
sensible default because only the default way will be built by Hydra.

If you need Hydra to build the package in more ways then you should make
two separate entries in all-packages.nix with the different flag setups
but if there's only one common case then one should suffice.

> D2) The latest release of Pandas is 0.14.0, the nix-pkgs version is 0.12.0.
> Should I use this opportunity to bump that version number, or should this
> go through a different channel?

I would advise you to bump the Pandas version while you're working
around it.

> I've read the contributing guides on the wiki (at least those that I could
> find). I learned a lot from them, but the above questions remained.
> 
> Best,
> 
> Andreas
> 
> 
> The numexpr `checkPhase` within a call to `buildPythonPackage`:
> --
> 
> # Run the test suite.
> # It requires the build path to be in the python module search path.
> checkPhase = ''
>   ${python}/bin/${python.executable} <   import sysconfig
>   import sys
>   import os
>   f = "lib.{platform}-{version[0]}.{version[1]}"
>   lib = f.format(platform=sysconfig.get_platform(),
>  version=sys.version_info)
>   build = os.path.join(os.getcwd(), 'build', lib)
>   sys.path.insert(0, build)
>   import numexpr
>   numexpr.test()
>   EOF
> '';
> 
> 
> 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> 

I hope that somehow helps. If anything is incorrect, I'm sure someone
will correct me.

-- 
Mateusz K.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Michael Raskin
>> I think that any proposal would be seriously considered. Any already
>> existing application creating such a problem would increase your chances
>> to make someone else propose a solution… Otherwise — I think the problem
>> can surface, but I have no idea ho to fix it sanely… I think many other
>> people think the same.
>
>Well, let's not try to decide the undecidable.  Would it be so bad to
>have explicit dependencies?  That seems to be the only sound solution.

Nobody would use it… except in the cases where there is no other 
solution. I think it is a nice solution.

>A straightforward way is to add a small dotfile via the builder script
>with paths to the output.  The only problem at this point is that there
>is pretty much no support for doing that, so it's error-prone.  I could
>totally see something like this happening:
>
>stdenv.mkDerivation {
>...
>requisites = [cacert dataFiles1 dataFiles2];
>...
>}
>
>All it does is to register the given derivations as requisites
>unconditionally, perhaps simply by writing the aforementioned dotfile
>at $out/.nixdeps.

I think it should be $out/nix-support/forced-runtime-deps

What do other people think?

Is it something the should be added to fixupPhase in stdenv or is it 
worth adding in a slightly more complicated way without a rebuild? 
(Yes, situations like this are exactly why I think composing the entire
builder in Nix has its merits)



___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


[Nix-dev] pytables, numexpr, and pandas

2014-06-14 Thread Andreas Herrmann
Hi,

I've created two new nix-expressions, that I need to use the Pandas package.

* PyTables is a python library for HDF5 storage. It's a different approach
than h5py in that it is not a pythonic wrapper of the C library, but rather
implements relational db alike features. Pandas depends on it to read/write
HDF5 files.
http://www.pytables.org/moin

* numexpr speeds up numpy ndarray expressions. PyTables depends on it.
https://github.com/pydata/numexpr

However, I have a few questions before I go ahead and send a pull-request.

A) Is there some kind of test suite that I need to run on the expressions
before I can send a pull request? (I can build and use them...)
B) Which branch should I send the pull-request to? Is master fine?
C1) Both packages have a test-suite. However, it's not integrated into `
setup.py`. I managed to get it running within the nix build process anyway.
The corresponding `checkPhase` is attached below. Is that an acceptable way
of doing this, or should I just set `doCheck = false;`?
C2) The test-suite of pytables takes quite some time and has a number of
failures of the "not-implemented" type. Is it worth including that
test-suite or should I just leave that one out?
D1) Pandas works without pytables. However, without it, it cannot
import/export hdf5 archives. Should I add PyTables as a dependency to
Pandas, or should every user individually add both to his environment if he
needs both? Or is there a way to make it an optional dependency?
D2) The latest release of Pandas is 0.14.0, the nix-pkgs version is 0.12.0.
Should I use this opportunity to bump that version number, or should this
go through a different channel?

I've read the contributing guides on the wiki (at least those that I could
find). I learned a lot from them, but the above questions remained.

Best,

Andreas


The numexpr `checkPhase` within a call to `buildPythonPackage`:
--

# Run the test suite.
# It requires the build path to be in the python module search path.
checkPhase = ''
  ${python}/bin/${python.executable} <___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Ertugrul Söylemez
On Sat, 14 Jun 2014 22:09:21 +0400
Michael Raskin <7c6f4...@mail.ru> wrote:

> >It's probably not a big deal.  On the other hand consider that it only
> >happens in a rather special scenario and may go unnoticed for quite
> >some time.
> >
> >Consider an application (like a game) with a bunch of separate sets of
> >data files, each optional and referring to each other, compressed to
> >save space.  This is an actual design that I'm following in one of my
> >projects.  I'll know to make it Nix-friendly, but non-NixOS developers
> >may fail to do so.
> 
> I think that any proposal would be seriously considered. Any already
> existing application creating such a problem would increase your chances
> to make someone else propose a solution… Otherwise — I think the problem
> can surface, but I have no idea ho to fix it sanely… I think many other
> people think the same.

Well, let's not try to decide the undecidable.  Would it be so bad to
have explicit dependencies?  That seems to be the only sound solution.

A straightforward way is to add a small dotfile via the builder script
with paths to the output.  The only problem at this point is that there
is pretty much no support for doing that, so it's error-prone.  I could
totally see something like this happening:

stdenv.mkDerivation {
...
requisites = [cacert dataFiles1 dataFiles2];
...
}

All it does is to register the given derivations as requisites
unconditionally, perhaps simply by writing the aforementioned dotfile
at $out/.nixdeps.


Greets,
Ertugrul

-- 
Ertugrul Söylemez 
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Michael Raskin
>It's probably not a big deal.  On the other hand consider that it only
>happens in a rather special scenario and may go unnoticed for quite
>some time.
>
>Consider an application (like a game) with a bunch of separate sets of
>data files, each optional and referring to each other, compressed to
>save space.  This is an actual design that I'm following in one of my
>projects.  I'll know to make it Nix-friendly, but non-NixOS developers
>may fail to do so.

I think that any proposal would be seriously considered. Any already
existing application creating such a problem would increase your chances
to make someone else propose a solution… Otherwise — I think the problem
can surface, but I have no idea ho to fix it sanely… I think many other
people think the same.



___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Ertugrul Söylemez
On Sat, 14 Jun 2014 13:43:24 -0400
Shea Levy  wrote:

> > I can't.  I'm just pointing out a potential problem here.  If you
> > disagree, I'll be happy to construct a proof of concept.
> 
> Of course it could happen, but we have a huge number of packages in nixpkgs, 
> including a significant subset of hackage, and it hasn't actually happened 
> yet as far as anyone knows.
> 
> No one claims hash scanning/hash rewriting is 100% sound. It is, however, 
> extremely convenient and has worked remarkably well up until now. If we start 
> finding real world use cases where it fails, then we can start thinking about 
> alternatives, but until then why is this a big deal?

It's probably not a big deal.  On the other hand consider that it only
happens in a rather special scenario and may go unnoticed for quite
some time.

Consider an application (like a game) with a bunch of separate sets of
data files, each optional and referring to each other, compressed to
save space.  This is an actual design that I'm following in one of my
projects.  I'll know to make it Nix-friendly, but non-NixOS developers
may fail to do so.


Greets,
Ertugrul

-- 
Ertugrul Söylemez 
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Shea Levy


> On Jun 14, 2014, at 13:29, Ertugrul Söylemez  wrote:
> 
> On Sat, 14 Jun 2014 09:36:21 -0400
> Shea Levy  wrote:
> 
>> Please point to a specific example of a package with an unretained 
>> dependency due to UTF-16 storage
> 
> I can't.  I'm just pointing out a potential problem here.  If you
> disagree, I'll be happy to construct a proof of concept.
> 

Of course it could happen, but we have a huge number of packages in nixpkgs, 
including a significant subset of hackage, and it hasn't actually happened yet 
as far as anyone knows.

No one claims hash scanning/hash rewriting is 100% sound. It is, however, 
extremely convenient and has worked remarkably well up until now. If we start 
finding real world use cases where it fails, then we can start thinking about 
alternatives, but until then why is this a big deal?

> 
> Greets,
> Ertugrul
> 
> -- 
> Ertugrul Söylemez 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Ertugrul Söylemez
On Sat, 14 Jun 2014 09:36:21 -0400
Shea Levy  wrote:

> Please point to a specific example of a package with an unretained dependency 
> due to UTF-16 storage

I can't.  I'm just pointing out a potential problem here.  If you
disagree, I'll be happy to construct a proof of concept.


Greets,
Ertugrul

-- 
Ertugrul Söylemez 
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Shea Levy


> On Jun 14, 2014, at 4:28, Ertugrul Söylemez  wrote:
> 
> On Fri, 13 Jun 2014 18:54:53 +0200
> Eelco Dolstra  wrote:
> 
>>> The path-rewriting proposal is a very bad idea and will cause a lot of
>>> breakage.  For many/enough applications rewriting will not work at all,
>>> because they might encode paths in data structures or be using a
>>> non-UTF8 multi-byte encoding.
>> 
>> Packages that store paths in UTF-16 don't work with Nix anyway, independent 
>> from
>> hash rewriting, because Nix finds runtime dependencies by scanning for plain
>> ASCII hashes. To my knowledge we've never encountered such a package in 
>> Nixpkgs.
>> 
>> This paper has a small evaluation of hash rewriting:
>> http://nixos.org/~eelco/pubs/secsharing-ase2005-final.pdf (section 6.1)
> 
> We certainly have.

Please point to a specific example of a package with an unretained dependency 
due to UTF-16 storage

>  As said, Haskell works that way:  while hardcoded
> paths are commonly stored in UTF-8, serialized text is more likely in
> UTF-16, because that's the internal format of the Text type.  Also
> continuous path strings are very uncommon, because we use
> platform-independent combinators to construct paths.
> 
> Nix' dependency scanning still works most of the time, because paths to
> other Haskell packages are stored at least in a GHC registration file as
> continuous strings.  It could fail for other requisites like data files
> from non-Haskell packages.  For example the CA certificates are stored
> in their own derivation.  If not half of the system would depend on that
> derivation, it would most likely be GCed away and Haskell programs
> would be unable to find them.
> 
> Rewriting is even worse, because now you are interpreting a
> structurally rich file as a mere byte-string and then changing things
> based on vague assumptions.  This is dangerous, in particular for ELF
> files, which are in fact Turing-complete (ELF itself, not the code it
> contains).  Just as a theoretical point consider that the question
> whether your rewriting algorithm is correct is undecidable in general.
> 
> More practically you would break half of Hackage.  The breakage would
> range from being as gentle as a Haskell program failing to find
> certificate files to as severe as a Haskell plugin-based web
> application (like clckwrks) fatal-error-crashing.
> 
> As a Haskell developer I must warn you about rewriting.  This /will/
> cause trouble at least for Haskell.  Even the current dependency
> scanning mechanism is a ticking time bomb for Haskell users, though it
> did not yet explode for me.
> 
> 
>> Or even better, ask the admin to use pam_namespace:
>> 
>> http://www.linux-pam.org/Linux-PAM-html/sag-pam_namespace.html
>> 
>> And hopefully, one day users won't need to be root to do bind mounts.
> 
> That would be very useful, yeah.  It's probably already possible anyway
> given the right capability, but I don't know.
> 
> 
> Greets,
> Ertugrul
> 
> -- 
> Ertugrul Söylemez 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev


Re: [Nix-dev] Binary packages for custom store path installs

2014-06-14 Thread Ertugrul Söylemez
On Fri, 13 Jun 2014 18:54:53 +0200
Eelco Dolstra  wrote:

> > The path-rewriting proposal is a very bad idea and will cause a lot of
> > breakage.  For many/enough applications rewriting will not work at all,
> > because they might encode paths in data structures or be using a
> > non-UTF8 multi-byte encoding.
> 
> Packages that store paths in UTF-16 don't work with Nix anyway, independent 
> from
> hash rewriting, because Nix finds runtime dependencies by scanning for plain
> ASCII hashes. To my knowledge we've never encountered such a package in 
> Nixpkgs.
>
> This paper has a small evaluation of hash rewriting:
> http://nixos.org/~eelco/pubs/secsharing-ase2005-final.pdf (section 6.1)

We certainly have.  As said, Haskell works that way:  while hardcoded
paths are commonly stored in UTF-8, serialized text is more likely in
UTF-16, because that's the internal format of the Text type.  Also
continuous path strings are very uncommon, because we use
platform-independent combinators to construct paths.

Nix' dependency scanning still works most of the time, because paths to
other Haskell packages are stored at least in a GHC registration file as
continuous strings.  It could fail for other requisites like data files
from non-Haskell packages.  For example the CA certificates are stored
in their own derivation.  If not half of the system would depend on that
derivation, it would most likely be GCed away and Haskell programs
would be unable to find them.

Rewriting is even worse, because now you are interpreting a
structurally rich file as a mere byte-string and then changing things
based on vague assumptions.  This is dangerous, in particular for ELF
files, which are in fact Turing-complete (ELF itself, not the code it
contains).  Just as a theoretical point consider that the question
whether your rewriting algorithm is correct is undecidable in general.

More practically you would break half of Hackage.  The breakage would
range from being as gentle as a Haskell program failing to find
certificate files to as severe as a Haskell plugin-based web
application (like clckwrks) fatal-error-crashing.

As a Haskell developer I must warn you about rewriting.  This /will/
cause trouble at least for Haskell.  Even the current dependency
scanning mechanism is a ticking time bomb for Haskell users, though it
did not yet explode for me.


> Or even better, ask the admin to use pam_namespace:
> 
> http://www.linux-pam.org/Linux-PAM-html/sag-pam_namespace.html
> 
> And hopefully, one day users won't need to be root to do bind mounts.

That would be very useful, yeah.  It's probably already possible anyway
given the right capability, but I don't know.


Greets,
Ertugrul

-- 
Ertugrul Söylemez 
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev