Re: About guix system init

[Chris Marusich]

rennes  writes:

> Is it necessary to have the services started correctly to run 'guix  
> system init ..' ?

No, I don't think it's necessary.  Judging by the source code, it looks
like "guix system init" does not start any services.  The relevant bits
are in the "install" procedure in guix/scripts/system.scm.

-- 
Chris


signature.asc
Description: PGP signature

Re: Guix to participate in the Google Summer of Code

[Chris Marusich]

Ricardo Wurmus  writes:

> Garbage collection in the context of Guix is not very difficult to
> understand.  The garbage collector may remove any item that has zero
> users; the link count on a store item is indicative of the number of
> users that item has.  For garbage collection you essentially go through
> all items and check if the link count is zero.

Is this correct?  My understanding is that the link count of a store
path (i.e., the count of hard links for a given file in the store) is
unrelated to whether or not that store path is live.  It looks like, in
nix/libstore/gc.cc, we basically walk the references graph starting from
the GC roots and delete anything that can't be reached from a root.
This makes sense, since anything that can be reached from a root is live
and must not be deleted, and anything that can't be reached from any
root can be deleted.  Link counts (e.g., the number of hard links) do
not seem to play a role here.

In addition to the C++ code, the Nix thesis has a good discussion of the
garbage collection algorithm.  I don't know if it's accurate, but it
helps shed light on the intent.

-- 
Chris


signature.asc
Description: PGP signature

Re: 01/01: gnu: tzdata: Preserve directory layout from before tzdata 2018a.

[Leo Famulari]

On Tue, Feb 20, 2018 at 06:17:34PM +0100, Marius Bakke wrote:
> Leo Famulari  writes:
> 
> > lfam pushed a commit to branch staging
> > in repository guix.
> >
> > commit 62868f124cf9443a43f4ea5867da692e32e77c58
> > Author: Leo Famulari 
> > Date:   Mon Feb 19 23:47:46 2018 -0500
> >
> > gnu: tzdata: Preserve directory layout from before tzdata 2018a.
> > 
> > tzdata 2018a instroduced some changes to the build and installation
> > procedures that commit e2832cdc3 failed to address. They are described
> > in the NEWS file.
> > 
> > This is a followup to commit e2832cdc3599e3bbc0362f9749835ffbae1b1944.
> > 
> > * gnu/packages/base.scm (tzdata)[arguments]: Set TZDEFAULT. Remove the
> > time-zone utility programs in a build phase.
> 
> This change breaks "tzdata-for-tests".  Are there preferences to either:
> 
> 1) Take the 2360 packages rebuild hit (and update dbus, glib and
>   friends); or
> 2) Copy the old package definition to "tzdata-for-tests".
> 
> I'm leaning towards 2), and instead do a new core-updates round in the
> near future, since the previous core-updates took way longer than normal.

Oops, of course it broke. I'll do 2 shortly.


signature.asc
Description: PGP signature

Re: 01/01: gnu: tzdata: Preserve directory layout from before tzdata 2018a.

[Marius Bakke]

Leo Famulari  writes:

> lfam pushed a commit to branch staging
> in repository guix.
>
> commit 62868f124cf9443a43f4ea5867da692e32e77c58
> Author: Leo Famulari 
> Date:   Mon Feb 19 23:47:46 2018 -0500
>
> gnu: tzdata: Preserve directory layout from before tzdata 2018a.
> 
> tzdata 2018a instroduced some changes to the build and installation
> procedures that commit e2832cdc3 failed to address. They are described
> in the NEWS file.
> 
> This is a followup to commit e2832cdc3599e3bbc0362f9749835ffbae1b1944.
> 
> * gnu/packages/base.scm (tzdata)[arguments]: Set TZDEFAULT. Remove the
> time-zone utility programs in a build phase.

This change breaks "tzdata-for-tests".  Are there preferences to either:

1) Take the 2360 packages rebuild hit (and update dbus, glib and
  friends); or
2) Copy the old package definition to "tzdata-for-tests".

I'm leaning towards 2), and instead do a new core-updates round in the
near future, since the previous core-updates took way longer than normal.


signature.asc
Description: PGP signature

Re: PYTHONPATH woes

[Pjotr Prins]

On Tue, Feb 20, 2018 at 04:18:40PM +0100, Andy Wingo wrote:
> In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's
> not quite the same I don't think.

Not exactly the same, but close enough ;). But it is clearer now that
mixing is the problem. Ricardos .pth may be an option if that works,
or we introduce a separate GUIX_PYTHONPATH2 and GUIX_PYTHONPATH for 3.
Python2 is at end of life, so we may set an example there for others.

Even so, I don't think it will solve the particular conflict that
Ricardo was describing. There will always be hairy mix-ins.

Pj.
-- 

Re: PYTHONPATH woes

[Ricardo Wurmus]

Hi Pjotr,

> On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
>> Would it be good to make the wrappers for Python scripts stricter and
>> not accept any user-set PYTHONPATH?
>
> I think that is a bad idea. You need to be able to opt out. Also
> people need to experiment with modules without understanding Guix per
> se. In my upcomping blog I would emphasize packaging at the point you
> become a serious user.
>
> That should come with a health warning ;). Similarly we should allow
> for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
> dangerous.

While I agree that it must be *possible* to opt out, I think the
defaults are wrong here.  We don’t ever set LD_LIBRARY_PATH when
building a profile, but we do set PYTHONPATH.  Having PYTHONPATH set
(even without the knowledge of the user) leads to problems here when it
shouldn’t.

I’d argue that the number of users who need to be able to override
individual Python packages for a tool like Ribodiff is way lower than
the number of those who accidentally get into a situation where
PYTHONPATH is set purely because of the contents of their profile,
leading to breakage of unrelated packages that just happen to use Python
modules.

>> How do we approach the problem of having both Python 2 modules and
>> Python 3 modules in the same profile?  PYTHONPATH will be set to refer
>> to the site-packages directories of both versions, which is never good.
>> Does Python offer us a way to do better?  Can we make use of pth files
>> to get around this problem somehow?
>
> Python should have created PYTHONPATH2 to split them out. We could
> patch python2 to do just that.

I think there’s an alternative, but I don’t know it well.  The official
way is to use “.pth” files instead of setting PYTHONPATH.  Maybe there’s
a way that doesn’t involve setting PYTHONPATH.  Instead we could nudge
Python towards reading the profile’s “.pth” file and read the package
locations from there.

This would require a new profile hook and possibly a patch to Python to
add a way to tell it to read the “.pth” file from a location provided by
a Guix environment variable.

> Even so, the real solution is separate profiles. I get that with
> versions of Ruby too.

Right, but it is not obvious what packages must be treated with extra
care.  When I install the Ribodiff package I don’t know or care that
it’s written in Python.  That shouldn’t matter at all.  But now I
actually have to pay attention to this and install Ribodiff in a profile
that doesn’t contain Python 3 things.

That’s rather complicated and unfriendly for users.  I don’t want the
users here to be anxious about installing software, just because a new
tool may be using Python and thus might break when installed to a
profile containing Python things.  We made these wrappers precisely to
isolate the tools from the current environment.  Allowing them to be
disturbed so easily is making wrappers much less useful.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: PYTHONPATH woes

[Andy Wingo]

On Tue 20 Feb 2018 16:01, Pjotr Prins  writes:

> On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
>> Would it be good to make the wrappers for Python scripts stricter and
>> not accept any user-set PYTHONPATH?
>
> I think that is a bad idea. You need to be able to opt out.

Why?  I am not sure this is the case for programs that just happen to be
written in Python.

> That should come with a health warning ;). Similarly we should allow
> for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
> dangerous.

In Guix we don't set LD_LIBRARY_PATH but we do set PYTHONPATH, so it's
not quite the same I don't think.

Andy

Re: PYTHONPATH woes

[Pjotr Prins]

On Tue, Feb 20, 2018 at 11:53:54AM +0100, Ricardo Wurmus wrote:
> Would it be good to make the wrappers for Python scripts stricter and
> not accept any user-set PYTHONPATH?

I think that is a bad idea. You need to be able to opt out. Also
people need to experiment with modules without understanding Guix per
se. In my upcomping blog I would emphasize packaging at the point you
become a serious user.

That should come with a health warning ;). Similarly we should allow
for LD_LIBRARY_PATH etc. It is what they exist for, even if it is
dangerous.

> How do we approach the problem of having both Python 2 modules and
> Python 3 modules in the same profile?  PYTHONPATH will be set to refer
> to the site-packages directories of both versions, which is never good.
> Does Python offer us a way to do better?  Can we make use of pth files
> to get around this problem somehow?

Python should have created PYTHONPATH2 to split them out. We could
patch python2 to do just that.

Even so, the real solution is separate profiles. I get that with
versions of Ruby too.

Pj.

Re: Staging

[Efraim Flashner]

On Tue, Feb 20, 2018 at 02:17:59PM +0100, Andreas Enge wrote:
> On Tue, Feb 20, 2018 at 03:14:39PM +0200, Efraim Flashner wrote:
> > CMake is just old, not especially in need of updates. It would be nice
> > to ungraft stuff though.
> 
> I just meant to not switch cmake versions together with other things:
> 
> $ ./pre-inst-env guix refresh -l cmake
> Building the following 1007 packages would ensure 2499 dependent packages are 
> rebuilt
> 
> A lot depends on cmake, so we can expect some breakage, and mixing this with
> other potential sources of breakage would make it more difficult to diagnose.
> 
> Andreas
> 

Works for me. Plus if we have a short staging cycle we can do it more
often¹.

¹hydra permitting

-- 
Efraim Flashner      אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: Staging

[Andreas Enge]

On Tue, Feb 20, 2018 at 03:14:39PM +0200, Efraim Flashner wrote:
> CMake is just old, not especially in need of updates. It would be nice
> to ungraft stuff though.

I just meant to not switch cmake versions together with other things:

$ ./pre-inst-env guix refresh -l cmake
Building the following 1007 packages would ensure 2499 dependent packages are 
rebuilt

A lot depends on cmake, so we can expect some breakage, and mixing this with
other potential sources of breakage would make it more difficult to diagnose.

Andreas

Re: Staging

[Efraim Flashner]

On Tue, Feb 20, 2018 at 01:07:11PM +0100, Andreas Enge wrote:
> On Tue, Feb 20, 2018 at 06:57:06AM -0500, Leo Famulari wrote:
> > On Tue, Feb 20, 2018 at 11:45:58AM +0200, Efraim Flashner wrote:
> > > Java currently only affects x86_64, so I'm not sure that it would need
> > > to be staging specifically. In any case if we can get it merged within
> > > the next week or so that'd be amazing. I know our cmake is in need of an
> > > upgrade, maybe libva?
> > 
> > Are there any specific issues that would be fixed by a CMake update, or
> > is it just "too old"? I think that changes on the staging branch should
> > have a low probability of breaking things, so that we can finish the
> > branch quickly.
> 
> Could we stage things on the staging branch? Do a few updates now, merge
> quickly, and then try to update only cmake in the next run?
> 

CMake is just old, not especially in need of updates. It would be nice
to ungraft stuff though.

-- 
Efraim Flashner      אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: Staging

[Arun Isaac]

>> I know our cmake is in need of an upgrade

I submitted a patchset for updating cmake a couple of weeks ago.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30390

Re: Staging

[Andreas Enge]

On Tue, Feb 20, 2018 at 06:57:06AM -0500, Leo Famulari wrote:
> On Tue, Feb 20, 2018 at 11:45:58AM +0200, Efraim Flashner wrote:
> > Java currently only affects x86_64, so I'm not sure that it would need
> > to be staging specifically. In any case if we can get it merged within
> > the next week or so that'd be amazing. I know our cmake is in need of an
> > upgrade, maybe libva?
> 
> Are there any specific issues that would be fixed by a CMake update, or
> is it just "too old"? I think that changes on the staging branch should
> have a low probability of breaking things, so that we can finish the
> branch quickly.

Could we stage things on the staging branch? Do a few updates now, merge
quickly, and then try to update only cmake in the next run?

Andreas

Re: Staging

[Leo Famulari]

On Tue, Feb 20, 2018 at 11:45:58AM +0200, Efraim Flashner wrote:
> Java currently only affects x86_64, so I'm not sure that it would need
> to be staging specifically. In any case if we can get it merged within
> the next week or so that'd be amazing. I know our cmake is in need of an
> upgrade, maybe libva?

Are there any specific issues that would be fixed by a CMake update, or
is it just "too old"? I think that changes on the staging branch should
have a low probability of breaking things, so that we can finish the
branch quickly.


signature.asc
Description: PGP signature

PYTHONPATH woes

[Ricardo Wurmus]

Hi Guix,

we have a couple of packages that provide scripts that depend on Python
modules.  We wrap them in PYTHONPATH to ensure that the correct Python
modules are found at runtime.

This is not enough.

We don’t wrap them tightly enough; instead we allow for a user-provided
PYTHONPATH value to be added to the PYTHONPATH we specified.  The result
is that a user-set PYTHONPATH can act like LD_LIBRARY_PATH — it causes
chaos.  This is despite the fact that we make sure that the wrapper’s
PYTHONPATH comes first!

Suppose a user installs python@2 and python2-statsmodels; at a later
point the user upgrades Guix, and then installs the ribodiff package.
The user does not know that ribodiff is written in Python, nor should
the user be aware of that.

Because Python is installed in the profile, etc/profile will contain a
definition for PYTHONPATH.  The user may source that etc/profile file to
set up all required environment variables.  But now running the ribodiff
scripts fails!

Here’s what happens: the PYTHONPATH that Guix sets for the profile now
contains an incompatible variant of the python2-statsmodels package.
Guix has been upgraded between installing python2-statsmodels and
ribodiff, so a different version of Python was used to build these
modules.  Since the ribodiff wrapper script gladly accepts any set
PYTHONPATH, it causes the ribodiff scripts to load the old and
incompatible python2-statsmodels package instead of the compatible one
from the wrapper.

I don’t know why this happens.  I find it puzzling that in this
particular case the user’s profile contains an *older* version of
statsmodels (0.6.1).  The wrapper includes the correct version of
statsmodels (0.8.0) in the PYTHONPATH.  Here’s the backtrace:

--8<---cut here---start->8---
Traceback (most recent call last):
  File 
"/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", 
line 81, in 
main()
  File 
"/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/bin/.TE.py-real", 
line 26, in main
import ribodiff.estimatedisp as ed
  File 
"/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/estimatedisp.py",
 line 7, in 
import rawdisp as rd
  File 
"/gnu/store/bz9l68hwlvwbp21msm2v002y7s8qfdd3-ribodiff-0.2.2/lib/python2.7/site-packages/ribodiff/rawdisp.py",
 line 8, in 
import statsmodels.api as sm
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/__init__.py",
 line 8, in 
from .tools.sm_exceptions import (ConvergenceWarning, CacheWriteWarning,
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/__init__.py",
 line 1, in 
from .tools import add_constant, categorical
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/tools/tools.py",
 line 11, in 
from statsmodels.datasets import webuse
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/__init__.py",
 line 5, in 
from . import (anes96, cancer, committee, ccard, copper, cpunish, elnino,
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/__init__.py",
 line 1, in 
from .data import *
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/anes96/data.py",
 line 90, in 
from statsmodels.datasets import utils as du
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/statsmodels-0.6.1-py2.7-linux-x86_64.egg/statsmodels/datasets/utils.py",
 line 13, in 
from pandas import read_csv, DataFrame, Index
  File 
"/home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/__init__.py",
 line 31, in 
"extensions first.".format(module))
ImportError: C extension: 
/home/uzinnal/.guix-profile/lib/python2.7/site-packages/pandas-0.18.1-py2.7-linux-x86_64.egg/pandas/hashtable.so:
 undefined symbol: PyUnicodeUCS2_FromStringAndSize not built. If you want to 
import pandas from the source directory, you may need to run 'python setup.py 
build_ext --inplace' to build the C extensions first.
--8<---cut here---end--->8---

Now you could say that this is the user’s fault for not using manifests.
But consider this: what happens if the user had a manifest and installed
“python-statsmodels” instead of the Python 2 variant?  Guix would still
set PYTHONPATH and the ribodiff wrapper would still prefer the profile’s
PYTHONPATH over the wrapped value, so it would cause Python 2 (from
ribodiff) to load a Python 3 module of statsmodels — these are not
compatible and again we have a runtime crash.

Manifests wouldn’t avoid this problem.

Avoiding this problem now 

Re: Staging

[Efraim Flashner]

On Tue, Feb 20, 2018 at 04:54:36AM +0100, Marius Bakke wrote:
> Leo Famulari  writes:
> 
> > There are some patches on the staging branch, including an update to the
> > time zone database.
> >
> > What are the plans for the next week or so? Should we try merging the
> > staging branch in that timeframe?
> 
> Yes, let's start this in a few days, when Hydra calms down.  Perhaps we
> can include the Java updates discussed in
>  as
> well.  Ricardo, WDYT?
> 
> I have a handful more updates in my queue and will try to get them done
> by tomorrow.

Java currently only affects x86_64, so I'm not sure that it would need
to be staging specifically. In any case if we can get it merged within
the next week or so that'd be amazing. I know our cmake is in need of an
upgrade, maybe libva?


-- 
Efraim Flashner      אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: bug#30537: glibc 2.26 refuses to run on CentOS 6.8

[Efraim Flashner]

On Mon, Feb 19, 2018 at 07:46:02PM +0100, Ricardo Wurmus wrote:
> Hi Guix,
> 
> I have a bad day.  After the upgrade to glibc 2.26 none of the
> Guix-installed software runs on the HPC cluster running CentOS 6.8.
> 
> The glibc 2.26 expects a minimum kernel version of 3.x on x86_64, but
> CentOS 6.8 only comes with a heavily patched 2.6.32.
> 
> The NixOS developers patch glibc to make sure that all software still
> runs on Linux 2.6.32:
> 
> 
> https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/libraries/glibc/allow-kernel-2.6.32.patch
> 
> Can we please also apply this?  Without this Guix on HPC is pretty much
> dead at the MDC.
> 

We should also update the docs with the (new) minimum specs as needed.

-- 
Efraim Flashner      אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: Talk about Guix and GuixSD at FSF Hungary

[Gábor Boskovits]

2018-02-18 14:22 GMT+01:00 Ricardo Wurmus :

>
> Hi Gábor,
>
> > On february the 20. at 18:00 local time I will give a talk about Guix and
> > GuixSD for the Hungarian FSF community.
>
> That’s great!
>
> > I based the style of my talk on one given by Ludo. I Left the copyright
> > notices intact, only adding my name. I hope this is the right thing to
> do,
> > but I would like to hear your opinon.
>
> This seems to be the correct way to do this.
>
> > One more thing, I am willing to add the source of this talk to the
> > maintainers repo. What is the right way to do that? Who should I contact?
>
> The repository is here:
>
> http://git.savannah.gnu.org/cgit/guix/maintenance.git/
>
> Usually, it’s just a matter of adding a new directory containing the
> sources.  You can send a URL to the sources (e.g. as a tarball) to
> guix-devel; we would download them from there and add the sources to the
> maintenance repo.
>
>
Thanks for the information. I uploaded the tarball to here:
https://ufile.io/hsft0


> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>