Re: Feedback from JRES in Dijon

[Konrad Hinsen]

Hi Bengt,

>> [1] 
>> https://webcast.in2p3.fr/video/les-enjeux-et-defis-de-la-recherche-reproductible
>> https://aramis.resinfo.org/wiki/lib/exe/fetch.php?media=pleniaires:aramis_keynote_enjeux-et-defis-recherche-reproductible_konrad_hinsen.pdf
>>
>
> Is [1] available as a libre video download?

No idea. I can answer most questions about the content since I am the
speaker, but I was not involved with recording and publishing.

> IMO if you can't reproduce bit-identical results, you should find out 
> _exactly_ why.
> And if you do get exactly the same result, you should also know exactly why 
> ;-)

That's a nice way to put it!

> The video touches on IEEE 754 (and I do believe the lecturer
> understands these issues, but he says (IIU.fr :) no programming
> language gives control over the FPU -- is that true?? I mean,
> including on-the-metal x86_64 assembly language??

I wouldn't count that as a programming language, but that is of course
debatable. For me, a programming language is a platform-neutral medium
that can be run on any type of computer. And yes, you can argue that
the subset of x86 assembly that doesn't manipulate hardware other than
registers fits that definition, but you have to draw the line somewhere.

More interestingly (definitions aren't really interesting), I am of
course not certain that there is no programming language that gives
access to the full IEEE754 programming model. I am not aware of any,
but if there is, I'd like to find out!

> BTW, if you are interested in floating point, links from here should be fun:
>
> --8<---cut here---start->8---
> https://en.wikipedia.org/wiki/William_Kahan
> --8<---cut here---end--->8---

Indeed. For those unfamiliar with that name, Kahan designed the 8087
floating-point processor for Intel and was the main driver behind the
IEEE 754 standardization effort.

Cheers,
  Konrad.

Re: Feedback from JRES in Dijon

[Konrad Hinsen]

Hi Simon,

> If I might, one the best presentation [1] -- that I am aware of -- on
> this. Sorry in French.

Thanks for the marketing :-)

> [1] 
> https://webcast.in2p3.fr/video/les-enjeux-et-defis-de-la-recherche-reproductible
> https://aramis.resinfo.org/wiki/lib/exe/fetch.php?media=pleniaires:aramis_keynote_enjeux-et-defis-recherche-reproductible_konrad_hinsen.pdf
>
> Maybe we could convert it to an entry for the HPC blog. What do you think?

I am working on an article (ultimately for "Computing in Science and
Engineering") that takes up some of those subjects and illustrates them
using Guix. It would be nice to get feedback from the Guix community
early, to make sure I don't do something stupid. So I could do a first
version for the Guix blog, and thus propose it here for review,
and then make it a bit less technical for a wider audience.

Cheers,
  Konrad.

Re: Feedback from JRES in Dijon

[Konrad Hinsen]

Hi Julien and Pierre,

> So they are doing physical simulation (fluid dynamics), so they don't
> (can't) get the same result when running the same experiment
> twice. They wart replicability, that is, even if the results are
> different, they are close enough to each other that you have to draw
> the same scientific conclusion, independent of your compiler or other
> package inputs.

That's a common point of view in the numerical simulation community.
What the people defending it don't realize is that both reproducibility
and replicability matter, but in different situations and for different
reasons. Reproducibility matters for verification ("was the computation
done correctly?"), replicability matters for validation ("was the
computation the right one for the scientific question?").

Moreover, there is a practical use for reproducibility when checking for
replicability. Suppose you have a program and a result, then you run the
program with a different compiler and get a different result, too
different to be scientifically equivalent. No replicability. Then what?
How do you figure out what went wrong? The very first thing you want to
check is reproducibility: can you get the same result by using the same
compiler? If yes, fine, you can then look at intermediate results in
both versions of the computation to figure out where the differences
come from. If not, there is no point in wasting time on that: there is
something wrong with the code or the data you got, and you have to check
your sources first.

Giving up on reproducibility thus means giving up a valuable debugging
technique.

Cheers,
  Konrad

Re: On DSLs

[Konrad Hinsen]

Hi Ludo,

>> For better illustration, I'll try to rewrite my own manifests in the
>> way I'd like to be able to write them. That's probably more useful
>> than theory (a tough statement to make for a theoretician ;-)
>
> Agreed!

Just to be clear: I actually intend to implement some infrastructure to
make this happen, so this will take a while. One of my convictions after
decades in computational science is that ideas without implementations
should not be discussed extensively, so I try to live by my own
standards ;-)

> That’s not true.  In some cases, people write something that’s actually
> code (in YAML, in JSON, etc.) and there’s an interpreter running it.

I have been fortunate enough not to have seen such things yet!

> IOW, I think you can have a declarative _style_ in a full-blown
> language, like:

Definitely, and that's what I am aiming for. There remains the security
issue of malevolent power users sneaking in innocuous-looking
non-declarative code that non-expert users might run without suspicion.
But I'd say we can make a lot of progress by having declarative style
for all routine configuration data.

> This is just to say that we should not conflate the style and the
> language.  I think what we care about is supporting a declarative style,
> and making it expressive enough that people don’t feel the need to
> resort to complicated code.

Exactly!

Cheers,
  Konrad.

Re: On DSLs

[Konrad Hinsen]

Hi Julien,

> Could this discussion be saved in the cookbook for instance? I'd like
> to have this kind of discussion on the approach of guix and ideas
> behind it somewhere more accessible than the ML archive. Does it make
> sense?

I think it makes sense to archive summaries of such discussions. I am
less convinced that the cookbook is the right place, because design
principles are pretty much the opposite of what I'd expect in a
cookbook. But I don't see any better place and multiplying the parts of
the documentation isn't great either. So... I am undecided ;-)

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Bengt and Simon,

Nice discussion, please go on. With Simon taking the pragmatic point of
view ("what can we do right now?") and Bengt the long-term idealist
perspective ("where should Guix be going?"), we might end up getting
both ends right.


Bengt Richter  writes:

> The first step is probably cheap, but if you just s/python2/python3/g
> and it "works," what do you know? Can you even begin to trust, unless
> automated tests were _really_^n well designed ??

That's every packager's problem. We see our main job as assembling
pieces into a whole, but we should ideally also check the quality
of our building blocks. Unfortunately, that kind of responsibility
has never been valued in the tech world.

Which means that, pragmatically speaking, we'll have to do what everyone
else does: delegate quality control to the end user. Otherwise we'd have
only a hundred packages in Guix and nobody would be interested.

> I think it represents more than that. It represents an instance of
> a kind of system viewer/browser.

That's exactly where I want to go in the long run. The kind of analysis
I have been doing with a Guile script should be integrated into a
decent user interface, emacs-guix being the obvious candidate.
Every software system should have a scriptable user interface,
in my opinion.

> Yeah, we can get the sources for the package manually and
> look at them various ways, but how long before you make
> yourself a viewer of some sort?

Unfortunately that's a hard problem, given the enormous diversity of
packages in Guix. But for specific cases, it looks doable. Much like we
have various build systems for frequent special cases, we could have
inspectors for them as well.

Back to Python, I could add a check for python-build-system to
distinguish between Python libraries/applications and non-Python
code that just contains small helper scripts.

> So  I think it would be good for guix if the top talent would
> favor allocating time to making wizard tools that will empower
> lesser mortals to see and operate on what they can't see
> without the tools.

While I disagree with your choice of terms (top vs. lesser), I do
agree with the principle of providing empowering tools for a wide
range of users (who are not "lesser mortals" for me, but people
with a different focus).

>> For example, considering rust, it is about the bootstrappability. See [1].
>> 
>> [1] http://guix.gnu.org/blog/2018/bootstrapping-rust/
>
> That looks horrible to me :)

Me too. Another long-term mission for Guixers (or perhaps better the
Reproducible Builds community) is education about bootstrapping.
I can understand the reasoning that led the Rust developers to their
approach, but it also shows that they are not aware that they are part
of a larger universe.

>> > It would be really interesting if you could tweak your 
>> > py2-dependent-package
>> > lister to show for each how many lines of py2 code are causing the py2 
>> > dependency!
>> 
>> It is really hard -- nor impossible. And I am not convinced that the
>> tough work will pay off.
>
> Why so hard? Is not all the information available in sources?

Yes, but the structure of the code is not very standardized. Counting
the lines of Python code in a source package is straightforward, but
knowing how important it is is a hard problem. Consider a three-module
Python package with 15 example scripts.

> That's the kind of tools I hope will emerge, if it proves
> feasible to reduce external dependencies that way.

Indeed. And not just for Python. Everybody is living in dependency hell
these days. Guix could in the long run help to fix that.

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[Konrad Hinsen]

Konrad Hinsen  writes:

> I'd say the very first thing we should do is look at all non-Python
> packages that depend indirectly on Python 2.

Here comes an update of my Python-2-in-Guix analysis. Sorted output,
distinction between likely libraries and likely application packages.
Plus the list of Python 2 libraries that something else depends on, with
special attention on those who don't currently have a Python 3
equivalent in Guix. Those would be good targets for porting work as
well.

Cheers,
  Konrad.

;;
(use-modules (guix packages)
 (gnu packages)
 (srfi srfi-1)
 (ice-9 format))

(define (library-in-python2-ecosystem? package)
  (string-prefix? "python2" (package-name package)))

(define (has-no-python3-equivalent? package)
  (define python3-name (string-replace (package-name package) "python" 0 7))
  (null? (find-packages-by-name python3-name)))

(define (dependencies package)
  (filter package?
  (map second
   (package-direct-inputs package

(define python2-dependent-packages
  (fold-packages
   (lambda (package result)
 (define deps (dependencies package))
 (cond ((library-in-python2-ecosystem? package)
result)
   ((and (not (null? deps))
 (every library-in-python2-ecosystem? deps))
(list (cons package (first result))
  (second result)
  (lset-union equal? deps (third result
   ((any library-in-python2-ecosystem? deps)
(list (first result)
  (cons package (second result))
  (lset-union equal? (third result
   (else result)))
   '(() () (

(define (package-< p1 p2)
  (or (string

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Hartmut,

> I assume many of these package can be updated. (Indeed I just updated
> pdfposter, which I'm maintaining.)

Great. That's the kind of response I was hoping for!

> bypthon2 and ptpython2 are REPLs for python2 - bypthon and ptpython are
> the resp. packages for python3.

There are a few other packages in my list that are part of the Python 2
universe (my own nmoldyn for example), but I see no easy way to detect
them in an analysis of the package graph. Perhaps I could filter out
packages whose dependencies are *all* from the Python 2 universe.

> Regarding the qt packages: I updated them on staging, but still have
> Python 2 in there. I will take care of these.

Excellent. Lots of other packages depend on those, so that will be a big
step towards Python 2 independence.

LLVM/clang looks like another important building block to liberate from
Python 2. I hope someone can take a look at that.

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Bengt,

> IOW, a bunch just differ by version -- I wonder how many of the
> packages that drew in old versions could run fine with respective
> latest versions of what they are dependent on?

That's a very good question!

> It would be really interesting if you could tweak your py2-dependent-package
> lister to show for each how many lines of py2 code are causing the py2 
> dependency!

That's a much harder job, and I doubt it can be automatized. What I
analyze is the package definitions in Guix. They don't provide any
information on how the package uses its dependencies. They merely say
that the packager has declared some dependencies.

BTW, the number of lines of Python 2 code doesn't really matter either.
What we need to know is (1) can the Python 3 dependencies be replaced
by equivalent Python 3 dependencies and (2) are these dependencies
essential, or do they merely enable some minor extra (such as a Python
API for a C library). In the second case, the package could be split
into two.

> It would be a shame if half a dozen lines of python were pulling in
> all that weight if it could have been a few lines of guile or bash instead.

That's yet another question: could we patch the upstream code to replace
Python by something else that's more convenient for Guix. That may
actually be a worthwhile approach to reduce software bloat in Guix,
but it also shifts some of the maintenance burden from upstream to Guix.

> The time it takes to figure out whether/how a trivial dependency can be
> eliminated is, ISTM, a major cause of gordian-knot dependency bloat -- which 
> IMO
> in turn is a kind of RYF threat: it's effectively a free-time
> pay-wall.

That's certainly true but on the other hand there is "never break a
working system". So there is a trade-off between long-term
maintainability and short-term reliability.

Guix might well contribute to a long-term improvement of everyone's
habits in this respect, by providing the bird's-eyes view of software
systems that developers and maintainers of individual packages cannot
have. What you describe is just the application of well-known ideas from
software engineering (cleaning up code before it becomes unmaintainable)
to the systems level. Which is possible because Guix turns the systems
level into just another software layer.

Cheers,
  Konrad.

Re: Store channel specification in profile

[Konrad Hinsen]

Pierre Neidhardt  writes:

> In this case, how would you intend to use guix time-machine to reproduce
> these profiles?

"guix time-machine" and inferiors are different ways to access specific
Guix versions. "guix time-machine" simply runs a different Guix version.
You can then use it to access inferiors, if that versions supports them.
Mixing both is probably confusing for users, so I'd recommend not to do
it, but technically it is possible.

Assuming that we end up storing channel specifications for each package
in a manifest, we could then create a manifest file automatically that
re-generates the same profile using inferiors. No need for "guix
time-machine".

Cheers,
  Konrad.

Re: Python 2 end-of-life?

[Konrad Hinsen]

Konrad Hinsen  writes:

> I'd say the very first thing we should do is look at all non-Python
> packages that depend indirectly on Python 2.

Here is an attempt at identifying them:

;;
(use-modules (guix packages)
 (gnu packages)
 (srfi srfi-1)
 (ice-9 format))

(define (in-python2-ecosystem? package)
  (string-prefix? "python2-" (package-name package)))

(define python2-dependent-packages
  (fold-packages (lambda (package result)
   (cond ((in-python2-ecosystem? package)
  result)
 ((any in-python2-ecosystem?
   (filter package?
   (map second
(package-direct-inputs package
  (cons package result))
 (else result)))
 '()))

(for-each (lambda (package)
(format #t "~a~%" (package-full-name package)))
  python2-dependent-packages)
;;

I find 313 packages (see list below). A few of them are still Python
stuff (has "python" in the name but not "python2"), but most of them
look like packages that are not themselves Python libraries.

Some of these packages are real heavyweights (llvm, texlive, qtwebkit,
...), so I'd say each package deserves a bug report of its own if we
agree that a dependency on Python 2 should be considered a bug for a
non-Python package.

Cheers,
  Konrad.


xcb-util-errors@1.0-1.5d660eb
xpra@2.5.3
arandr@0.1.9
keybinder@0.3.1
polybar@3.4.0
wicd@1.7.4
woof@2012-05-31
linkchecker@9.4.0
wabt@1.0.12
xen@4.11.1
criu@3.11
bubblewrap@0.3.3
ffmpeg@4.2.1
youtube-dl-gui@0.3.8
handbrake@1.2.2
ffmpeg@3.4.6
git@2.24.0
git-annex-remote-hubic@0.3.1
cvs-fast-export@1.45
miniupnpc@2.1.20190824
miniupnpc-monero@2.1-monero-0.12.3.0-0.6a63f99
tryton@4.6.2
tor@0.4.1.6
lyx@2.3.2-2
texlive-bin@20180414
rubber@1.1
ceph@13.2.6
pspp@1.2.0
sdcc@3.7.0
scribus@1.5.5
zn-poly@0.9.1
rust@1.28.0
rust@1.26.2
rust@1.21.0
rust@1.20.0
rust@1.29.2
rust@1.31.1
rust@1.30.1
rust@1.32.0
rust@1.35.0
rust@1.27.2
rust@1.23.0
rust@1.36.0
rust@1.24.1
rust@1.25.0
rust@1.34.1
rust@1.37.0
rust@1.33.0
rust@1.22.1
rrdtool@1.7.1
qtwebkit@5.212.0-alpha3
qtmultimedia@5.11.3
qt@4.8.7
qtbase@5.11.3
qtdeclarative@5.11.3
ptpython2@0.34
python-rope@0.11.0
bpython2@0.18
python-cookies@2.2.1
pulseaudio-dlna@0.5.2-1.4472928
pdfposter@0.6.0
impressive@0.12.0
stapler@0.3.2
patches@0.0-1.ef1b8a7
openbox@3.6.1
ocaml-dose3@5.0.1
gourmet@0.17.4
node@10.16.0
openvswitch@2.12.0
gtklick@0.6.4
solfege@3.22.2
mod-host@0.10.6-3.1726ad06b
beast@0.10.0
lilypond@2.19.80
mono@4.4.1.0
pidgin@2.13.0
hexchat@2.14.2
bitlbee@3.5.1
pybitmessage@0.6.3.2
petsc-complex-openmpi@3.11.2
flann@1.8.4
slepc-complex-openmpi@3.11.1
slepc-openmpi@3.11.1
petsc-openmpi@3.11.2
slepc@3.11.1
sundials@3.1.1
sundials-openmpi@3.1.1
mlucas@18
petsc-complex@3.11.2
slepc-complex@3.11.1
lapack@3.7.1
petsc@3.11.2
atril@1.22.0
mate-applets@1.22.0
pluma@1.22.0
mate-menus@1.22.0
hoedown@3.0.7
alot@0.5.1
offlineimap@7.2.4
claws-mail@3.17.4
opensmtpd-extras@5.7.1
postorius@1.0.3
ghmm@0.9-rc3-0.2341
kaldi-gstreamer-server@0-1.1735ba4
lci@0.11.2
clang-runtime@8.0.0
clang@3.9.1
clang@6.0.1
emacs-clang-format@8.0.0
clang-runtime@3.9.1
clang@8.0.0
clang-runtime@3.7.1
clang-runtime@3.8.1
clang@7.0.1
llvm@8.0.0
clang-runtime@3.6.2
clang@3.7.1
llvm@6.0.1
emacs-clang-rename@8.0.0
llvm@7.0.1
llvm@3.6.2
clang-runtime@3.5.2
clang-runtime@7.0.1
clang@3.8.1
clang-runtime@6.0.1
clang@3.6.2
llvm@3.8.1
llvm@3.9.1
clang@3.5.2
llvm-for-extempore@3.7.1
llvm@3.7.1
llvm@3.5.2
crda@3.18
libnl@3.5.0
perf@5.3.8
iotop@0.6
tegaki-wagomu-simplified-chinese@0.3
tegaki-wagomu-japanese-kyoiku@0.3
tegaki-wagomu-japanese@0.3
tegaki-zinnia-japanese-light@0.3
tegaki-zinnia-simplified-chinese@0.3
tegaki-wagomu-traditional-chinese@0.3
tegaki-zinnia-traditional-chinese@0.3
tegaki-zinnia-traditional-chinese-light@0.3
tegaki-wagomu-japanese-joyo@0.3
tegaki-zinnia-japanese@0.3
tegaki-zinnia-japanese-joyo@0.3
tegaki-zinnia-japanese-kyoiku@0.3
tegaki-zinnia-simplified-chinese-light@0.3
kodi@18.4
kodi-wayland@18.4
key-mon@1.17
kfilemetadata@5.55.0
julia@1.1.1
inkscape@0.92.4
vigra@1.11.1
mcomix@1.2.1
mia@2.4.6
ghc@8.0.2
ghc@7.10.2
chirp@20181205
graphene@1.6.0
libdbusmenu@16.04.0
dot2tex@2.9.0
openimageio@1.7.19
openimageio@1.8.17
rapicorn@16.0.0
icecat@68.2.0-guix0-preview3
mozjs@52.0-1.6507e63
conkeror@68.2.0-guix0-preview3
mozjs@24.2.0
mozjs@38.2.1.rc0
mozjs@60.2.3-2
mozjs@17.0.0
pius@2.2.7
gnurl@7.63.0
gnunet@0.10.1
totem@3.30.0
gcr@3.28.1
rhythmbox@3.4.3
gnome-doc-utils@0.20.10
terminator@1.91
bluefish@2.2.10
glade@3.22.1
evince@3.34.1
gnome-keyring@3.28.2
deja-dup@34.3
caribou@0.4.21
gnumeric@1.12.45
telepathy-glib@0.24.1
gimp@2.10.12
osm2pgsql@0.96.0
gnubackgammon@1.06.002
gnubg@1.06.002
golly@3.2
slingsh

Re: Store channel specification in profile

[Konrad Hinsen]

Hi Pierre,

> One question arises though: channel specifications only make sense for
> profiles generated with manifests.

Not even for those, if the manifest uses inferior-packages. I'd go for
per-package channel specifications. They could be optimized (more
compact, more efficiently usable) by first storing a list of all
distinct channels and then a (package . channels) list.

Cheers,
  Konrad

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

I'll start from the end:

> What do we disagree on, actually?  :-)

This:

>> 2. Power users will always write code in powerful languages that exceed
>>what less advanced users can deal with. And since power users are not
>>necessarily benevolent, this creates a trust issue for the less
>>advanced ones.
>
> Good point.  I tend to (naively?) view it the other way around: that it
> gives people an incentive to try and write actual code rather than mere
> declarations.

I'd say we should encourage people to write declarations as much as
possible and resort to executable code only when declarations become
too messy. As a corollary, we should support most configuration-style
use cases with suitable declarative forms, much like these:

> The goal for Guix was to have the ‘package’ and ‘operating-system’
> forms, for instance, look exactly like what you’d write in JSON etc.,
> only with a different syntax.

For better illustration, I'll try to rewrite my own manifests in the
way I'd like to be able to write them. That's probably more useful
than theory (a tough statement to make for a theoretician ;-)

The main reason why I want to see more declarative style is:

>> The problem with powerful formal languages (read: Turing-complete) is
>> not writing, but (1) debugging and (2) reading.
>
> Yes and no.  Guile has a debugger, whereas XML/JSON/YAML don’t.  As for
> reading, it’s subjective, but I don’t think a full-blown language is
> necessarily harder to read.

It's harder to read because you need to understand the language's
execution model if there is one. YAML etc. don't, there is nothing
but literals. Which is also why they don't need a debugger.

>   (define lst (list 1 2 3))
>
>   lst = [1, 2, 3]

Fine. But then a power user comes along and writes

   (define lst (cons* 1 2 '(3)))

That may be bad style, but as a reader I have to be able to deal with it
nevertheless. And bad style may in fact serve to obfuscate evil
intentions.

> Interesting!  The conclusion you seem to draw is that embedded DSLs as
> found in Racket fit the bill, and to me, that’s pretty much what we’ve
> been trying to do with Guix as well, no?

Indeed, just not enough for my taste!

Cheers,
  Konrad.

An occasion to demonstrate Guix for reproducible research

[Konrad Hinsen]

Hi Guix,

For those of you working in scientific research, here's an occasion
to show the utility of Guix to the world: the ten-year reproducibility
challenge run by the journal ReScience. You can find the announcement
below.

The main role I see for Guix in this challenge is to ensure
reproducibility for the future, possibly resurrecting old versions of
packages in the process. I proposed two reproduction attempts myself for
this challenge, and I intend to package all dependencies in Guix
and add instructions to my code on how to reproduce it exactly
using "guix time-machine".

Cheers,
  Konrad


Did you ever try to run an old code that you wrote for a scientific
article you published years ago? Did you encounter any problems? Were
you successful? We are curious to hear your story. This is the reason
why we are editing a special issue of ReScience to collect these
stories.

The Ten Years Reproducibility Challenge is an invitation for researchers
to try to run the code they’ve created for a scientific publication that
was published more than ten years ago . This code can be anything
(statistical analysis, numerical simulation, data processing, etc.), can
be written in any language and can address any scientific domain. The
only mandatory condition to enter the challenge is to have published a
scientific article before 2010 , in a journal or a conference with
proceedings, which contains results produced by code, irrespectively of
whether this code was published in some form at the time or not.

Note that we do not ask you to write a new version of your old code. We
ask you instead to try to make your old code to run on modern
hardware/software (with minimal modifications) in order to check if you
can obtain the exact same results that were publised at least ten years
ago.

Sounds easy? We have good reasons to think this might be more difficult
than you think. And maybe the first problem you’ll have to solve is to
find your own source code.

More information at: http://rescience.github.io/ten-years/ 

Re: Python 2 end-of-life?

[Konrad Hinsen]

Hi Simon,

> What do you do?

Well, me, personally, I continue to do most of my research using Python 2
because I cannot afford to port everything to Python 3. And since I do
only number crunching, meaning nothing with security implications, I am
not particularly worried.

> Do we deprecate the python2 packages? If yes, what would be the
> schedule? If no, do we move all the python2 packages to a
> python2-xyz.scm file?
> Do we do nothing? Based on what rationale?

I'd say the very first thing we should do is look at all non-Python
packages that depend indirectly on Python 2. I remember during a recent
installation on a virtual machine that my very first package install,
which is always Emacs, downloaded Python 2 among many others, so there
remains some work to be done.

Once Python 2 lives in a largely isolated package sub-universe, I don't
see much harm keeping it in Guix for now. If security issues become
apparent, we might have to do something more drastic.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Simon,

> So, you propose to "enrich" the DSL describing the manifest files, right?

Perhaps. I suppose there are several possible approaches, and I don't
claim to know what works best before actually trying.

One way is to "harden" manifest files, providing richer constructs for
defining and transforming packages but putting them into a
sandbox. Another way is to push as much information as possible into
channels and make manifests little more than lists of package
references.

Cheers,
  Konrad.

Re: guix pull failed after 8 hours

[Konrad Hinsen]

Hi Ludo and Chris,

> Could it be that those old systems were talking to hydra.gnu.org and
> thus not getting any substitutes?
>
>   https://lists.gnu.org/archive/html/info-guix/2019-06/msg1.html

That's what happened to me on my oldest Guix installation (hosted on
Ubuntu). I had done regular pulls on my user account, but not on the
root account, meaning that I was running an ancient daemon that tried to
connect to hydra.

Fix (reconstructed from memory!): on the root account, do

 guix pull –substitute-urls=https://ci.guix.gnu.org
 guix package update

and reboot.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Ludovic Courtès  writes:

> To me it’s not entirely clear that a unified command would be easier to
> use for newcomers.  For example, I’m not fond of “guix profile” as a

Indeed. It's also not clear to me if a single command line package
for beginners and power users is the best way to go. Maybe we should
have two levels of commands, with tutorials using only the simpler one.

> So I definitely agree we need to homogenize what ‘guix environment’ and
> ‘guix package’ currently provide, but I’m not sure where to go from
> there.  Perhaps we need to discuss on concrete CLI proposals to have a
> better idea?  Dunno!

If enough people are willing to work on this (beginners included!), we
could run a "CLI working group" that plays with alternatives to the
current CLI, implemented as a separate Guile package so it won't perturb
business as usual. That requires a few people working on implementation,
and a much larger group willing to test-drive the design(s) and provide
feedback.

> I have an actionable wishlist though:  :-)
>
>   1. Arrange so that ‘--ad-hoc’ becomes the default in ‘guix
>  environment’, and add a ‘--devel’ (?) option for what’s currently
>  the default.

That sounds good to me, with

>   2. Add that same ‘--devel’ option to ‘guix package’.

being the obvious (and less controversial) complement.

I'd probably prefer the more descriptive –-inputs-of for the option name.

>  Tricky because of compatibility considerations, but would be an

That's the eternal question!

>>  - Better support for adding/managing channels at the user account
>>level. Users shouldn't have to edit Guile code (unless they want to).
>
> You mean like ‘guix channel add URL’?  Are you looking at this from a UI
> perspective?

Yes.

> Are you also suggesting a “GuixHub” (I already hate that name :-)) that

That's not something I personally care about (I am all for the
distributed Web), but if there's a demand, why not.

>>  - Support for creating and managing channels without having to
>>descend to the file and repository level.
>
> Not entirely sure what you mean.  Could you give examples of what you’d
> type to create a channel, say?

For management, something like git's remotes:

 guix channel add my-channel URL
 guix channel remove my-channel
 guix channel describe my-channel
 guix channel show-all

For channel creation, something like git's init:

 guix make-channel directory-name

That would create the directory name, run "git init", add minimal code
making a template (something like the "hello" package), and perhaps a
file with a summary of instructions.

>>  - Support for something one could call super-packages or named
>>manifests. Named package groups that live in a channel (or a guix.scm
>>distributed with software external to Guix). Perhaps parametrizable
>>and/or containing configuration files, so you could put something
>>like (nginx #:port 8080) into a manifest file.
>
> So this thing would be a profile + services?

A bit less than a service, because no process would be started or
managed. I might want to run that nginx instance manually and/or
under my user account. Also on a foreign distribution that doesn't
know about Guix services. Think of something like a Jupyter server.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> I’d like to think that writing Guile declarations for the OS config,
> manifest, etc. is not just for “power users”.  After all people, or
> rather “computer-savvy” people in a broad sense, write JSON, YAML,
> custom config files etc. routinely, and I don’t think the typical config
> we propose is any “harder”.  You may say I’m a dreamer, but I’m not the
> only one. 텞텟  ;-)

The problem with powerful formal languages (read: Turing-complete) is
not writing, but (1) debugging and (2) reading.

1. Writing a manifest file in Guile is no harder than writing a list
   in YAML or whatever. But leave out the closing quote behind a
   package name, and you will get an error message that will make
   no sense to someone unfamiliar with the *complete* Scheme syntax.
   For a small and simple file, you can usually spot the problem by
   inspection (i.e. without the error message), but for more complex
   files, it makes sense to use a more constrained language in order
   to provide better error reporting.

   BTW, the Racket team makes that point to argue for their rather
   complex macro system. It takes much longer to master than traditional
   Lisp or Scheme macros, but it supports writing macros with error
   reporting that makes sense to someone who didn't read the macro code.

2. Power users will always write code in powerful languages that exceed
   what less advanced users can deal with. And since power users are not
   necessarily benevolent, this creates a trust issue for the less
   advanced ones.

For a long version of these arguments, see
   https://hal.archives-ouvertes.fr/hal-01966145/document

> I think we need to focus on specific scenarios though.

Definitely!

> IOW, users of a channel have to trust it to not be malicious,
> regardless of the fact that its Guile code runs unrestricted.

Yes. That's perhaps something that the manual should point out
explicitly. Also, a more specific user interface ("guix channel add
URL") could show a warning.

> For manifests shared over the net, the situation may be different: a
> manifest could refer to packages in the channels you trust, and thus
> there’s value in not having to trust the manifest code itself.

Exactly, and that's the idea that got me into this thread.

> It’s still a bit too abstract, but for the purposes of sharing and
> publishing “super packages” as you wrote, we could define a
> purely-declarative format (could be JSON, could be Guile code that can
> run under (ice-9 sandbox) or with ‘eval/container’) that people could
> use instead unrestricted as is currently the case.

Yes, that's one way to go. BTW, I didn't know about eval/container,
I'll have to look at it!

Cheers,
  Konrad.

Re: A better XML, config is code (was Re: Profiles/manifests-related command line...)

[Konrad Hinsen]

Hi Giovanni,

> The real question is: a configure file is code or data?  IMHO is code,

Code is data with execution semantics, so "code" is a subset of "data".

I'd reformulate the question as: should configuration data be
literal data, or the result of a computation? The second opton
is more general, and therefore more powerful. If that is good or bad
depends on the application. If you are writing the configuration, you
appreciate more power. If you use someone else's, you might well prefer
it not being more powerful than what you can understand.

> Mumble... but every user *is* a power user when installing and
> configuring a system, no?

"Is", no. That would assume that everybody knows their limits. Not true
in my experience with human nature.

> ...so yes, if it's not a channel under your control - or of someone you
> decide to trust - you should better not use it (and do not copy/paste
> configuration files you do not understand)

Fine with me, but then we should (1) say so somewhere in the manual and
(2) not recommend using such configuration files for performing tasks
that ought to be accessible to ordinary users.

> I recently read this "Curl to shell isn't so bad" article (thanks ARota)
> https://arp242.net/curl-to-sh.html
>
> «In the end it’s still just running code you didn’t personally audit on
> your computer, and a matter of trust.»

Exactly. It's OK for us to ask users to trust the Guix team, which they
have to do anyway if they use Guix. So configuration files provided by
Guix itself are not a problem.

But if we tell people that Guix is great for reproducing someone else's
computation, and that the best way to share a computation is publishing
a manifest file, then we are encouraging people to run code from
untrusted sources. Which leaves three options:
 1. Provide a safe way to re-create environments from untrusted
sources.
 2. Don't recommend reproducing someone else's computation using Guix.
 3. Explain why reproducing someone else's computation is
a risky procedure that should be reserved to power users.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Andy,

> I wrote this for that purpose:
>
>   
> https://www.gnu.org/software/guile/manual/html_node/Sandboxed-Evaluation.html

Right, I had found this when searching for something. It seems to solve
a couple of problems that I don't quite understand, but not so much
those I do (file/network access). Would be nice to see this extended.

> In practice Guix's "containerized" build jobs are much more effective
> than in-language barriers.

Indeed, but if Guix is compromised by malware, the build jobs may build
code that has already been tampered with.

Maybe one could have config and manifest files interpreted by the build
daemon for safety. Except that some manifest files (read: mine) need
read access to the file system.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> Of course, using a general-purpose language upfront also comes at a
> price, as you note.  But I think that what it has to offer to users
> outweighs the costs, and that’s a lesson learned from Emacs.  Just to
> say I’m not willing to replace ‘config.scm’ with ‘config.yaml’, if
> that’s what you had in mind.  :-)

YAML is for kids. Real managers won't settle for less than full XML. ;-)

Seriously, as a power user, I am perfectly happy with Guile for
everything. I certainly don't want less. And for now, it's safe to
assume that most Guix users are power users. The question is if we want
Guix to remain exclusively a power tool for power users. If not, we need
to make sure that it won't become a malware platform, by making it safe
to use for people who don't read Guile code. In particular, common use
cases should not require users do download unrestricted Guile code from
untrusted sources.

Emacs is an interesting comparison in many ways, but also a much less
interesting target for malware than Guix. An attack on Guix can
undermine all the guarantees it provides through reproducible builds.
Maybe Ken Thompson should do an update of his famous "Trusing trust"
that extends the discussion of compilers to build tools in general.

One direction could be to add a sandboxing feature to Guile, which would
be nice-to-have for other uses as well if Guile is to become a
general-purpose systems scripting language. There are some interesting
ideas in shill (http://shill.seas.harvard.edu/) for this scenario.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> I agree this is an important use case.  It seems to me that the problem
> here is being able to aggregate more than just packages.

Yes, it's a bit more than that. We'd probably have to look at a couple
of use cases to see what the most general content would be.

> The Web application example above seems to be somewhere between a mere
>  and an full-blown .  Perhaps we need an
> intermediate abstraction that bundles together a profile along with some
> lightweight services?
>
> Or, to put it differently, how do we define “super-package”?

That's the question. I don't pretend to have an answer, I just wanted to
raise the question.

There are use cases, such as Web servers, that move towards the service
or even system abstraction. Other use cases would be just packages
plus configuration, with no automatic execution implied.

> ‘guix package --list-profiles’ was added to improve this situation where
> there’s no enforced convention.

Unfortunately it makes no distinction between package profiles
and channel profiles, so it's hard to use in scripts.

> The question seems to be whether we should add more convention and less
> configuration.

Or have all configuration in a single file per user, with convenient
defaults encoding a set of conventions. Trying to please everyone :-)

Right now the only conventions I can think of are the two default
profile names (~/.guix-profile for packages and ~/.config/guix/current
for channels). And I wonder BTW if the latter is not an abuse of
~/.config, given that it contains a ton of software rather than just
small configuration files, but that's a different question.

> We should boil that discussion down to a list of things to implement.
> :-)

Definitely. But I'd like to start by compiling a list of use cases,
to see how well they are served by what we have and then how they
could be served better.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Simon,

>> I can of course share a manifest file by putting it in a public
>> repository. But that isn't necessarily the best way.
>
> From my opinion, it is not the problem of Guix. Guix should not force
> a practise, as good it should be. The way the users share manifests is
> the problem of users, not the Guix's one, IMHO.

Guix should support good practices in my opinion. Sharing files that can
contain arbitrary code (such as manifests) is not a good practice, for
security reasons. For personal use, on the other hand, that flexibility
is very valuable.

> From my opinion, Docker tutorials seems really easy because 1.
> everything is centralized and 2. they cheat (it is hard to replay a
> random tuto couple of months later; from my experience.)

I agree. I certainly don't want Guix to follow Docker in everything.
We can always do better.

> From what I understand, "guix time-machine" is the answer to this kind
> of scenario. Right?

Yes, in combination with "guix describe".

> Personally, I do not like the "directory defined by convention and
> referring to them by name" approach. The "package manager" Conda does
> that and after some years using it (before switching to Guix \o/),
> this approach appears to be also a mess. I never remembered the name

That's valuable feedback. I have never used conda myself systematically,
just a few times for following tutorials.

> I prefer the Guix way and I store the profile (symbolic link) in the
> project directory. This way, I always remember the name, I can quickly
> create another one with a cryptic name -- naming is hard :-) -- and
> weeks later I remember that it is attached to this project.

That's how I work as well. But Pierre's recent document convinced me
that profiles can also be a good way to modularize the global software
collection of my account. It would indeed be nice to be able to update
some parts (e.g. scientific software) more often than others
(e.g. TeXLive).

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Pierre Neidhardt  writes:

> By the way, the same should be possible from the installer:
>
> - Specify which channels you want to use.
> - Specify which config.scm you want to use.  Indeed, the config.scm
>   might rely on channels.
>
> This way, the Guix installation process would boil down to the above 2
> items, that is to say... 1 minute of the user time!  Great, no?

This does open some interesting possibilities, such as experts defining
personalized system configurations which are then easy to install for
anyone. Although I am not terribly fond of the idea of typing URLs into
an installer.

The bigger issue is config.scm - again unrestricted Guile code, like
manifests. That's not good for publishing because we shouldn't encourage
anyone to run unrestricted code from untrusted sources.

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Ludo,

> I think having ephemeral + persistent and declarative + imperative is
> cool—I’d call that “flexible” rather than “messy”.  :-)

I agree. What's messy is how the concepts map to commands. How many Guix
users understand that profiles are persistent environments, or
environments ephemeral profiles? And how many understand what "guix
environment -r" does exactly, and how it should be used?  It took me a
while to figure this out.

What we have is two commands (package and environment) each designed for
one main usage pattern, plus options to get something else. But even
those options don't overlap completely in functionality. For example,
how do I make a profile with the dependencies of a package?

The current discussion started with adding more commands for different
specific usage patterns. If we go that route the mess will become worse.

> It’s great to have “guix install” as an entry point; I’m sure it allows
> us to reach out to more people, and that matters too.  (I actually use
> it, BTW, so it’s not an expert vs. newbie thing!)

Me too :-) It's "guix package" that is the worst offender in my
opinion. It does two distinct things: querying the package database and
managing profiles. And now that we have "guix search" for queries, I'd
like to see "guix package" go away, to be replaced by either "guix
profile" for profile management, with as much overlap as possible in
options with "guix environment", or by a single command that handles
environments and profiles in a unified way.

> I agree that sharing and publishing is important, and I think
> manifests support that.

They do, but not very well in my opinion. I think everything meant to be
shared, published, and maintained should be accessible by name in a
database. A channel, for example.

Some ideas that could make this possible (and convenient):

 - Better support for adding/managing channels at the user account
   level. Users shouldn't have to edit Guile code (unless they want to).

 - Support for creating and managing channels without having to
   descend to the file and repository level.

 - Support for something one could call super-packages or named
   manifests. Named package groups that live in a channel (or a guix.scm
   distributed with software external to Guix). Perhaps parametrizable
   and/or containing configuration files, so you could put something
   like (nginx #:port 8080) into a manifest file.

> I think we need to support “unions” of manifests, and
> that means (as always) supporting it at the API level and at the
> command-line level (allowing for multiple ‘--manifest’ flags).

Yes, I agree. That's the easiest part.

> What we’re now saying is “look, you can write a manifest, and then you
> can have it under version-control and publish it and it’s all up to you
> how you do that”; but you seem to suggest that we should offer a
> higher-level, more integrated solution, is that correct?

Exactly. Every functionality that requires end-users to manage
Guile code will always be restricted to expert users. Manifest
files may look simple and understandable for simple cases,
but if you expect users to install manifests downloaded from
someone else, they need to be able to inspect them and be sure
that installing the manifest file won't delete their user account.
And that means they have to understand a lot about Guile.

> Like, we would enforce certain conventions by default, perhaps have
> direct Git integration so that one can refer to a “manifest” just like
> they refer to a channel, things like that.  Do I get it right?

Yes, that's the idea!

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Simon,

> I am not sure to see which feature is missing.

My main point is that the future evolution of functionality (and user
interfaces) in this space should take into account usage scenarii,
rather than adding features in an ad-hoc fashion.

I can of course share a manifest file by putting it in a public
repository. But that isn't necessarily the best way.

Take the typical example from Docker tutorials: bundling a Web server
with some Web application and a database. It's easy to make a manifest
file for collecting the required packages. But it would make more sense
to me to have a module in a Guix channel that defines a parametrizable
super-package for the Web application that has no source code of its own
but various inputs and perhaps configuration files. Users can then
install several Web apps bundled in this fashion, sharing the Web
server. This kind of composition is not possible (currently) with
manifest files.

> From my point of view, ephemeral (environment) vs persistent (profile)
> depends on use-case but at the end the only concern is: how to
> populate them? And the only answer should be: declarative (manifest);
> the ad-hoc (package -i) should not be the usual way but only a
> quick test.

I agree. So maybe each profile should have an associated manifest file,
with "guix install" merely adding to it and the updating the profile.
But then it would make sense for guix to manage manifest files in git
repositories by default, to encourage good habits.

>> This third dimension also
>> raises the question of where the information (profiles, manifests, ...)
>> are stored and managed (version control?),
>
> Profiles are managed by Guix, isn't it?

Sure, but how exactly? Right now, a profile is a directory anywhere in
the file system that Guix knows about. Recent discussions have proposed
alternatives, such as keeping all of a user's profile in some directory
defined by convention and referring to them by name. What's the better
way to use as a default? I don't know, but I think we should discuss it
rather than adding new sub-commands with different behavior and thus
adding to the mess.

> Manifests are managed by the user. And they does what they wants. ;-)

True again, but again, is that the best way to do it?

Cheers,
  Konrad.

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Hi Hartmut,

> This becomes even messier if we would implement a "guix develop"
> command, which is yet just another version of environment/profile.

Yes, I worry also about making more of a mess by implementing
special-purpose variants.

> And adding another dimension: spawning a sub-shell (environment) or not
> (profile).

How is this different from the ephemeral vs. persistent dimension?
Creating an ephemeral package set makes sense only if you spwan a
process in it (not necessarily a shell).

Cheers,
  Konrad

Re: Profiles/manifests-related command line interface enhancements

[Konrad Hinsen]

Pierre Neidhardt  writes:

> I'm actually surprised you find it surprising! :)
> I can think of Simon, maybe Konrad(?) and myself who mentioned it
> before.

Yes, me too. I could add to Pierre's list of use cases, but I prefer to
shift the discussion to a higher level.

What we have been discussing here recently is the organization of
software one level above packages. The vague idea is "groups of packages
that go together". Outside of the Guix universe, this is the realm of
(Docker) containers. A quick look at what happens in that universe
shows that composing such groups of packages corresponds to a frequent
need (see docker compose, Kubernetes, ...).

In Guix we have ephemeral (environment) vs. persistent (profile), ad-hoc
(package -i, environment from package lists, ...) and declarative
(manifests). It's a bit of a mess. Hacks such as "environment -r" show
that we ought to think about a better overall design. In addition to the
two dimensions I mentioned, this should include shareable/re-usable
vs. personal. People publish and share Docker images, so why wouldn't
they publish and share Guix super-packages? This third dimension also
raises the question of where the information (profiles, manifests, ...)
are stored and managed (version control?), and how they are referred to
(name, filename, ...). And of course how they are composed - in Guile,
at the shell level, or yet something else?

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Konrad Hinsen  writes:

>> Looking forward to a patch!  :-)
>
> If all goes well, next week !

Done:

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37978

Cheers,
  Konrad.

Re: “Guix Profiles in Practice”

[Konrad Hinsen]

Hi Pierre,

>> Jumping to symbol definitions, for example, or access to documentation.
>
> I can jump to definition with no problem.  Is your load path set
> properly?

After inspection... no. At least not in all my Guix installations.
It does work fine with the right path.

> Access to the manual has been broken forever apparently.
>
> Work-around here: https://gitlab.com/jaor/geiser/issues/252

Great, thanks! The biggest improvement in my Guix experience for a
while!

Cheers,
  Konrad.

Re: “Guix Profiles in Practice”

[Konrad Hinsen]

On 27/10/2019 12:30, Pierre Neidhardt wrote:

Another talk could be about programming Guix from the REPL. Guix as a
DSL if you will. Anyone interested in doing a talk about that?

I would be very interested if someone could explain how they hack with
Geiser: I'm still frustrated with it on a daily basis :p I'm in
particular interested in debugging.


I thought I was the only one having difficulties with Geiser and Guix. 
Plenty of Geiser features that work fine with small programs seem to 
fail with Guix, perhaps because of the sheer size of the codebase. 
Jumping to symbol definitions, for example, or access to documentation.


Konrad.

Re: “Guix Profiles in Practice”

[Konrad Hinsen]

Pierre Neidhardt  writes:

> "Thompson, David"  writes:
>
>> if it's a good idea.  Probably not.  So, I wonder if maybe a new
>> subcommand, say 'guix develop', could address this common development
>> use-case while allowing 'guix environment' to continue being the swiss
>> army knife that it is.  Some simple naming conventions could make this
...

> This is essentially what you propose, but to have it at the CLI level
> would provide several benefits:
...

> Both issues could be addressed the same way.

Perhaps. But then, "could" doesn't imply "should". I'd love to see all
that functionality easily accessible from the CLI, but I can't say I
know how this should best be organized.

Maybe we should start a Guix CLI nursery. A project/repository separate
from Guix itself that contains a copy of the "guix" script under a
different name ("guixx" for guix-extras?) and with the same interface
for scripting modules. We could then use this to play collectively with
ideas, and if something turns out to work well, migrate it to the
official Guix CLI.

Does this sound like a good idea? Would anyone else participate in such
an experiment?

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Pierre,

> In the meantime I've played with Guix + Emacs and wrapped Guix CLI in
> hopefully a more convenient way that makes it easy to track the channel
> specifications for various manifests, to reproduce them, etc.
>
> See https://gitlab.com/emacs-guix/emacs-guix/issues/13, maybe that'll
> inspire you :)

That looks like a layer on top of the Guix commands which does
profile/manifes management based on a few conventions. Nice!

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> I see.  In a way one could argue that it’s not Guix’ problem, but OTOH
> it’s clearly a problem that Guix doesn’t make it more convenient.

I don't know much about Guix' problem, not being a software
psychologist, but in the meantime I am trying to solve a
problem that some Guix users have :-)

> That sounds reasonable to me: it’s a simple and easily-understood UI
> (easier than the ‘guix pull -p’ dance!), and the implementation should
> be easy (thus the maintenance cost is very low).

Indeed.

> Looking forward to a patch!  :-)

If all goes well, next week !

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

zimoun  writes:

>> No, at least not explicitly. My goal is reproducing computations from
>> the past, so I need to re-animate old manifest files. These could of
>> course contain references to inferior-packages, so they could be
>> multi-commit, but this is not my focus.
>
> But does it fit with the current practice?

With mine, yes :-)

Installing packages one by one into my profile is something I only do
for quick tests. All my profiles are defined by manifest files under
version control, and occasionally updated as a whole.

If there were a straightforward way to save profiles to a manifest
file, I might be less strict about this procedure, though I don't find
it constraining in practice. It fits well with my older habit of having
all configuration files under version control.

> I do not know if it is a way to go: allow profile/environment
> populated with binaries from different commits (without speaking about
> possible clash.)

I'd say it's doable, but so far not implemented. Restoring such an
environment is likely to be expensive, as you have to download multiple
Guix commits and then maybe compile dozens of versions of old compilers
etc.

Assuming that one day in the near future, all scientists use Guix and
routinely publish their manifest files, I'd expect reviewers to ask
authors to clean up their manifest files for the benefit of everyone
else. So I adopt this habit now before I am forced to ;-)

> This information is already available in one location:
> /manifest, I guess.

Indeed, but it's an undocumented feature for now. The format is
undocumented and not accepted by "guix package" or "guix environment".

> Pierre wrote an example to bridge the profile to the manifest.
> See https://ambrevar.xyz/guix-advance/index.html
> section Generate a manifest  (bottom)

Interesting, but it records only one commit, for the current Guix.
If you have older non-updated packages, this will go unnoticed.
I see this mainly as a tool for migrating from historical-accident
profiles to managed profiles, but that's perhaps just my own bias.

Cheers,
  Konrad

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Simon,

> I am not sure to understand everything, so my questions are:
>
>  - Do you consider binaries coming from multiple commits and/or
> multiple channels?

No, at least not explicitly. My goal is reproducing computations from
the past, so I need to re-animate old manifest files. These could of
course contain references to inferior-packages, so they could be
multi-commit, but this is not my focus.

>  - Why splits the channel file and the manifest file? I am thinking to
> improve the DSL of the manifest file.

I have thought about this as well, but I am not convinced this is a good
idea, for two reasons.

First, the manifest file is hand-written, the channel file is produced
by "guix describe". How would a combined file be created? And how would
it be updated? The two main kinds of update are 1) updating the software
and 2) changing the components of the environment. Today each part is
covered by one file, so there is no conflict. With a single file, would
"guix pull" have an option to patch in a new commit number?

Second, combining both informations would likely require a restriction
of what is allowed in a manifest file. For example, only allow package
specifications by name and version, without transformations. Otherwise
the implementation would be quite complicated. Imagine someone applying
a transformation using today's guix to a package from last year's guix
that didn't have a feature used in the transformation.

What a combined file would make sense for is saving a profile to a
manifest file. As far as I know there is no support for that at the
moment, but it could be useful.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> Did you define an environment along the lines of the example at
> ?
>
> If so, this should behave in exactly the same way as a “regular” ‘guix
> environment’, so I’m not sure where the issues regarding access to the
> tty come from?

I ended up finding an approach that does work, though it relies on
undocumented properties:

   (use-modules (guix channels)
(guix inferior))

   (define channels
 (list (channel
(name 'guix)
(url "https://git.savannah.gnu.org/git/guix.git;)
(commit
 "44881cad93801de9462d469500d582af79b99959"

   (define inferior
 (inferior-for-channels channels))

   (define inferior-executable
 (inferior-eval
  '(car (program-arguments))
  inferior))

   (define inferior-command-line
 (cons inferior-executable (cdr (program-arguments

   (apply system* inferior-command-line)

This script uses the inferior mechanism for creating and caching an
older guix, but then calls its executable directly.

Of course this idea could be implemented in a better way, by factoring
out the caching mechanism from inferior-for-channels and providing a
command-line interface. Something like

   guix time-machine channels.scm command args ...

Does this sound reasonable? I volunteer for implementing this, but I'd
like to have an expert opinion first.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> Did you define an environment along the lines of the example at
> ?

No. I did look at this approach briefly, but it looks difficult at
least, if not impossible.

The situation I need to address is re-creating an environment from
- a channel file (from "guix describe")
- a manifest file (typically hand written)

This should not require editing the manifest file, and it should work
for any manifest file, even if it included package transformations or
literal package definitions (yes, I do that, shame on me, I ought to set
up my own channel instead).

A manifest definition using only lookups by package name and version
would be easy to translate into a manifest containing instances of
inferior-package, but I don't see how to deal with the more exotic
cases. I'd have to construct corresponding instances of inferior-package
myself, which is perhaps possible, but not obvious.

> Depending on the use case, another option is to not use inferiors at all
> and simply do:
>
>   guix pull --commit=XYZ -p ./my-old-guix
>   ./my-old-guix/bin/guix environment --ad-hoc whatevern
>
> But maybe I’m missing something, let me know.  :-)

That was my starting point. It's OK for manual execution, but if I want
an Org-Mode file to this automatically, I need to generate a temporary
file in place of ./my-old-guix with all the associated hassles: check
that it doesn't exist before, and clean up in the end even in the case
of errors. Maybe I'd even have to worry about two processes accessing
the same temporary file and do some locking.

Moreover, I may end up having a performance problem with "guix pull"
being run again for every little script in my Org-Mode file. The big
difference between Jupyter and Org-Mode is that Jupyter has only one
kernel process per notebook, whereas Org-Mode by default has one process
per code block.

My dream would be to have something like

  guix environment -C channels.scm -m manifest.scm

with reasonable caching of the pulled channels. That's in fact what I
was trying to implement.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

>>> At the API level, there’s ‘inferior-for-channels’ which does that +
>>> registers a GC root + maintains a cache so that the second time you use
>>> a given instance of Guix it’s immediately available.
>>
>> Just what I need...
>
> Awesome, let us know how it goes!

Not so well...

If I run "guix environment" in the inferior, it's the inferior that
forks and runs the specified command in the newly created environment.
But the inferior doesn't have access to the tty, so this is bound to
fail for running interactive commands.

Unless there is some Linux magic that I am not aware of for transferring
the tty temporarily to another process, the fork would thus have to
happen in the main Guix process rather than in the inferior. But any
scheme I can come up with for creating a profile in the inferior and
using it in the main process ends up generating the same problem that I
am trying to solve: the need for a temporary reference (profile or GC
root) that must later be removed.

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> (Though printing addresses in a REPL isn’t “bad practice” IMO, it’s just
> that it doesn’t mesh well with the intended use of notebooks.)

And that's exactly the difference between a REPL and a reproducible
document. Unfortunately, Jupyter tries to be both and thus never
explains the differences clearly.

> At the API level, there’s ‘inferior-for-channels’ which does that +
> registers a GC root + maintains a cache so that the second time you use
> a given instance of Guix it’s immediately available.

Just what I need...

> Now, I think you can safely use it in your code, I don’t foresee any
> significant change at this point.
>
> As to whether you’d want to talk about it in the MOOC for instance, you

Certainly not. I just want to be sure that the functionality won't go
away. If the API changes a bit, that can be fixed.

I'll start playing with this...

Cheers,
  Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo,

> That reminds me of an interesting issue regarding
> bitwise-reproducibility that was raised on the Reproducible Builds
> mailing list:
>
>   
> https://lists.reproducible-builds.org/pipermail/rb-general/2019-September/001657.html

We ran into this problem as well in the Reproducible Research MOOC. It's
hard to test equality for notebooks. But we went for solution a) because
our main goal is to teach good practices for writers of future
notebooks, rather than patch bad practices of the past with an
additional layer of build tools.

>> It would be nice in fact to adapt the ideas behind Guix-Jupyter (and
>> perhaps parts of the code) to Org-mode. Some integration with Emacs will
>> be necessary to tell Org-mode to start Python etc. from the Guix
>> environment.
>
> I’d love to see that happen!  I thought perhaps we could trick Alex Kost
> or Pierre Neidhardt to hack on that, let’s see.  :-)

I might also do it myself because I suspect it would be less work to
write the required Emacs package than to explain how to live without it.
Tutorial-driven development :-)

What would be the best way to run code in a specific environment created
for recorded channels? The obvious approach would be

   guix pull -C channels.scm -p /tmp/temp-profile
   /tmp/temp-profile/bin/guix environment –-pure -m manifest.scm –-
   python script.py
   rm -rf /tmp/temp-profile

but doing that properly involves the usual messy precautions for dealing
with temporary directories, and it's probably expensive to do the "guix
pull" repeatedly for the same channel file just because the temporary
profile gets deleted immediately.

This looks like a use case for "guix inferior", but is that already
stable enough to be talked about in public?

Konrad.

Re: Towards reproducibly Jupyter notebooks with Guix-Jupyter

[Konrad Hinsen]

Hi Ludo and Simon,

Ludovic Courtès  writes:

> I’m happy to announce the first release of Guix-Jupyter!

This looks very good, even though I will probably have to rework my
reproducible-research-tutorial-with-guix tutorial now ;-)

I haven't looked at this yet in any detail, but I wonder how you deal
with Jupyter pluging (e.g. widgets). For reproducibility, they ought to
be taken from the notebook's specific environment, rather than from the
environment from which Jupyter and the Guix kernel are taken. But this
looks difficult to do, given Jupyter's messy architecture.


zimoun  writes:

> To me, it is very funny that all the folks find Jupyter amazing when
> it is a worse version of Org-mode (troll troll) ;-)

Yes and no. Org-mode doesn't have anything close to Jupyter's widgets.
But otherwise, I agree.

It would be nice in fact to adapt the ideas behind Guix-Jupyter (and
perhaps parts of the code) to Org-mode. Some integration with Emacs will
be necessary to tell Org-mode to start Python etc. from the Guix
environment.

> Is the UI ;;guix aligned with the CLI UI ($ guix)?

I actually wonder which way the alignment should be done - the current
guix CLI is a bit of a mess already.

Cheers,
  Konrad.

Re: GNU Guix maintainer collective expands

[Konrad Hinsen]

Am 05.10.19 um 14:46 schrieb Ludovic Courtès:


Ricardo Wurmus and I are thrilled to announce that Marius Bakke, Maxim
Cournoyer, and Tobias Geerinckx-Rice are joining us in maintaining Guix!


That's good news and an excellent opportunity to thank everyone working 
hard on Guix, which means in particular the current and future maintainers!


Konrad.

Re: Let’s merge ‘core-updates’!

[Konrad Hinsen]

Hi Ludo,


Please try to upgrade your system and your user profile to see if
anything’s wrong for you (I did that a few days ago and I’m happy!).


It mostly works fine for me, except that the build of next-gtk-webkit 
fails (which works fine on master). The cause is:


make[2]: Entering directory '/tmp/guix-build-webkitgtk-2.26.1.drv-0/build'
make[2]: *** No rule to make target 'JavaScriptCore-4.0.gir', needed by 
'WebKit2-4.0.gir'.  Stop.


Konrad.

Re: Updates on the guix home manager

[Konrad Hinsen]

Hi everyone,

>> A new "guix home" subcommand, with support for "guix home build" (that
>> builds a home configuration), "guix home reconfigure" (that builds and
>> ...
>
> Nice!

+1

> Anyway, Guix Home is really cool and it’d be great to provide something
> along these lines out-of-the-box with Guix.  I still find the read-only
> home approach somewhat too radical, but maybe I’m not daring enough.
> ;-)

Home is used for too many things: configuration, personal data storage,
caching in hidden subdirectories, etc. It will take time to sort this
out, but I like the direction, including read-only for the configuration
stuff.

It's probably worth coordinating with other ideas for renovating home,
such as

  https://www.phoronix.com/scan.php?page=news_item=systemd-homed

Different objectives, but also interesting ones, and it would be nice if
all that would end up being compatible.

Cheers,
  Konrad.

Re: How to reference external program used in shell-scripts?

[Konrad Hinsen]

Ricardo Wurmus  writes:

> Hartmut Goebel  writes:
>
>> Anyway: IMHO missing "dynamic binding" is one of the major drawbacks of
>> functional deployment, as it requires updating (and esp. downloading)
>> much more packages compared to a rpm/deb based system.
>
> At the same time static binding is also one of the major advantages as
> deployments are stateless and thus much more predictable, reliable, and
> inspectable.

Indeed. We are having the same dynamic vs. static debate at the systems
level that has been going on at the programming language level for a
while. And for both situations, we still have to find the right
equilibrium between the two approaches, which both have their good and
bad aspects.

At the systems level, the last decades have seen a strong emphasis on
dynamic binding, in particular with the dominance of dynamic linking and
shared libraries. This has lead us into dependency hell to which we have
found quick-hack solutions (coarse-grained packages such as Docker
containers) and then more thorough ones (Nix and Guix).

Now the challenge is to put the right dose of dynamic binding back into
mostly static systems, and in the right place. For Guix, I'd say the
right place is the individual profile, and in particular the user's
default profile, so /usr/bin/env looks like a good start. My current way
of doing my work with Guix is to play dynamically in my default profile
but deploy stable applications in –pure environments. That's a bit more
work than what I could dream of, but it's so much better than
everything else I have used before that I am not complaining.

Cheers,
  Konrad.

Re: 01/01: services: Add ‘/usr/bin/env’ special file.

[Konrad Hinsen]

Hi Ricardo and everyone else,


Using a custom script with a /usr/bin/env shebang is pretty common.  You
don’t need to be a power user for that, and certainly not a *Guix* power
user.


I definitely agree with this. In my work environment, it is very common 
for people to distribute shell or Python scripts with a /usr/bin/env 
shebang line, together with the instructions "make executable and put on 
your $PATH". Most of my colleagues can handle those instructions, but 
wouldn't consider themselves power users.


Konrad.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Ludovic Courtès  writes:

> Do you observe similar slowness with other Docker images—e.g., Debian
> images?

No. But I don't observe any slowness with Guix either. Once it's up and
running, performance is normal (except for the known problem with access
to macOS file systems). It's only the daemon startup failure that
suggests a problem. I haven't had such problems with other Docker images
based on Debian or Ubuntu.

Konrad.

Re: Increasing the timeout for shepherd services

[Konrad Hinsen]

Hi Ludo,

> ‘read-pid-file’ in the Shepherd has a hard-coded default value of 5
> seconds.  You can change it for each service, but there’s currently no
> way to change the default value globally.

That's what I did: change it for each service.

> The patch below adds a SRFI-39 parameter in the Shepherd that would
> allow you to change the default PID file delay globally.

That sounds useful. Beyond my immediate problem, it seems weird to
hard-code a delay in the Shepherd source code.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Hi Ludo,

>> A more fundamental question is whether I will be able to do useful work
>> with this setup. It is well known that host file access from Docker on
>> macOS is slow, but I didn't realize how slow it is until I compiled Guix
>> in... four hours. On the same machine booted to Linux, it's about 40
>> minutes.
>
> Ouch.  Is this a fact of life or are there special settings that could
> make it less slow?

Docker for macOS has some options that use caching for more efficient
host filesystem access, but they come with restrictions, in particular
file changes on one side not being propagated immediately to the other
side.

> Five seconds to spawn a process is already relatively long; we could
> increase it a little, but the slowness here is a concern.

Whereas the problems with file access are known and understood
(ultimately it is the important differences in low-level file handling
between Linux and macOS), I have no idea why the startup time for these
daemons is so high. There is no significant CPU performance overhead
when running Guix in Docker, nor do the daemons require any access to
files managed by macOS.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Maxim Cournoyer  writes:

> I'm happy to read you could successfully make it work! It's strange
> that you had that issue though!

By now I have a system that starts all services without failure. I got
there by increasing the timeout to 30 s everywhere (compared to the
default value of 5). Perhaps 30 is more than necessary, I didn't try
intermediate values. So it looks like Guix in Docker is a bit slow to
boot, at least on the Mac.

A more fundamental question is whether I will be able to do useful work
with this setup. It is well known that host file access from Docker on
macOS is slow, but I didn't realize how slow it is until I compiled Guix
in... four hours. On the same machine booted to Linux, it's about 40
minutes.

Konrad.

Increasing the timeout for shepherd services

[Konrad Hinsen]

Hi everyone,

When running Guix in a docker image, I get systematic failures for
starting several services: syslogd, nscd, openssh. It turns out that in
all these cases the daemons are started, but so slowly that shepherd
declares a failure before they are fully operational.

I managed to fix this for syslogd by setting #:pid-file-timeout to 30
s. But I don't see a similar parameter for nscd and openssh, and
moreover it seems more reasonable to increase the timeout for all
services. Is this possible? I didn't find anything in the Shepherd
manual.

Konrad.

Re: syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

On 03/07/2019 10:12, Konrad Hinsen wrote:


All this suggests that perhaps the problem is not with syslogd, but with
shepherd wrongly concluding that syslogd failed.

Looks like I will have to dig a bit more into shepherd :-(


My hypothesis turned out to be correct: increasing the timeout was 
sufficient to get syslogd working.


Konrad.

syslogd can't be started when running Guix in Docker

[Konrad Hinsen]

Hi everyone,

I am exploring the possibility of running Guix under macOS via Docker.
First I ran 'guix system docker-image` on a minimal system
configuration, which I then transferred to the Mac and started.
Overall it works admirably well. There are however a few services
that don't work, in particulary syslogd:

   root@guix /# herd status
   Started:
+ file-systems
+ guix-daemon
+ host-name
+ loopback
+ root
+ root-file-system
+ udev
+ urandom-seed
+ user-file-systems
+ user-processes
+ virtual-terminal
   Stopped:
- nscd
- ssh-daemon
- syslogd
   One-shot:
* user-homes

   root@guix /# herd start syslogd
   Service syslogd could not be started.
   herd: failed to start service syslogd

I have no idea why this should fail, and worse, I have no idea how to
debug this. I checked /var/log/messages and /var/log/debug, but I can't
see anything looking like an error message except for shepherd reporting
that syslogs and ssh-daemon could not be started.

I'd be grateful for ideas and suggestions on how to figure out what is
going wrong. I'd also be interested in working dockerized versions
of Guix for comparison.

Thanks in advance,
  Konrad

Re: Down with PYTHONPATH!

[Konrad Hinsen]

Hi Pjotr,

> It would be interesting to see how others solve this problem.
> Including Nix and Conda.

Conda uses the same approach as Python's virtualenv: create a seperate
Python installation made up mainly of linke to files shared with other
such installations.

We could probably do something similar in Guix as well, assuming we can
make the Python executable figure out from which environment it was
started.

Konrad

Re: Down with PYTHONPATH!

[Konrad Hinsen]

Hi Ludo,

> How does virtualenv work, if not by setting PYTHONPATH?

It creates a new filetree corresponding to a complete new Python
installation, and the uses soft links to share most of the actual files
with the parent installation.

> Setting up an environment all about augmenting the search path, so I
> naively thought that virtualenv sets PYTHONPATH or the corresponding
> Python variable.

PYTHONPATH only adds directories to the search path. The standard
library and packages installed via pip are already on the search path
before PYTHONPATH is looked at. So virtualenv could not work by
modifying PYTHONPATH.

Konrad.

Re: Lightning talk at IPFS camp

[Konrad Hinsen]

Ludovic Courtès  writes:

> (They could have chosen Guile instead of a custom Lisp, but that’s an
> “implementation detail”.  :-))

>From my reading of the whitepaper, no. They have pretty strict
constraints on their language because they send live code updates
to running instances and want to be able to make certain guarantees.
Guile, or any other Scheme, would be too flexible.

Konrad.

Re: Lightning talk at IPFS camp

[Konrad Hinsen]

Ludovic Courtès  writes:

> Hey!
>
> I stumbled upon this:
>
>   https://github.com/ipfs/package-managers#readme

And I just stumbled on this one:

   https://github.com/ipfs/roadmap

Quote:

   2019 Priority

   We will be focusing our efforts into a single (lazer focus)
   priority.  Package Managers

This looks like just the right moment to tell them about the one true
package manager ;-)

Konrad.

Re: Lightning talk at IPFS camp

[Konrad Hinsen]

Pierre Neidhardt  writes:

>> I was thinking of the Guix package definitions. In the long run,
>> assuming IPFS turns out to be reliable enough, we could put all source
>> into IPFS with a CID reference, rather then today's many ways to
>> download source files.
>
> There would be nothing special about it beside implementing an IPFS
> fetcher, or would it?  Let me know if I misunderstood.

What's special about it is that it could replace all the other ones.

> Didn't know Radicle, it looks fantabulous!  And... it uses (or plans to)
> a Scheme-based language! :)

Exactly.

> So you are saying that we could move the guix.git to a Radicle project,
> right?

Yes, assuming all that becomes stable at some point.

Konrad.

Re: Lightning talk at IPFS camp

[Konrad Hinsen]

Pierre Neidhardt  writes:

>>   - A unified way to refer to stuff (I am thinking of IPLD here)
>>   No more tarballs, git commits, etc. CIDs everywhere.
>
> Do you have a concrete use case?

I was thinking of the Guix package definitions. In the long run,
assuming IPFS turns out to be reliable enough, we could put all source
into IPFS with a CID reference, rather then today's many ways to
download source files.

>>   - A unified storage scheme for all data, both "system" and "user".
>
> Can you elaborate?

Again in the long run, if we don't mind depending on IPFS, we don't need
the Guix store any more. Package installation would amount to local
pinning. Anyone could then build a package anywhere (home directory,
...) and just add it to IPFS. Since that also eliminates the technical
constraints of the store, the same mechanism could be used for any kind
of data processing, with the results stored in IPFS. Reproducibility of
any kind of computation via Guix, with building software just an
important special case.

> I'm not too sure about how human input would be logged, but at the very
> least the idea of distributing the store seems amazing.

For human input, Git would be OK, with repositories stored in IPFS
(there's already some support for that, see
https://github.com/ipfs/go-ipld-git). A more radical redesign is
Radicle (http://www.radicle.xyz/), which uses IPFS as a collaboration
platform (still at the git level). I guess Radicle could be used for
much more than that in Guix, but I haven't looked at that in detail.

Konrad.

Re: Lightning talk at IPFS camp

[Konrad Hinsen]

Am 03.06.19 um 18:19 schrieb Pronaip:


-   All data comes with provenance tracking:
 -   computations are tracked via Guix
 -   human input is logged (interactivity) or version controlled


unless you vision of the distant future somehow also includes most social 
problems having been solved, these would be ripe for abuse by malicious actors.


I didn't say that provenance information should be made public, even if 
the data itself is. It could well be encrypted. More generally, 
encryption is an essential ingredient for dealing with non-public data 
in IPFS. Which is indeed a weak point considering that you never know 
for how long encryption will be safe.



Not having any local file system is also just plain wasteful.


Note that I said file system, not storage. There should be local storage 
for efficiency, but it would act like a cache, not as a separately 
managed storage space.


BTW, data management is not really part of IPFS at all, it needs to be 
handled by another layer and so far there is little more than 
experiments. Textile looks promising, for example.


Konrad.

Re: Lightning talk at IPFS camp

[Konrad Hinsen]

Hi Pierre,

> I'm going to the IPFS camp (https://camp.ipfs.io/) on June 27th and I've
> been asked to give a (5 minute) lightning talk about IPFS & Guix.  Yaik! :)

That sounds like a great opportunity. Guix and IPFS are in my humble
opinion two of the most interesting ongoing projects in the computing
world.. Bringing the two together can only make this better.

> I'll have to prepare some slides (probably in Org reveal?) soon, so now
> is the time to pitch in some ideas ;)

A common issue: augmenting trust in digital information
  - IPFS: unalterable references to arbitrary data
  - Guix: provenance tracking for software builds

What IPFS can do for Guix:
  - A better way to archive and distribute Guix and the software
it builds.
  - A unified way to refer to stuff (I am thinking of IPLD here)
No more tarballs, git commits, etc. CIDs everywhere.
  - A unified storage scheme for all data, both "system" and "user".

What Guix can do for IPFS:
  - Provenance tracking for data that has been processed by software

A vision for a (remote) future:
  - All data lives in IPFS: no local filesystem, no Guix store.
A personal computing device only stores references to information
that its owner cares about.
  - All computations are equal: no distinction between "software builds"
and everything else.
  - All data comes with provenance tracking:
 - computations are tracked via Guix
 - human input is logged (interactivity) or version controlled

Konrad.

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

Hi Ricardo,

>> "My channel" doesn't exist (because I haven't yet found the time to
>> figure out how to set up and manage a channel, although it's been on my
>> to-do list for a while).
>
> I’d be happy to assist.

Thanks! I might come back to that offer when I find at least enough time
to figure out precisely what information I might be lacking
(that's meta-lack-of-time).

>> But... how about splitting off *all* of Python 2 and everything that
>> depends on it into a separate channel, which would then be maintained
>> by a separate team?
>
> Currently this is not feasible, in my opinion, as a lot of packages in
> Guix still depend on Python 2 for some reason or the other.  When Python
> 2 reaches EOL, however, I think this would be a reasonable thing to do.

Sounds good, then the only remaining issue is defining a transition
protocol. What I'd like to avoid is that packages disappear one by one
from Guix and then have to be dug out one by one from Git history for
setting up a Python 2 channel.

More generally, I think it would be useful to collect in some place all
package definitions that are removed from Guix. For example, a list with
package names plus the last Guix commit that had the package.

Konrad.

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

Hi Ricardo,

> Konrad, going forward it might be reasonable to keep copies of required
> Python 2 packages in your channel.  We aren’t going to remove Python 2
> packages now, but in the future we may not be able to fix unmaintained
> packages and may have to remove them.

"My channel" doesn't exist (because I haven't yet found the time to
figure out how to set up and manage a channel, although it's been on my
to-do list for a while).

But... how about splitting off *all* of Python 2 and everything that
depends on it into a separate channel, which would then be maintained
by a separate team?

This would remove the Python 2 maintenance burden from the Guix team,
but still allow shared maintenance (expected to be low-effort) of
a coherent distribution of Python 2 packages.

If everybody starts a personal channel of whatever subset of those
packages they require, there's a lot of duplicated effort and no clear
location to turn to for end users. And if parts of Python 2 remain in
Guix but other parts move elsewhere, there's needless friction in
coordinating two projects that will end up having different policies.

Konrad.

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

Efraim Flashner  writes:

> I checked 'guix refresh -l python2' and I got a list of the python2-*
> packages which are leaf packages. I'm not suggesting that we right now
> get rid of them, but it seems to me something worth keeping an eye on.

Please be careful with "leaf packages". The leaves of my software
dependency tree are in my home directory, not in Guix. But I depend on
Guix to supply most of the dependencies of these leaves.

In the case of Python, any package that provides a Python library API
should not be considered a leaf package.

Konrad.

Re: the upcoming Great Python2 Purge™

[Konrad Hinsen]

On 26/12/2018 10:38, Efraim Flashner wrote:

We're now about a year out from the official EOL for python2 (Jan 1,
2020). So far we've been not adding python2 variants of packages that
are new unless they're actually needed for something. Do we want to
start removing python2 packages when updating other packages if they are
leaf packages?


What would we gain by that? Plenty of people are still using Python 2. 
Many rather specialized packages still exist only for Python 2, and may 
well depend on packages that are leaves within Guix. Why make their life 
more difficult as long as there is no clear benefit in terms of Guix 
maintenance effort?



Konrad.

Re: Happy birthday!

[Konrad Hinsen]

On 23/11/2018 14:41, Ludovic Courtès wrote:

Today, it’s been six years already!


Let's have a party then:

(define-public guix-birthday-cake
  (package
    (name "guix-birthday-cake")
    (synopsis "A birthday cake for Guix")
    (build-system birthday-cake)
    (arguments
 `(#:number-of-candles 6))
    (inputs
 `(("cake" ,cake)
   ("candle" ,candle)

Thanks to everyone who contributed to make this happen!

Konrad.

Re: Meetup in Paris, Dec. 10th!

[Konrad Hinsen]

Am 11.10.18 um 15:55 schrieb Ludovic Courtès:


I’ve booked a room at Inria’s offices in Paris for the Dec. 10 gathering
I proposed a while back, see:


Great! At what time does it start?


A couple of notes: the room capacity is 16 but I’ll look for a larger


There's actually plenty of space for more in that room, it's just a 
matter of finding chairs.


Looking forwards to meeting other Guixers!

Konrad.

Re: Meetup in Paris, Dec. 10th?

[Konrad Hinsen]

Am 27.09.18 um 14:01 schrieb Ludovic Courtès:


Some of us will be in Paris, France, for the Reproducible-Build Summit¹,
December 11–13.

I was wondering if people would be willing to gather, say, the day
before the summit (December 10) to discuss Guix things and/or have hack
sessions.  Who’s in?  :-)


Well, if you come to my doorstep, it would be unpolite for me not to 
join ;-)


Konrad.

Re: Python 2 retirement — what should Guix do?

[Konrad Hinsen]

Leo Famulari  writes:

> I wonder, what should Guix do?
>
> Personally, I think our set of Python packages is relatively hard to
> maintain. There is a lot of brittle code in there. I'd be happy to drop
> our policy that Python libraries have both Python 2 and 3 packages by
> default.

I expect Python 2 to be widely used for a few years longer. In my
environment (computational chemistry), there is still a lot of Python 2
code in use that is too expensive to port to Python 3 (mainly because of
C modules). Most people seem to expect that life will go on like before,
not worrying about the end of support for Python 2 at all. The few who
do are making plans for running their Python 2 code in sandboxed
environments.

Guix is one of the best tools to manage this legacy code, so I hope
Python 2 will stay in for those packages that need it. But the general
policy of having both for all packages can certainly be softened without
much loss.

Konrad.

Re: “Tarballs, the ultimate container image format”

[Konrad Hinsen]

On 17/05/2018 07:53, Chris Marusich wrote:


Great article!  Thank you for sharing it.  The manual is nice, but I
have to admit, I enjoy reading these blog posts quite a bit, too.  It's
nice to read a brief article that's focused on introducing a specific
aspect of Guix, with cross-references for further reading.


+1

Nice new features like this one are hard to discover by reading the manual.

KOnrad.

Re: Dealing with language bindings for libraries.

[Konrad Hinsen]

On 09/05/2018 20:00, Julien Lepiller wrote:


We already have such a case: capstone and python-capstone. There is no
redundancy since python-capstone knows how to load the shared library
created in the capstone package. So we have two packages, with the same


My situation is a bit different. The package I am working on (OpenBabel, 
http://openbabel.org/wiki/Main_Page) has a monolithic build process 
based on CMake that builds the library plus bindings for all languages 
that it detects in its environment. There is no obvious way to build the 
language bindings to go with an already installed library.


It's almost trivial to package in Guix for any language combination you 
care about: just add the languages you need to the inputs. It's also 
straightforward to make each language binding a separate output of the 
package. But choosing this option makes building the package very expensive.


Konrad.

Re: Dealing with language bindings for libraries.

[Konrad Hinsen]

On 09/05/2018 17:21, Fis Trivial wrote:


An ideal scenario would be the one that we can specify multiple outputs
for one packages, each output corresponds to one language binding, and
we can specify different dependencies and build system for each
output. Is there any chance we can do that in guix?


+1

I am currently in exactly the same situation.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

Ludovic Courtès  writes:

> Yes. OTOH we use the “python2-” prefix for 2.x packages and “python-”
> for 3.x packages.

Indeed. What a mess!

>> This does of course raise the question of how this will evolve in the
>> long run, but since so many bad decisions were already taken, I am not
...

> Not our business I’d say.  :-)

Right now, no. But we might one day have to decide how to respect
upstream recommendations that change over time, if that change has a
potentially undesirable impact on Guix users.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

On 14/03/2018 12:39, Hartmut Goebel wrote:

Am 13.03.2018 um 22:52 schrieb Ludovic Courtès:

   2. Use different package names when we know things can be
  parallel-installed: “python2” vs. “python” (I’m talking about the
  package name, not its version string.)  That’s what distros usually
  do, and I think it’s good enough.


I'd prefer this.


That sounds like a good basis. But perhaps "python" for Python 2 and 
"python3" for Python 3 would make more sense, since those are the names 
of the executables.


This does of course raise the question of how this will evolve in the 
long run, but since so many bad decisions were already taken, I am not 
trying to guess what will happen. For now, the upstream recommendation 
remains to use "python" and "python3" to distinguish the executables. 
But what will happen in 2020? The Python community might be tempted to 
change the naming to mark the end of Python 2 support, but that would be 
at the price of another round of breaking everybody's scripts.


Konrad.

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

Ricardo Wurmus  writes:

> It is an unnecessary restriction to *prevent* users from installing
> Python 2 and 3 interpreters into the same profile.  Any errors we see

I agree. But the current question is not if we should allow people to
shoot themselves into the foot, but how much effort we should invest to
make this possible.


Pjotr Prins  writes:

>> It is an unnecessary restriction to *prevent* users from installing
>> Python 2 and 3 interpreters into the same profile.  Any errors we see
>> with mixing 2 and 3 is due to a bug in Guix which is due to our use of
>> PYTHONPATH.
>
> Alright, if you choose to use them as two different tools/languages.

Which is what they are - the biggest mistake in the Python 2->3
transition was, in my opinion, to pretend they are the same language.
It is possible to write programs that are valid in both languages but
produce different results; this even happens regularly by mistake
due to the change of division semantics. So by my definition, they are
two similar but distinct languages.

> Sure. Note that Python2 is quickly becoming obsolete. It is escalating
> because packages are now dropping support. After 10 years of joy
> mixing versions...

I expect Python 2 to remain widely used for another 10 years after
official support ends. Applications that have no security concerns can
simply continue to build on the last official release of Python 2 and
its library ecosystem. For applications with a high migration cost, it's
still not worth the effort to port to Python 3.

Konrad.

Re: installing python 2 and python 3 in the same profile

[Konrad Hinsen]

On 10/03/2018 09:34, Pjotr Prins wrote:


It still works with ad-hoc environments, but manifests containing both
Python versions cannot be instantiated any more.  This is too strict,
because we know that these two variants don’t cause conflicts.


That is not my experience. Any mix is a problem.


I agree. Even if "it works" superficially, it's hard to do serious work 
with Python when you have the two versions in the same profile.


In fact, I find that Guix is the best package manager for dealing with 
this precisely because it allows me to have multiple independent profiles.



I suggest not to go there. Teach people about profiles and
environments and all should be fine. You can support Python 3, Python


Yes, and perhaps make working with profiles a bit easier. For example, 
I'd appreciate a "guix profile" resembling "guix environment":


   guix profile ~/profiles/python2

would start a shell with the python2 profile, and

   guix profile ~/profiles/python2 command

would run "command" in the context of the profile. This is actually on 
my to-do list.



I must add that in most of my Guix experience I relied on
$HOME/.guix-profile. But these days I hardly use that anymore. Just
for some basic desktop tooling. Using multiple profiles is the only
way to mix versions and keep a level of sanity.


Same here.

Konrad.

Re: Outreachy'18

[Konrad Hinsen]

On 06/03/2018 20:34, Reshu Singh wrote:

I am trying to install GUIX for ubuntu using sh script. I have named the 
script "guix-install.sh".I created the sh file in nano and ran using 
bash guix-install.sh Encountering an error "guix-install.sh: line 249: 
/home/reshu/.guix-profile/etc/profile: No such file or directory".


The instructions for installing Guix on a Ubuntu system are at:

  https://www.gnu.org/software/guix/manual/guix.html#Binary-Installation

There is no reference to any shell script there, so I wonder what you 
are trying to accomplish using your script?


Konrad.

Re: Use guix to distribute data & reproducible (data) science

[Konrad Hinsen]

Hi George,

myg...@gmail.com writes:

>> The three missing pieces are:
>>
>>  - Dealing with measurements, which might involve interacting with
>>experimental equipment or databases. Moreover, since data from
>>such sources can change, its hash in the store must be computed
>>from the contents, not just from the reference to the contents.
>
> Why not "enclose" a measurement set and it's provenance in a git
> "package"?

For text-based data of reasonable size, that's an option. Many people
are already using git for data management. But then, data is so diverse
that no rule and no tool will satisfy everybody's requirements.

Konrad.

Re: Use guix to distribute data & reproducible (data) science

[Konrad Hinsen]

Hi,

> In other words, on the paper, what are the benefits of a management of
> some piece of data in the store ? For example for the applications of
> weights of a trained neural network; or of the positions of the atoms in
> protein structure.

Provenance tracking. In a complex data processing workflow, it is
important to know which computations were done in which order using
which software. This is technically almost the same as software
dependency tracking, so it would be nice to re-use the Guix
infrastructure for this.

> For me --maybe I have wrong-- the way is to define a package (or
> workflow) that fetches the data from some external source, cleans if
> needed, does some checks, and then puts it to /path/to/somewhere/
> outside the store. In parallel computing, this /path/to/somewhere/ is
> accessible by all the nodes. Moreover, this /path/to/somewhere/ contains
> something hash-based in the folder name.
>
> Is it not enough ?

Whether for software or for data, dependencies are DAGs whose terminal
nodes are measuremnts (for data) or human-supplied information (code,
parameters, methodological choices). Guix handles the latter very well.

The three missing pieces are:

 - Dealing with measurements, which might involve interacting with
   experimental equipment or databases. Moreover, since data from
   such sources can change, its hash in the store must be computed
   from the contents, not just from the reference to the contents.

 - Dealing with data that is much larger than what Guix handles
   well in the store.

 - Exporting results with provenance tracking to the outside world,
   which may not be using Guix. Big data aside, this could take the
   form of a new output format to "guix pack", for example a
   research object (http://www.researchobject.org/) or a Reproducible
   Document Archive (https://github.com/substance/dar).

> Why do you need the history of changes ? as git provide ?

I'd say git is fine for everything not "too big".

> Secrets is another story than reproducible science toolchain, I guess.

Yes, indeed. And not something I need to deal with, so I will shut up
now!

Konrad.

Re: Do you use packages in Guix to run neural networks?

[Konrad Hinsen]

On 14/02/2018 05:43, Fis Trivial wrote:


Sorry for bothering with a completely unrelated topic.
I'm curious do you train neural network with packages in Guix? Or did
you packaged related libraries yourself?


My needs are modest, all I use is the multilayer perceptron from 
scikit-learn, which is already packaged in Guix.


Konrad.

Re: Use guix to distribute data & reproducible (data) science

[Konrad Hinsen]

Hi everyone,

zimoun  writes:

> From my point of view, there is 2 kind of datasets:
>  a- the ones which are part of the software, e.g., used to pass the
> tests. Therefore, they are usually small, not always;
>  b- the ones which are applied to the software and somehow they are
> not in the source repository. They are big or not.

I was thinking of the second kind only.

> It does not appear to me a good idea to try to include in the store
> datasets of case b-.
> Is it not the job of data management tools ? e.g., database etc.

It depends. The frontier between data and code is not as clear as it may
seem. An example: the weights of a trained neural network can be seen as
data (a bunch of numbers), but also as code for a special-purpose
processor defined by the neural network.

Starting from that example, consider that the weights of a neural
network are not fundamentally different from fit parameters in other
scientific models. For example the positions of the atoms in a protein
structure. Using the same analogy as for the neural network, these
positions are the code for a special-purpose processor that computes
estimations for the Bragg reflections that are measured in protein
crystallography.

On the other hand, any sufficiently small dataset can be replaced by a
piece of code that defines a literal data structure. Does that turn the
data into code or not?

My point is that trying to define a distinction between data and code is
always arbitrary and rarely of practical interest. I prefer to take a
pragmatic stance and ask the question: what are the advantages and
problems associated with managing some piece of data in the store? And I
suspect that exploring this question for a couple of applications will
lead to new ways to profit from the store.

However, distribution of data is an entirely different question from
managing data on a single machine. I have no idea how well suited the
store is for distributing data (Hydra does exactly that, right?), so
I'll happily listen to the Guix experts.

> It appear to me as a complement of these points ---and personnally, I
> learn some points about the design of GWL--- with this thread:
> https://lists.gnu.org/archive/html/guix-devel/2016-05/msg00380.html

Thanks for the pointer!

>> It would be nice if big datasets could conceptually be handled in the same
>> way while being stored elsewhere - a bit like git-annex does for git. And
>> for parallel computing, we could have special build daemons.
>
> Hum? the point is to add data management a la git-annex to GWL ? Is it ?

At least consider it - I don't know where that will lead.


Amirouche Boubekki  writes:

>> For big datasets, some other mechanism is required.
>
> Big as in bigger than ram?

Bigger that one is willing to handle as files that are copied between
machines for distribution. For me, personally and at this moment in
time, that's somewhere between 1 GB and 10 GB.

> What I was thinking about, is use guix to distribute data packages
> just like we distribute softwares from pypi. The advantage of using
> guix seems obvious, but apparantly it's not desirable or possible and
> I don't understand why.

I think there are two questions:
 1. Can guix and its current infrastructure be used for data
 distribution?
 2. Can a yet-to-be-designed data distribution infrastructure
 use the guix store as its storage backend?

My understanding from Ludo's comment is that 1) may not be a good idea,
but that still leaves 2) as a topic for exploration.

> And for parallel computing, we could have special build daemons.
>
> That's where OWL comes in?

Exactly.

Konrad.

Why don't "guix pack" and "guix environment" accept manifests?

[Konrad Hinsen]

Hi everyone,

Today I tried to convert one of my Guix profiles into a Docker image, 
and naively tried


guix pack -f docker -m my-manifest.scm

Now I know it doesn't work, but I wonder if there is a good reason or if 
this is just not implemented? The same question applies to "guix 
environment". Maybe even "guix build", although I don't see a use case 
for that. But I'd expect to be able to use a manifest wherever a list of 
packages is required.


Konrad.

Re: Can we speed it up? Prev: compiling guix is too slow?

[Konrad Hinsen]

Pjotr Prins  writes:

>> I wonder if anyone has analyzed the dependency graphs of software
>> packages (not necessarily for Guix, some big distribution like
>> Debian would be more interesting), with the goal if identifying good
>> splits based on simple criteria.
>
> Yeah, that would be a neat exercise. Any student here inclined to have
> a go?

As was to be expected, a quick search found some promising pointers, e.g.:

  https://arxiv.org/abs/0905.4226

  http://ieeexplore.ieee.org/document/7490780/

Plus a lot more on language-specific dependency analysis, which is less
directly useful but may contain interesting methods that could be
generalized.

So, yes, this would make a good research project (master's level for
example) with the potential for academic recognition.

Konrad.

Re: Can we speed it up? Prev: compiling guix is too slow?

[Konrad Hinsen]

On 05/02/2018 08:34, Pjotr Prins wrote:


compiled yet). Or generate a meta list for a source tree. Or
subcategorize packages so only those packages get included that are
asked for (assuming there are no deeper dependencies). For example,
few people need the bioinformatics packages. We could have the sub
section of the graph split out and have people do:

   guix package --topic=bio -i samtools


Or move special-topic packages to separate channels, once they get 
implemented. The hard part is of course *where* to split the graph, not 
how to implement it.



Sectioning the graph may be hard (you'd be inclined to section off
languages and window managers), but I think it can be dictated by
whether a sub graph can live on its own.
I wonder if anyone has analyzed the dependency graphs of software 
packages (not necessarily for Guix, some big distribution like Debian 
would be more interesting), with the goal if identifying good splits 
based on simple criteria.


Konrad.

Re: [RFC] A simple draft for channels

[Konrad Hinsen]

On 24/01/2018 13:33, n...@n0.is wrote:


In my honest opinion: No. We can not prevent this. All we can do
is to provide a list of *official* channels. Beyond that I don't
think we should try to regulate what's in an unofficial channel
and what's allowed.


+1

The best option in my opinion is to make it as easy as possible for 
users to find out the policy of any channel, and to do automatic 
filtering (for the day when there will be hundreds of channels to manage).


One simple measure would be a field in the channel description that says 
"complies with Guix' rules for free software yes/no". An added benefit: 
it forces channel designers to think about the question right from the 
start.


Konrad.

Re: Building from a local source code checkout

[Konrad Hinsen]

l...@gnu.org (Ludovic Courtès) writes:

> I agree the matching-name constraint can be annoying.  The alternative
> would be allow the user to (optionally?) specify the name of the package
> whose source is being changed, as in:
>
>   guix build python-activepapers \
> --with-source=python-activepapers=$HOME/Dev/AP
>
> WDYT?

Looks much clearer to me. But, as you say, this could well be optional
if we can live with an ambiguity in the rare case of filenames containg
an equal sign.

Konrad.

Re: Building from a local source code checkout

[Konrad Hinsen]

Hi Carlo,

> On 20 December 2017 9:19:00 pm AEDT, Konrad Hinsen 
> <konrad.hin...@fastmail.net> wrote:
>> > guix build python-activepapers 
>>--with-source=~/Development/python-activepaper
>>guix build: error: lstat: No such file or directory: 
>>"~/Development/python-activepapers"
>
> I found earlier today that the tilde wasn't expanded in guix package
> --manifest. Maybe it's the same for --with-source. Try putting your
> home directory in as an absolute path.

Oops, you are right! So

   guix build python-activepapers 
--with-source=$HOME/Development/python-activepapers/

actually works, but

   guix build python-activepapers 
--with-source=$HOME/Development/python-activepapers

(without the terminating slash) doesn't:

  guix build: error: guix/scripts/build.scm:155:6: package
  `python-activepapers-0.2.2' has an invalid input: ("source"
  "/gnu/store/bbi86dn5nq6f79lhvs7ghzisjahjaq5p-python-activepapers")

Perhaps I should stop trying to understand computers ;-)

Thanks,
  Konrad.

Re: Building from a local source code checkout

[Konrad Hinsen]

Hi Ludo,

>> For debugging package definitions, it would be really nice to be able
>> to build a package from a checkout of the original project source
>> code, rather than having to make a tarball for each modification of
>> that source code. Is this possible somehow?
>
> You should be able to use “guix build foo --with-source=./foo” (note
> that the directory name must match the package name).
>
>   
> https://www.gnu.org/software/guix/manual/html_node/Package-Transformation-Options.html

The name matching condition is my problem with this approach, which is
why I was looking for a way to do specify a local source in the package
definition itself.

I am working in the Python universe where package names almost never
match the name of the checkout directory because of the python- prefix.
Moreover, I want to build both python- and python2- packages from my
checkout.

>> The manual suggests that a package source could be something else than
>> an origin object, such as a local-file, but I cannot get even
>> local-file to work.
>
> It should work (you must import (guix gexp) to get ‘local-file’).
>
> Can you share what errors you got?

None - once I import (guix gexp)  ;-)

Thanks,
  Konrad.

Building from a local source code checkout

[Konrad Hinsen]

Hi everyone,

For debugging package definitions, it would be really nice to be able to 
build a package from a checkout of the original project source code, 
rather than having to make a tarball for each modification of that 
source code. Is this possible somehow? The manual suggests that a 
package source could be something else than an origin object, such as a 
local-file, but I cannot get even local-file to work.


Konrad

Re: GNU Guix & GuixSD 0.14.0 released

[Konrad Hinsen]

On 07/12/2017 13:45, Ludovic Courtès wrote:


We are pleased to announce the release of GNU Guix & GuixSD 0.14.0,
representing 5,192 commits by 88 people over 6 months.


Congratulations and thanks to all contributors!

The more I use Guix, the more I appreciate its qualities, but also the 
enormous amount of work that went into it.


Konrad.

Re: Changing HTTP proxy settings in GuixSD

[Konrad Hinsen]

Hi Ludo,

> I’ll update the ‘guix’ package soon so that this change is available to
> daemon-side code such as ‘guix substitute’.  In the meantime you can run
> the daemon from a checkout:
>
>   sudo -E ./pre-inst-env guix-daemon …

That works fine - thanks!

Konrad.

Re: Changing HTTP proxy settings in GuixSD

[Konrad Hinsen]

Hi Chris,

> That error looks suspicious.  Perhaps you already know this, but it
> probably means that the string "mirror.hydra.gnu.org" wound up being
> used where a procedure should probably have been used instead.  For
> example, in your Guile REPL, you can reproduce this kind of error like
> this:
>
> scheme@(guile-user)> ("this is not a procedure")
> ERROR: ERROR: Wrong type to apply: "this is not a procedure"

Suspicious indeed...

> The workaround that Ludo described, in which you temporarily stop the
> guix-daemon, and then start it in an environment where http_proxy is set
> appropriately, should work.

That's exactly what I did.

> If it isn't, then can I ask to see the commands and conigs you're
> using which fail?

Here are the commands I typed:

   sudo -s
   herd stop guix-daemon
   http_proxy=http://proxy.synchrotron-soleil.fr:8080/  guix-daemon 
--build-users-group=guixbuild
   guix system reconfigure /etc/config.scm

The config I used in this first test (attached) is exactly the same one
I am currently running, with no reference to the proxy at all. I was
planning to add that in a second step.

Konrad.



config.scm
Description: Binary data

Re: Building Docker images of GuixSD

[Konrad Hinsen]

Hi Chris,


I've run GuixSD in a Docker container and returned to tell the tale!

Congratulations! And thanks for exploring all this.

> Is this helpful?  Is it worth polishing up and maintaining?  I'm not
> entirely sure, and I'd like to know what you think.

I think it is useful, mainly for reason 3:


* If you want to run Guix on a system to which Guix hasn't been ported
   (like macOS) but your system does run Docker, now you can run Guix on
   that system by running it from a GuixSD Docker container.


To which I might add a less obvious one: with GuixSD nicely integrated 
into the Docker universe, it has a better chance of adoption by people 
committed to Docker, and thus a better chance of becoming a/the 
preferred way of constructing Docker images. In other words, an attempt 
to take over the (Docker) world from inside.



For the third bullet point, I don't know of any other reasonable way to
get Guix working in Docker (although one could certainly run Guix in a
VM using a technology other than Docker, such as QEMU).  To run Guix,


I have been trying this approach for a while, but I am still much 
further away from running Guix on my Mac than you are.


First, QEMU: forget it on the Mac. It's not nearly as advanced/stable as 
it is under Linux. You can run Guix with QEMU under macOS, but it's slow 
and crashes a bit too often to rely on it.


I moved on to VirtualBox, which runs a basic GuixSD without any major 
problem. But what you get is an isolated virtual machine. I haven't yet 
found a usable strategy for accessing the macOS file system from GuixSD. 
VirtualBox relies on its proprietary guest OS add-ins. I suspect they 
could be ported to GuixSD from a technical point of view, but it's not a 
trivial job and you'd have to remove the term "free software" from your 
brain for a while to do it. Currently I am trying NFS, exporting my Mac 
home directory via an NFS server on the Mac (easy) and mounting it from 
GuixSD (no success so far, because of the very incomplete NFS support in 
GuixSD). I expect this will work eventually, but in terms of performance 
it will probably never get to what you can achieve with Docker.


Konrad.

Re: Changing HTTP proxy settings in GuixSD

[Konrad Hinsen]

Hi Ludo,


A quick workaround would be to do something along these lines:

   # herd stop guix-daemon
   # http_proxy=… guix-daemon --build-users-group=guixbuild
   # guix system reconfigure config.scm

where config.scm has the relevant proxy configuration of in
‘guix-configuration’.

Would that work for you?


No. Downloads fail with messages of the type

ERROR: Wrong type to apply: "mirror.hydra.gnu.org"

This looks like the proxy returns something that guix-daemon doesn't 
expect. Unfortunately, this may well be the fault of my institutions 
proxy, which has caused lots of people lots of trouble of various kinds.


Konrad.

Re: NFS mounts

[Konrad Hinsen]

l...@gnu.org (Ludovic Courtès) writes:

>> Of course, the real problem is that I cannot mount it at all because
>> rpc.statd is missing.
>
> Ooh, got it.  Well we could (ab)use the ‘dependencies’ field of
> ‘file-system’ to introduce a dependency on the rpc.statd daemon startup.

Is there a service that starts rpc.statd? I can't find rpc.statd
anywhere in the Guix source.

Konrad.

Re: NFS mounts

[Konrad Hinsen]

Hi Ludo,

> By default, file systems are automatically mounted at boot time.  To
> avoid that, you must add:
>
>   (mount? #f)

That works well indeed. But I actually want that NFS filesystem mounted
at boot time, ideally (not strictly required though).

Of course, the real problem is that I cannot mount it at all because
rpc.statd is missing.

Konrad.

Re: NFS mounts

[Konrad Hinsen]

Hi Ludo,

> Could you test it in a VM, pass “console=ttyS0” as a kernel argument,
> and “-serial stdio” so that we see all the messages on the console?

It took a while, but here it is. My config.scm is attached as well.

Konrad.



config.scm
Description: Binary data


guixsd-console.log
Description: Binary data

Re: Hacks to install Guix packages without root

[Konrad Hinsen]

On 26/10/2017 23:46, Ricardo Wurmus wrote:


How about an extension of “guix pack” that will rewrite the /gnu/store
references to a user-provided directory before bundling things up in a
tarball?  I’d *really* like to be able to just use the tarball bundle
“guix pack” produces by default, but currently deploying it requires
root access or proot shenanigans at runtime.


That would be nice indeed as a deployment option. In particular for 
distributing binaries to less experienced users.



On 27/10/2017 02:48, Ludovic Courtès wrote:

> The tarballs could include proot-static and another statically-linked
> program that essentially tries to call unshare(2).  Would that make
> sense?

That's perhaps good enough as well. Suggestion: try to write the user 
instructions for deploying these tarballs first, and pick the method 
that is easiest to describe.


I see basically two usage scenarios. One is a form of software 
installation: users download a tarball, follow the instruction 
procedure, and than have some additional commands in their PATH. The 
other scenario is the creation of a somewhat isolated environment, with 
users wanting a shell for working in just that environment. For that 
second scenario, proot is probably just fine, but I am less convinced 
about the first one.


Konrad.