Re: Image generation

[Leo Famulari]

On Tue, Apr 21, 2020 at 08:59:31PM +0200, Mathieu Othacehe wrote:
> On my computer, this takes 6m50 versus 2h30 for the master version. I
> tested the image in QEMU, everything seems fine.

Wow! Why did it take 2h30? Do you have KVM?

Image generation

[Mathieu Othacehe]

Hello,

I made some progress on the image generation topic. As discussed
previously, the goal is to use the same principles as genimage[1], to
achieve faster image generation, without resorting to VM.

A very related topic, is to bring the possibility to create Guix System
images with custom layouts. That includes position, size and type of the
bootloader partition, offset of the root partition and so on. While this
is not really important for desktop usage, it is almost mandatory for
embedded usage.

The wip-disk-image branch allows to define a Guix System image along the
lines of:

--8<---cut here---start->8---
(define my-image
  (image
   (format 'disk-image)
   (operating-system my-os)
   (partitions
(list (partition
   (size (* 40 (expt 2 20)))
   (label "GNU-ESP")
   (file-system "vfat")
   (flags '(esp))
   (initializer (gexp initialize-efi-partition)))
  (partition
   (size 'guess)
   (label "Guix_image")
   (file-system "ext4")
   (flags '(boot))
   (initializer (gexp initialize-root-partition)))
--8<---cut here---end--->8---

then, you can call '(system-image my-image) and get a bootable
disk-image.

On this branch, it is already possible to generate an EFI disk-image,
with the traditional command:

--8<---cut here---start->8---
./pre-inst-env guix system disk-image gnu/system/examples/desktop.tmpl
--8<---cut here---end--->8---

On my computer, this takes 6m50 versus 2h30 for the master version. I
tested the image in QEMU, everything seems fine.

Now there's still plenty of work. This branch needs some more
cleaning. Then we need to:

* Add support for ISO images.
* Extend support to grub (non-efi), extlinux and u-boot bootloaders.
* Check everything works with --system and --target arguments.

I've re-implemented some parts of genimage in (gnu build disk-image)
module. Now, we could also go further and remove the use of this tool
completely.

Please tell me what you think,

Thanks,

Mathieu

[1]: https://github.com/pengutronix/genimage

Re: 04/05: gnu: autoconf: Support cross-build.

[Jan Nieuwenhuizen]

Jan Nieuwenhuizen writes:

(fix typo/update renamed phase in commit message)

>From 2b7ae7542fd77b35d7a143c90556cf32a7f9ae48 Mon Sep 17 00:00:00 2001
From: "Jan (janneke) Nieuwenhuizen" 
Date: Sat, 18 Apr 2020 19:49:54 +0200
Subject: [PATCH] gnu: autoconf: Support cross-build.

Autoconf cannot be cross-built properly: it lacks the concept of
-for-build.  It runs the host `autom4te' (a perl script) during build.

* gnu/packages/autotools.scm (autoconf)[inputs]: When cross-building, add perl
and m4.
[native-inputs]: when cross-building, use -for-build names.
[arguments]: When cross-building, add `patch-non-shebang-references' phase to
substitute m4 and perl.
---
 gnu/packages/autotools.scm | 28 +++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/autotools.scm b/gnu/packages/autotools.scm
index 99ca52730e..fa3ced182f 100644
--- a/gnu/packages/autotools.scm
+++ b/gnu/packages/autotools.scm
@@ -55,12 +55,38 @@
(base32
 "113nlmidxy9kjr45kg9x3ngar4951mvag1js2a3j8nxcz34wxsv4"
 (build-system gnu-build-system)
+(inputs
+ ;; TODO: remove `if' in the next rebuild cycle.
+ (if (%current-target-system)
+ `(("perl" ,perl)
+   ("m4" ,m4))
+ '()))
 (native-inputs
  `(("perl" ,perl)
("m4" ,m4)))
 ;; XXX: testsuite: 209 and 279 failed. The latter is an impurity. It
 ;; should use our own "cpp" instead of "/lib/cpp".
-(arguments `(#:tests? #f))
+(arguments
+ `(#:tests? #f
+   ,@(if (%current-target-system)
+ `(#:phases
+   (modify-phases %standard-phases
+ ;; `patch-shebangs' patches shebangs only, and the Perl
+ ;; scripts use a re-exec feature that references the build
+ ;; hosts' perl.  Also, M4 store references hide in the
+ ;; scripts.
+ (add-after 'install 'patch-non-shebang-references
+   (lambda* (#:key build inputs outputs #:allow-other-keys)
+ (let ((m4 (assoc-ref inputs "m4"))
+   (perl (assoc-ref inputs "perl"))
+   (out  (assoc-ref outputs "out"))
+   (store-directory (%store-directory)))
+  (substitute* (find-files (string-append out "/bin"))
+(((string-append store-directory "/[^/]*-m4-[^/]*")) m4)
+(((string-append store-directory "/[^/]*-perl-[^/]*"))
+ perl))
+  #t)
+ '(
 (home-page "https://www.gnu.org/software/autoconf/;)
 (synopsis "Create source code configuration scripts")
 (description
-- 
2.26.0


-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: 04/05: gnu: autoconf: Support cross-build.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello,

>> gnu: autoconf: Support cross-build.

> Thanks for fixing it!  Some comments:

>> +(inputs
>> + (if (%current-target-system)
>> + `(("perl" ,perl)
>> +   ("m4" ,m4))
>> + '()))
>>  (native-inputs
>> - `(("perl" ,perl)
>> -   ("m4" ,m4)))
>> + (if (%current-target-system)
>> + `(("perl-for-build" ,perl)
>> +   ("m4-for-build" ,m4))
>> + `(("perl" ,perl)
>> +   ("m4" ,m4
>
> You can remove the ‘if’ in both cases: we always need Perl/M4 both as a
> native input and as an input in both cases.

...yes

> However, that might trigger a rebuild, so perhaps you’ll have to leave
> the ifs, but with a TODO telling to remove it on the next rebuild or
> Marius will be mad at us.

Indeed; so changed to:

(inputs
 ;; TODO: remove `if' in the next rebuild cycle.
 (if (%current-target-system)
 `(("perl" ,perl)
   ("m4" ,m4))
 '()))
(native-inputs
 `(("perl" ,perl)
   ("m4" ,m4)))


>> +(arguments
>> + `(#:tests? #f
>
> Nope.  :-)

Ahum, yes, well...this made me blush for a moment ("I should not rely on
a review by Ludovic for catching such a development hack") but look at
the three preceeding lines:

 ;; XXX: testsuite: 209 and 279 failed. The latter is an impurity. It
 ;; should use our own "cpp" instead of "/lib/cpp".
-(arguments `(#:tests? #f))

Tests are disabled currently (even on master!), so I guess that
inserting this line break is fine :-)

>> +   ,@(if (%current-target-system)
>> + `(#:phases
>> +   (modify-phases %standard-phases
>> + ;; Autoconf cannot be cross-built properly: it lacks the
>> + ;; concept of -for-build.  It even runs the host
>> + ;; `autom4te' (a perl script) during build.
>> + (add-after 'install 'fake-cross-build
>> +   (lambda* (#:key build inputs outputs #:allow-other-keys)
>> + (let ((m4 (assoc-ref inputs "m4"))
>> +   (perl (assoc-ref inputs "perl"))
>> +   (out  (assoc-ref outputs "out")))
>> +  (substitute* (find-files (string-append out "/bin"))
>> +(("/gnu/store/[^/]*-m4-[^/]*") m4)
>> +(("/gnu/store/[^/]*-perl-[^/]*") perl))
>> +  #t)
>
> Why is this needed?  The ‘patch-shebangs’ phase normally takes the
> inputs, not the native inputs, when changing shebangs.

Because it's not only the shebangs...but

Good question...especially because it teaches me about patch-shebangs!
Also, I failed to determine exactly what went wrong.  Without m4, perl
in INPUTS, the shebangs are wrong (obviously).

After adding m4,perl in INPUTS, the shebangs are indeed correct.  I
hadn't noticed that before, because look:

--8<---cut here---start->8---
$ head $(./pre-inst-env guix build --target=i586-pc-gnu autoconf)/bin/autoheader
#!/gnu/store/ drz7805gcsrqkgr8v43r1f7zydlsxh05-perl-5.30.2/bin/perl
# -*- Perl -*-
# Generated from autoheader.in; do not edit by hand.

eval 'case $# in 0) exec 
/gnu/store/8zvc5mvk0xm3ygrxsgpyy5ilxb5rzjry-perl-5.30.2/bin/perl -S "$0";; *) 
exec /gnu/store/8zvc5mvk0xm3ygrxsgpyy5ilxb5rzjry-perl-5.30.2/bin/perl -S "$0" 
"$@";; esac'
if 0;
--8<---cut here---end--->8---

shebangs correct, re-execute EVALs: wrong.

I have now changed this bit to:

   ,@(if (%current-target-system)
 `(#:phases
   (modify-phases %standard-phases
 ;; `patch-shebangs' patches shebangs only, and the Perl
 ;; scripts use a re-exec feature that references the build
 ;; hosts' perl.  Also, M4 store references hide in the
 ;; scripts.
 (add-after 'install 'patch-non-shebang-references
   (lambda* (#:key build inputs outputs #:allow-other-keys)
 (let ((m4 (assoc-ref inputs "m4"))
   (perl (assoc-ref inputs "perl"))
   (out  (assoc-ref outputs "out"))
   (store-directory (%store-directory)))
  (substitute* (find-files (string-append out "/bin"))
(((string-append store-directory "/[^/]*-m4-[^/]*")) m4)
(((string-append store-directory "/[^/]*-perl-[^/]*"))
 perl))
  #t)

> (You previously found that something’s wrong there, but I forgot what…)

(yes, and I did not have the whole story)

Attaching the updated patch in full.

(automake is pretty similar, i'll send an updated patch for that right
after this review is finished).

Greetings,
janneke

>From 3d776e0077d62000f20d23a1b42b32fef718a503 Mon Sep 17 00:00:00 2001
From: "Jan (janneke) Nieuwenhuizen" 
Date: 

Re: TLS certificates for web browsers in guix environment --container

[Leo Famulari]

On Tue, Apr 21, 2020 at 06:17:58PM +0200, Pierre Neidhardt wrote:
> Note that the "--expose=/etc/ssl/certs/" is important.
> 
> Should we consider this a bug?  If not, then should we document
> it?

No, it's not a bug.

TLS X.509 certificates need to be looked up dynamically at run-time,
because their validity depends on the current time. We need to be able
to change the certificates without requiring the packages that use them
to rebuild. Otherwise built packages would become obsolete just because
some time has passed.


signature.asc
Description: PGP signature

Re: TLS certificates for web browsers in guix environment --container

[Pierre Neidhardt]

It works!

--8<---cut here---start->8---
guix environment -C -N --expose=/etc/machine-id --expose=/etc/ssl/certs/ \
  --share=$HOME/.local/share/eolie/=$HOME/.local/share/eolie/ \
  --ad-hoc dbus eolie coreutils nss-certs -- \
  env DISPLAY=$DISPLAY eolie
--8<---cut here---end--->8---

Note that the "--expose=/etc/ssl/certs/" is important.

Should we consider this a bug?  If not, then should we document
it?

Maybe this could be automated a bit.

--8<---cut here---start->8---
guix size webkitgtk glib-networking
--8<---cut here---end--->8---

does not return nss-certs.  So if we made nss-certs an input of
webkitgtk (or glib-networking?), we would not need nss-certs in the guix
environment invocation.

Finally, I'm not sure how to fix the /etc/ssl/certs issue.  Why do we
have to put it under /etc/ in the first place?

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: 04/05: gnu: autoconf: Support cross-build.

[Ludovic Courtès]

Hi!

guix-comm...@gnu.org skribis:

> commit 6833403fb393a513c77aa3cb8fca7d57b87befe1
> Author: Jan (janneke) Nieuwenhuizen 
> AuthorDate: Sat Apr 18 19:49:54 2020 +0200
>
> gnu: autoconf: Support cross-build.
> 
> Autoconf cannot be cross-built properly: it lacks the concept of
> -for-build.  It runs the host `autom4te' (a perl script) during 
> build.
> 
> * gnu/packages/autotools.scm (autoconf)[inputs]: When cross-building, add 
> perl
> and m4.
> [native-inputs]: when cross-building, use -for-build names.
> [arguments]: When cross-building, add `fake-cross-build' phase to 
> substitute
> m4 and perl.

Thanks for fixing it!  Some comments:

> +(inputs
> + (if (%current-target-system)
> + `(("perl" ,perl)
> +   ("m4" ,m4))
> + '()))
>  (native-inputs
> - `(("perl" ,perl)
> -   ("m4" ,m4)))
> + (if (%current-target-system)
> + `(("perl-for-build" ,perl)
> +   ("m4-for-build" ,m4))
> + `(("perl" ,perl)
> +   ("m4" ,m4

You can remove the ‘if’ in both cases: we always need Perl/M4 both as a
native input and as an input in both cases.

However, that might trigger a rebuild, so perhaps you’ll have to leave
the ifs, but with a TODO telling to remove it on the next rebuild or
Marius will be mad at us.

> +(arguments
> + `(#:tests? #f

Nope.  :-)

> +   ,@(if (%current-target-system)
> + `(#:phases
> +   (modify-phases %standard-phases
> + ;; Autoconf cannot be cross-built properly: it lacks the
> + ;; concept of -for-build.  It even runs the host
> + ;; `autom4te' (a perl script) during build.
> + (add-after 'install 'fake-cross-build
> +   (lambda* (#:key build inputs outputs #:allow-other-keys)
> + (let ((m4 (assoc-ref inputs "m4"))
> +   (perl (assoc-ref inputs "perl"))
> +   (out  (assoc-ref outputs "out")))
> +  (substitute* (find-files (string-append out "/bin"))
> +(("/gnu/store/[^/]*-m4-[^/]*") m4)
> +(("/gnu/store/[^/]*-perl-[^/]*") perl))
> +  #t)

Why is this needed?  The ‘patch-shebangs’ phase normally takes the
inputs, not the native inputs, when changing shebangs.

(You previously found that something’s wrong there, but I forgot what…)

Thanks,
Ludo’.

Re: TLS certificates for web browsers in guix environment --container

[John Soo]

Hi Pierre,

I think you need the nss-certs package in the environment, to start. Does 
adding them help?

- John

GNU Shepherd 0.8.0 released

[Ludovic Courtès]

We are pleased to announce the GNU Shepherd version 0.8.0.  This release
represents 31 commits by 7 people, primarily bug fixes and small
additions to the programming interface.

• About

  The GNU Daemon Shepherd or GNU Shepherd is a service manager written
  in Guile that looks after the herd of system services.  It provides a
  replacement for the service-managing capabilities of SysV-init (or any
  other init) with a dependency-based system with a convenient
  interface.  The GNU Shepherd may also be used by unprivileged users to
  manage per-user daemons (e.g., tor, privoxy, mcron, etc.)  It is
  written in Guile Scheme, and is configured and extended using Guile.

  The GNU Shepherd is developed jointly with the GNU Guix project; it is
  used as the init system of Guix, GNU’s advanced GNU/Linux distribution.

  https://www.gnu.org/software/shepherd/


• Download

  Here are the compressed sources and a GPG detached signature[*]:
https://ftp.gnu.org/gnu/shepherd/shepherd-0.8.0.tar.gz
https://ftp.gnu.org/gnu/shepherd/shepherd-0.8.0.tar.gz.sig

  Use a mirror for higher download bandwidth:
https://ftpmirror.gnu.org/shepherd/shepherd-0.8.0.tar.gz
https://ftpmirror.gnu.org/shepherd/shepherd-0.8.0.tar.gz.sig

  Here are the SHA1 and SHA256 checksums:

  1b1cea9c1271ef21611e8b717e9fd37c3c165bdc  shepherd-0.8.0.tar.gz
  940eb3e8a6f2ee710925b35ace3ee003cc8a38ff017a121d471bb5573e628b0a  
shepherd-0.8.0.tar.gz

  [*] Use a .sig file to verify that the corresponding file (without the
  .sig suffix) is intact.  First, be sure to download both the .sig file
  and the corresponding tarball.  Then, run a command like this:

gpg --verify shepherd-0.8.0.tar.gz.sig

  If that command fails because you don't have the required public key,
  then run this command to import it:

gpg --keyserver keys.openpgp.org \
--recv-keys 3CE464558A84FDC69DB40CFB090B11993D9AEBB5

  and rerun the 'gpg --verify' command.

  This release was bootstrapped with the following tools:
Autoconf 2.69
Automake 1.16.2
Makeinfo 6.7
Help2man 1.47.13


• Changes since version 0.7.0 (excerpt from the NEWS file)

  ** Kill the whole process group when the PID file doesn’t show up
 ()
  ** ‘make-kill-destructor’ kills the process group
  ** New ‘default-pid-file-timeout’ SRFI-39 parameter
  ** New #:file-creation-mask parameter for ‘make-forkexec-constructor’
  ** ‘make-forkexec-constructor’ creates log files as #o640
 ()
  ** Improve documentation and examples
  ** Ensure man pages are up to date
 ()
  ** Fix compilation on systems without ‘prctl’ such as GNU/Hurd
  ** Remove kludge that would send SIGALRM every second
  ** Address “error in finalization thread” warning
  ** ‘make-forkexec-constructor’ no longer supports old calling convention

  The first argument must be a list of strings.  Passing several strings has
  been deprecated since 0.1.

Please report bugs to bug-g...@gnu.org.
Join guix-devel@gnu.org and gnu-system-disc...@gnu.org for discussions.

Ludovic, on behalf of the Shepherd herd.


signature.asc
Description: PGP signature

TLS certificates for web browsers in guix environment --container

[Pierre Neidhardt]

Hi!

I'd like to run browsers in `guix environment` which seems to be a good
idea! :)

IceCat has been discussed in the past.  Now I'd like to run
WebKitGTK-based browsers.

Let's try with Eolie for now, since it seems to have less issues than,
say, Epiphany.

--8<---cut here---start->8---
$ guix environment -C -N --expose=/etc/machine-id --expose=/etc/ssl/certs/ 
--share=$HOME/.local/share/eolie/=$HOME/.local/share/eolie/ --ad-hoc dbus eolie 
coreutils -- env DISPLAY=$DISPLAY SSL_CERT_DIR="$SSL_CERT_DIR" 
SSL_CERT_FILE="$SSL_CERT_FILE" eolie

(org.gnome.Eolie:1): Gtk-WARNING **: 12:50:06.747: Could not find the icon 
'go-previous-symbolic-ltr'. The 'hicolor' theme
was not found either, perhaps you need to install it.
You can get a copy from:
http://icon-theme.freedesktop.org/releases
[ERROR] 2020-04-21 12:50:07 DatabasePhishing::__save_rules():Expecting value: 
line 1 column 1 (char 0) -> b'error code: 1020'
[WARNING] 2020-04-21 12:50:08 TaskHelper::__on_request_send_async(): 
g-io-error-quark: Operation was cancelled (19)
[ERROR] 2020-04-21 12:50:08 DatabaseAdblock::__on_load_uri_content(): 
https://adaway.org/hosts.txt
--8<---cut here---end--->8---

It seems to work well except for TLS certificate validation.  I
guess the certificate files cannot be found in the container since they
can be found in a --pure environment:

--8<---cut here---start->8---
$ guix environment --pure --ad-hoc dbus eolie coreutils -- env DISPLAY=$DISPLAY 
eolie
--8<---cut here---end--->8---

so I presume the certificate errors are not due to my environment and
'bin/eolie' exports the right environment variables.

My guess is that webkitgtk, glib-networking, glib or gnutls tries to
fetch the certificates in a well-known location.

Any idea about this?

--
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature