Re: Medium-term road map

[raingloom]

On Sat, 25 Apr 2020 15:37:44 +0200
Ludovic Courtès  wrote:

> Hello Guix!
> 
> We released 1.1.0, but what’s coming next?  What would you like to
> see?
> 
> There are many exciting things being developed and great ideas
> floating around.  For myself, I feel like focusing on “consolidating”
> what we have in the coming weeks.  Here are the areas I hope to focus
> on (and embarking as many of you as possible :-)):
> 
>   1. Authentication.  I want to finally provide a standardized
> mechanism to allow channels to be authenticated upon ‘guix pull’.
> “make authenticate” was a first milestone, let’s get it done.  See
>  .
> 
>   2. Performance.  There are many things we can improve there, first
> and foremost: the “Computing derivation” part of ‘guix pull’, Guile’s
>  compiler terrible time and space requirements, further optimizing
>  core operations like ‘package-derivation’, as well as low-level
>  stuff as described at .
> 
>  Related to that is the question of substitute availability,
> another major hindrance to usability.  We should address this both in
> the build farm (reducing the
>  time-to-push-to-time-of-substitute-availability, tracking random
>  build failures), and on the client side (can we provide ways for
>  users to pull to a commit that won’t require them to build
>  everything from source, while not compromising on their
> security?).
> 
>   3. G-exps.  We should really finish the migration to gexps, as in
> the ‘wip-build-system-gexp’ branch, and adjust our APIs accordingly.
> 
>   4. User interface.  Let’s get our act together with ‘guix shell’ and
>  ‘guix run-script’, and let’s address other annoyances that
>  newcomers keep stumbling upon!
> 
> Thoughts?
> 
> Happy hacking!  :-)
> 
> Ludo’.
> 

I would love to see better tools for debugging boot time issues and a
way to not have to rewrite the entire target directory during a
repeated `guix system init` that only changes a few things. I don't
want to waste my SSD's write cycles. And it's also pointlessly slow.

Right now the debugging experience in the Guile rescue shell is...
ugh. Not great. GRUB 0.9 has a CLI that's easier to navigate.
What I'd like to see:
 - tab autocomplete
 - recovery from errors
The latter is quite hard, because certain steps don't work well when
run multiple times. For example, if you want to re-run the procedure
that mounts essential file systems, it will complain that /dev is
already mounted, instead of just realizing that it can skip mounting it.

Something equivalent to `arch-chroot` would also be nice, again for
debugging. I'm sure there is some way to make modifications to a store
on a different device, but it's not easy to find.

I've written something simple that adresses the overwriting issue, but
since it's a modification to Guix itself, it triggered a full rebuild
of the world, and I did not have the patience to sit through that, so I
just installed a more minimal system instead.

Re: initrd: why do we need static linking?

[raingloom]

On Tue, 05 May 2020 11:33:17 +0200
Ludovic Courtès  wrote:

> Hi,
> 
> Ricardo Wurmus  skribis:
> 
> > why do we need to link packages statically to include them in the
> > initrd?  Can’t we just copy the package closure into the initrd?  
> 
> You can refer to any package in the initrd gexp and its whole closure
> is automatically included in the initrd.
> 
> However, to minimize the size of the initrd (it must fit in memory,
> uncompressed), we typically use stripped statically-linked packages.
> 
> HTH!
> 
> Ludo’.
> 

It'd actually be nice if static linking could be used in places other
than the initrd. I'm experimenting with some old resource constrained
machines, and it made me realize just how much space even a basic `guix
pull` can take up.

Re: GNU Guix maintainer collective update

[Björn Höfling]

On Mon, 04 May 2020 09:31:46 -0400
Maxim Cournoyer  wrote:

> Let's take a moment to thank Ricardo for his time as a Guix
> co-maintainer.  He's been immensely useful to the project and I hope
> we'll continue to see him around.

Thanks Ricardo for being a kind member of Guix and for really
supporting and fostering this kind, respectful community. I never found
an online community that is as respectful as the one here and I learned
a lot from you online, offline and within Outreachy.

And thanks for all the work you did. Hope to see you still around.

 
> Let's also welcome Mathieu Othacehe in his new role!  Congratulations,
> Mathieu!  :-)

Congratulations, Mathieu :-)

Björn


pgp1v40H3Wg4f.pgp
Description: OpenPGP digital signature

Re: branch master updated: gnu: Add warsow-qfusion.

[Marius Bakke]

Pierre Neidhardt  writes:

> Marius Bakke  writes:
>
>> On a related note, the 'ccl' package has a similar issue, and actually
>> produces an invalid derivation on aarch64 and armhf.  It can be seen by
>> running e.g. 'guix weather -s armhf-linux'.  I've been meaning to file a
>> bug about it forever, but typically just comment the package and forget
>> about it.  :-/
>
> I don't understand why the `ccl' package would have an invalid
> derivation on armhf-linux: All `match' seem to be correct, but I can't
> test at the moment.  Are you sure this is a similar issue?

Admittedly it's only tangentially related.  It is really easy to test by
running 'guix build -s armhf-linux --no-grafts -d ccl'.


signature.asc
Description: PGP signature

Re: branch master updated: gnu: Add warsow-qfusion.

[Marius Bakke]

Pierre Neidhardt  writes:

> Marius Bakke  writes:
>
 +  (let ((commit "c4de15df559410aff0ca6643724e24cddb0ecbbd")
 +(arch (match (or (%current-target-system)
 + (%current-system))
 +("x86_64-linux" "x86_64")
 +("i686-linux" "i386")
 +(_ ""
 Don't do this if you just need to determine architecture at build time.
>>>
>>> Just to be sure, you suggest moving this inside the 'really-install
>>> phase, right?
>>
>> Yes.  Otherwise it gets needlessly evaluated by the Guix "front-end"
>> every time one uses 'guix search' etc -- even on unsupported
>> architectures.  It's better to do it at build time, especially when it
>> is only used within a single phase.
>
> I don't understand the difference.
>
> (match (or (%current-target-system)
>(%current-system))
>("x86_64-linux" "x86_64")
>("i686-linux" "i386"))
>
> must be unquoted inside the phase, so it's evaluated when the package
> is and not at build time, isn't it?
>
> Else how do you match against the build architecture at build time?

You can use (lambda* (#:key target system #:allow-other-keys)) to access
those variables at build time.


signature.asc
Description: PGP signature

Re: unexpected reproducibility of reproducible blog post?

[Konrad Hinsen]

Hi Ludo,

> Grafts are normal derivations, and they’re deterministic: it’s just
> about replacing a set of strings by another set of strings.
>
> On the implementation, see also
> .

Thanks for the clarification. That post makes it clear that grafts only
modify the build process of grafted packages.

Cheers,
  Konrad

Re: 18/36: services: hurd: Add dummy loopback.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi,

>> Something like that: sizeof-ifconf is set to 16 (like on x86_64), but
>> it's size should be 8.
>
> Oh, that’s because ‘define-c-struct’ does all the work at
> macro-expansion time, with the host types and alignment constraints.  I
> wonder how we didn’t notice it earlier.  Cross-compiling from x86_64 to
> ARMv7 has the same problem, but maybe the networking ioctls work by
> chance.
>
> The patch below appears to fix it:

Yes, it fixes the struct sizes in the cross-compiled .go file.  Thanks!
I have added it as the attached patch to "wip-hurd-vm" for now.  I did

./pre-inst-env guild compile --target=i586-pc-gnu -o syscalls.go 
guix/build/syscalls.scm 

and copied that to the Hurd.

When running this (changed #$... to UPPERCASE)

--8<---cut here---start->8---
(use-modules (guix build syscalls))

(define LOOPBACK? #t)
(define IP "127.0.0.1" )
(define INTERFACE "lo")
(define NETMASK #f)
(define GATEWAY #f)

(format #t "ifreq-struct-size: ~a\n" (@@ (guix build syscalls) 
ifreq-struct-size))
(format #t "sizeof-ifconf: ~s\n" (@@ (guix build syscalls) sizeof-ifconf))
(define (test)
  (let* ((addr (inet-pton AF_INET IP))
 (sockaddr (make-socket-address AF_INET addr 0))
 (mask (and NETMASK
(inet-pton AF_INET NETMASK)))
 (maskaddr (and mask
(make-socket-address AF_INET
 mask 0)))
 (gateway  (and GATEWAY
(inet-pton AF_INET GATEWAY)))
 (gatewayaddr (and gateway
   (make-socket-address AF_INET
gateway 0
(configure-network-interface INTERFACE sockaddr
 (logior IFF_UP
 (if LOOPBACK?
 IFF_LOOPBACK
 0))
 #:netmask maskaddr)
(when gateway
  (let ((sock (socket AF_INET SOCK_DGRAM 0)))
(add-network-route/gateway sock gatewayaddr)
(close-port sock)

(test)
--8<---cut here---end--->8---

Now I see the correct sizes...

root@guixygnu# guile -L . -C . static.scm 
;;; note: source file /root/static.scm
;;;   newer than compiled 
/root/.cache/guile/ccache/3.0-LE-4-4.2/root/static.scm.go
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;   or pass the --no-auto-compile argument to disable.
;;; compiling /root/static.scm
;;; compiled /root/.cache/guile/ccache/3.0-LE-4-4.2/root/static.scm.go
ifreq-struct-size: 32
sizeof-ifconf: 8
Backtrace:
In ice-9/boot-9.scm:
  1736:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
   8 (apply-smob/0 #)
In ice-9/boot-9.scm:
718:2  7 (call-with-prompt _ _ #)
In ice-9/eval.scm:
619:8  6 (_ #(#(#)))
In ice-9/boot-9.scm:
   2806:4  5 (save-module-excursion _)
  4351:12  4 (_)
In static.scm:
 24:4  3 (test)
In guix/build/syscalls.scm:
   1529:8  2 (configure-network-interface "lo" #(2 2130706433 0) 9 # _)
  1464:18  1 (_ _ "lo" _)
In unknown file:
Exception thrown while printing backtrace:
In procedure primitive-call-ip: Wrong type argument in position 1 (expecting 
PRIMITIVE_P): #

ERROR: Value out of range 0 to 4294967295: -1

...but still/again a similar range error; Ideas?

Greetings,
janneke

>From 33fbfcd7f46cf675cc24db92b1110d38afa51ae5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= 
Date: Tue, 5 May 2020 11:53:39 +0200
Subject: [PATCH] syscalls: Cross-build fix for alignof*, sizeof*.

See .
Reported by Jan (janneke) Nieuwenhuizen .

* guix/build/syscalls.scm (sizeof*, alignof*): When cross-compiling, emit code
to call at runtime.
---
 guix/build/syscalls.scm | 23 +--
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/guix/build/syscalls.scm b/guix/build/syscalls.scm
index 73b439fb7d..00d8ceb480 100644
--- a/guix/build/syscalls.scm
+++ b/guix/build/syscalls.scm
@@ -23,6 +23,7 @@
 
 (define-module (guix build syscalls)
   #:use-module (system foreign)
+  #:use-module (system base target)
   #:use-module (rnrs bytevectors)
   #:autoload   (ice-9 binary-ports) (get-bytevector-n)
   #:use-module (srfi srfi-1)
@@ -194,9 +195,14 @@
  (* (sizeof* type) n))
 ((_ type)
  (let-syntax ((v (lambda (s)
-   (let ((val (sizeof type)))
- (syntax-case s ()
-   (_ val))
+   ;; When compiling natively, call 'sizeof' at expansion
+   ;; time; otherwise, emit code to call it at run time.
+   (syntax-case s ()
+ (_
+  (if (= (target-word-size)
+ 

Re: Packaging ‘clang-tools-extra’ (‘clang-tidy’, etc.)

[Ludovic Courtès]

Hello!

Clang-tools-extra is really meant to built within the Clang tree.
Attach is an attempt to build it out-of-tree but so far it fails with
undefined references for Clang libraries and the obvious solution of
adding ‘-l’ flags until it’s happy doesn’t work.

In my original attempt, the ‘clang-tidy’ binary is 40 MiB, and 32 MiB
are text.  This prolly comes from the fact that many Clang libraries
(libclang[A-Z]*.a) are statically linked in each of these programs.

Ideas welcome!

Ludo’.

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 33d863ae11..896b7afb47 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -382,6 +382,52 @@ output), and Binutils.")
 (define-public clang-toolchain-10
   (make-clang-toolchain clang-10))
 
+(define (make-clang-tools-extra clang)
+  ;; XXX: An attempt to build clang-tools-extra outside the Clang tree.
+  (package
+(inherit clang)
+(name "clang-tools-extra")
+(source (origin
+  (method url-fetch)
+  (uri (llvm-download-uri "clang-tools-extra"
+  (package-version llvm-10)))
+  (sha256
+   (base32
+"074ija5s2jsdn0k035r2dzmryjmqxdnyg4xwvaqych2bazv8rpxc"))
+  (patches (search-patches "clang-tools-extra-cmake.patch"
+(arguments
+ '(#:build-type "Release"
+   #:phases (modify-phases %standard-phases
+  (add-before 'configure 'copy-clang-cmake-module
+(lambda* (#:key inputs #:allow-other-keys)
+  (let ((clang-source (assoc-ref inputs "clang-source")))
+(mkdir "../clang-source")
+(with-directory-excursion "../clang-source"
+  (invoke "tar" "xf" clang-source))
+
+;; 'cmake/modules' is in the search path of the
+;; top-level 'CMakeLists.txt', so copy it there.
+(install-file (car (find-files "../clang-source"
+   "^AddClang\\.cmake$"))
+  "cmake/modules")
+#t)))
+  (add-before 'build 'augment-header-path
+(lambda _
+  (let ((context-h (car (find-files "../clang-source"
+"^Context\\.h$"
+(setenv "CPATH"
+(string-append (dirname
+(canonicalize-path context-h))
+   ":" (getenv "CPATH")))
+#t))
+(inputs
+ `(("clang" ,clang)
+   ("clang-source", (package-source clang))
+   ,@(package-inputs clang)
+
+(define-public clang-tools-extra-10
+  (make-clang-tools-extra clang-10))
+
 (define-public llvm-9
   (package
 (inherit llvm-10)
diff --git a/gnu/packages/patches/clang-tools-extra-cmake.patch b/gnu/packages/patches/clang-tools-extra-cmake.patch
new file mode 100644
index 00..96672bb2dd
--- /dev/null
+++ b/gnu/packages/patches/clang-tools-extra-cmake.patch
@@ -0,0 +1,38 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 57bb970..6e5de7c 100644
+--- a/CMakeLists.txt
 b/CMakeLists.txt
+@@ -1,3 +1,33 @@
++cmake_minimum_required(VERSION 3.4.3)
++
++project(Clang-Tools-Extra)
++
++find_package(LLVM REQUIRED HINTS "${LLVM_CMAKE_PATH}")
++list(APPEND CMAKE_MODULE_PATH ${LLVM_DIR})
++
++# This is the place where we put Clang's 'AddClang.cmake'.
++list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")
++
++set(LLVM_TOOLS_BINARY_DIR ${TOOLS_BINARY_DIR} CACHE PATH "Path to llvm/bin")
++set(LLVM_LIBRARY_DIR ${LIBRARY_DIR} CACHE PATH "Path to llvm/lib")
++set(LLVM_MAIN_INCLUDE_DIR ${INCLUDE_DIR} CACHE PATH "Path to llvm/include")
++set(LLVM_BINARY_DIR ${LLVM_OBJ_ROOT} CACHE PATH "Path to LLVM build tree")
++set(LLVM_MAIN_SRC_DIR ${MAIN_SRC_DIR} CACHE PATH "Path to LLVM source tree")
++
++set(PACKAGE_VERSION "${LLVM_PACKAGE_VERSION}")
++
++include(AddClang)
++include(AddLLVM)
++include(TableGen)
++include(HandleLLVMOptions)
++include(VersionFromVCS)
++include(LLVMDistributionSupport)
++
++# Dummy targets to appease CMake.  These are normally found in the
++# Clang tree.
++add_custom_target(clang-resource-headers)
++add_custom_target(ClangSACheckers)
++
+ include(CMakeDependentOption)
+ 
+ add_subdirectory(clang-apply-replacements)

Re: unexpected reproducibility of reproducible blog post?

[Ludovic Courtès]

Hi,

Konrad Hinsen  skribis:

> I looked a bit at grafts. The documentation at
>
>   https://guix.gnu.org/manual/en/html_node/Security-Updates.html
>
> isn't very explicit about the reproducibility of grafts. In particular,
> it doesn't say if a package containing patched binaries retains its
> original hash, or receives a new unique one. With a unique hash, grafts
> would just be a tweak in the build system, and no less reproducible than
> standard builds. It looks like I have to dive into the source code to
> find out!

Grafts are normal derivations, and they’re deterministic: it’s just
about replacing a set of strings by another set of strings.

On the implementation, see also
.
I’m also preparing a post of the recent (pre-1.1.0) changes in that
area.

Ludo’.

Re: 18/36: services: hurd: Add dummy loopback.

[Vincent Legoll]

Hello,

On Tue, May 5, 2020 at 11:23 AM Ludovic Courtès  wrote:
> The patch below appears to fix it:

Is target-word-size checking always equivalent to cross-compiling ?

Aren't there cross-compilation host/target combos where this will
be true, but still there will be other differences (endianness maybe) ?

I don't know if the question really makes sense, though, so please
forgive my ignorance...

-- 
Vincent Legoll

Re: initrd: why do we need static linking?

[Ludovic Courtès]

Hi,

Ricardo Wurmus  skribis:

> why do we need to link packages statically to include them in the
> initrd?  Can’t we just copy the package closure into the initrd?

You can refer to any package in the initrd gexp and its whole closure is
automatically included in the initrd.

However, to minimize the size of the initrd (it must fit in memory,
uncompressed), we typically use stripped statically-linked packages.

HTH!

Ludo’.

Re: 18/36: services: hurd: Add dummy loopback.

[Ludovic Courtès]

Hi,

Jan Nieuwenhuizen  skribis:

>> Could it be ‘ifreq-struct-size’ that’s off on GNU/Hurd, or one of its
>> friends?
>
> Something like that: sizeof-ifconf is set to 16 (like on x86_64), but
> it's size should be 8.

Oh, that’s because ‘define-c-struct’ does all the work at
macro-expansion time, with the host types and alignment constraints.  I
wonder how we didn’t notice it earlier.  Cross-compiling from x86_64 to
ARMv7 has the same problem, but maybe the networking ioctls work by
chance.

The patch below appears to fix it:

--8<---cut here---start->8---
$ ./pre-inst-env guild compile --target=i686-pc-linux-gnu 
guix/build/syscalls.scm -o guix/build/syscalls.go
wrote `guix/build/syscalls.go'
$ guix environment -s i686-linux -C --ad-hoc guile-next -- guile -L . -C .
GNU Guile 3.0.2
Copyright (C) 1995-2020 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> ,use(guix build syscalls)
scheme@(guile-user)> ,m(guix build syscalls)
scheme@(guix build syscalls)> sizeof-ifconf
$1 = 8
scheme@(guix build syscalls)> %host-type
$2 = "i686-unknown-linux-gnu"
--8<---cut here---end--->8---

Thank you!

Ludo’.

diff --git a/guix/build/syscalls.scm b/guix/build/syscalls.scm
index 73b439fb7d..00d8ceb480 100644
--- a/guix/build/syscalls.scm
+++ b/guix/build/syscalls.scm
@@ -23,6 +23,7 @@
 
 (define-module (guix build syscalls)
   #:use-module (system foreign)
+  #:use-module (system base target)
   #:use-module (rnrs bytevectors)
   #:autoload   (ice-9 binary-ports) (get-bytevector-n)
   #:use-module (srfi srfi-1)
@@ -194,9 +195,14 @@
  (* (sizeof* type) n))
 ((_ type)
  (let-syntax ((v (lambda (s)
-   (let ((val (sizeof type)))
- (syntax-case s ()
-   (_ val))
+   ;; When compiling natively, call 'sizeof' at expansion
+   ;; time; otherwise, emit code to call it at run time.
+   (syntax-case s ()
+ (_
+  (if (= (target-word-size)
+ (with-target %host-type target-word-size))
+  (sizeof type)
+  #'(sizeof type)))
v
 
 (define-syntax alignof*
@@ -208,9 +214,14 @@
  (alignof* type))
 ((_ type)
  (let-syntax ((v (lambda (s)
-   (let ((val (alignof type)))
- (syntax-case s ()
-   (_ val))
+   ;; When compiling natively, call 'sizeof' at expansion
+   ;; time; otherwise, emit code to call it at run time.
+   (syntax-case s ()
+ (_
+  (if (= (target-word-size)
+ (with-target %host-type target-word-size))
+  (alignof type)
+  #'(alignof type)))
v
 
 (define-syntax align ;as found in (system foreign)

Re: Adding a subcommand "load-profile"

[Ludovic Courtès]

Hi,

Vagrant Cascadian  skribis:

>> I’m interested in your feedback then.
>>
>> To me, there are already several ways to do that, notably:
>>
>>   1. guix environment (but it’s slower)
>>
>>   2. sh
>>  . /my/profile/etc/profile.sh
>>  exit
>>
>>   3. sh
>>  eval `guix package -p /my/profile --search-paths`
>>  exit
>>
>> Options #2 and #3 are more verbose, and less obvious to someone not
>> familiar with the shell.
>
> The only way i understand how to trivially get a precise environment is
> "guix environment --pure --ad-hoc package1, package2, ... packageN"
>
> Don't options #2 and #3 also include everything from the default profile
> and system profile?

Option #2 yes, option #3 no: there’s ‘--search-paths=exact’ (the
default), ‘--search-paths=prefix’, and ‘--search-paths=suffix’.

Ludo’.