A minimal rootless podman configuration

[Development of GNU Guix and the GNU System distribution.]

Existing attempts to use podman rootless run into issues with the 
cgroups file system being mounted by elogind. Since we now have 
seatd and greetd, we can bypass elogind. Using them, I have 
finally been able to use rootless podman. Since this is something 
that comes up in IRC with some regularity, I wanted to share the 
operating system configuration here.


In case you couldn't tell, you can test this out by logging in 
with the user ‘test’ and password ‘test’. It works in a VM, but 
there are some issues with how ‘/’ is mounted, so you'll get some 
warnings. Once logged in, try: ‘podman run --rm 
docker.io/library/hello-world’. Networking works as well. I was 
able to get a full Rust development environment running in an 
Archlinux container, and everything works as I expect.


It'd be nice if there were a ‘podman-service-type’ where we could 
specify configuration in Scheme, for at least the root-level 
configuration and hopefully the user-level one as well, but I'm 
out of time for a while to work on that.


Enjoy!

#+begin_src scheme
(use-modules (gnu))
(use-service-modules desktop networking)

(define podman-containers-policy
 "{\"default\": [{ \"type\": \"insecureAcceptAnything\" }],
   \"transports\": {
   \"docker-daemon\": {
   \"\": [{ \"type\": \"insecureAcceptAnything\" 
   }]

  }}}")

(operating-system
(locale "en_US.utf8")
(timezone "America/New_York")
(keyboard-layout (keyboard-layout "us"))
(host-name "podman-test")

(bootloader
 (bootloader-configuration
  (bootloader grub-efi-bootloader)
  (targets '("/boot/efi"))
  (keyboard-layout keyboard-layout)))

(file-systems
 (cons* (file-system
 (mount-point "/")
 (device
  (uuid "14f4e958-be9e-41bb-bd25-e90a7330093c"
'btrfs))
 (type "btrfs"))
(file-system
 (mount-point "/boot/efi")
 (device (uuid "6866-56B1" 'fat32))
 (type "vfat"))
%base-file-systems))

(users
 (cons* (user-account
 (name "test")
 (group "users")
 (password (crypt "test" "$6$test")))
%base-user-accounts))

(packages
 (cons*
  (specification->package "nss-certs") ;; podman pull verifies 
  certs

  (specification->package "podman")
  %base-packages))

(services
 (cons*
  ;;
  ;; not strictly required, but without them podman will resort 
  to

  ;; single user mapping.
  ;;
  (simple-service 'subuid-subgid etc-service-type
  (list `("subuid"
  ,(plain-file "subuid"
   (string-join
'("root:65536:65536"
  "test:16777216:65536")
 "\n" 'suffix)))
`("subgid"
  ,(plain-file "subgid"
   (string-join
'("root:65536:65536"
  "test:16777216:65536")
 "\n" 'suffix)

  ;;
  ;; this can also be managed per-user in ~/.config/containers.
  ;;

  ;; TODO: make ‘podman-service-type’ which creates the global
  ;; /etc/containers configs. preferably something that can also 
  be

  ;; used for per-user configs.
  (simple-service 'podman-containers-conf etc-service-type
  (list `("containers/policy.json"
  ,(plain-file "policy.json"
   podman-containers-policy

  (service seatd-service-type
   (seatd-configuration
(loglevel "debug")))

  (service greetd-service-type
   (greetd-configuration
(greeter-supplementary-groups
 '("input" "seat"))
(terminals
 (list (greetd-terminal-configuration
(terminal-vt "1")
(terminal-switch #t))
   (greetd-terminal-configuration
(terminal-vt "2")
(terminal-switch #t))
   (greetd-terminal-configuration
(terminal-vt "3")
(terminal-switch #t))
   (greetd-terminal-configuration
(terminal-vt "4")
(terminal-switch #t))
   (greetd-terminal-configuration
(terminal-vt "5")
(terminal-switch #t))
   (greetd-terminal-configuration
(terminal-vt "6")
(terminal-switch #t))

  (service dhcp-client-service-type)

  (modify-services %base-services
   ;; cgroups conflict with seatd
   (delete elogind-service-type)

   ;; conflicts with greetd terminals
   (delete mingetty-service-type)
#+end_src

-bjc

Preservation of Guix (PoG) report 2023-03-13

[Timothy Sample]

Hi Guix,

It’s been a while!  :)

Allow me to present to you a long-overdue update to the Preservation of
Guix (PoG) report: .  

Note that you can link to the most recent version of the report using
.

What is this?  Well, I added a description to the report itself, but
here’s a brief teaser.  The PoG report shows what we know about the
archival status of the approximately 54K sources (and counting) Guix has
linked to since around the time of the 1.0 release.

For this edition, I took a bit of time to fix the contrast and colours
to be a bit more accessible.  They’re about half as garish as they used
to be, too.

Over the whole set, 77.1% are known to be safely tucked away in the
Software Heritage archive.  But it’s actually much better than that.  If
we only look at the most recent sampled commit (from Sunday the 5th),
that number becomes 87.4%, which is starting to look pretty good!

I have a few more notes on the report, but I want to put this near the
top of the message so that people will see it.  :)  I wrote a script
(see attached) that uses the PoG database to find missing sources on a
packge-by-package basis.  That is, you can run

guix repl specification-to-swhids.scm pog.db bash

and it will print a table of all of the transitive sources needed to
build Bash, along with their preservation status.  Here’s a (heavily
edited and snipped to fit an email message) sample of its output:

[... many “stored” inputs]
sha256 0r5p. swh:1:dir:02f7. stored  /gnu/store/.-gmp-6.0.0a.tar.xz
sha256 0c3k. swh:1:dir:6027. stored  /gnu/store/.-mescc-tar.xz
sha256 1r1z. swh:1:dir:6087. stored  /gnu/store/.-bash-2.05b.tar.gz
sha256 14l0. unknown unknown /gnu/store/.-gcc-4.9.4.tar.bz2
sha256 0m2y. unknown unknown /gnu/store/.-ed-1.17.tar.lz
[... more “unknown” inputs]

(I had to pipe the output to “sort -k 4” to have it sorted by status.)

The first two columns are the Guix hash.  The next two columns are the
SWHID (if known) and whether SWH has it (if known).  That last column is
the store filename (which is nice because it usually tells you what it
is we are looking at).  In this sample, you can see that GMP, MesCC
Tools, and Bash are all safe.  However, we don’t know about GCC 4 and
ed.  This is kinda like an automated version of Simon’s recent
investigation [1].  The “unknown” two are due to Disarchive’s lack of
support for those compression formats.  I just wrote this script today
(mind the rough edges), and I’ve learned a lot from trying it on a few
packages.  It’s a little like a terrifying robotic TODO list, since it
shows a lot of problems, but it’s also exiting because solving all the
problems for the Guix package, say, would be a massive leap forward.
Here’s a rough road map for that based on a glance at the script’s
output:

• Subversion support (for TeX-based documentation stuff, I guess)
• bzip2 support for Disarchive (there are 45 bzip2 tarballs)
• ZIP support for Disarchive (for the 8 ZIP files)
• lzip support for Disarchive (or a workaround for ed)
• Fix some issues (gettext is .tar.gz, but something went wrong)
• Do something with the static bootstrap binaries

[1] https://lists.gnu.org/archive/html/guix-devel/2023-02/msg00398.html

If you want to try it out for yourself, you’ll need to download the
database .  Heads up:
it’s just over 200M, and my server can be pretty slow.

One other stray thought: the script should work with the time machine,
so you can check on packages from the past.  I didn’t test it, but I bet
it’s fine.

Okay.  Here are the rest of my notes about the report itself.

One thing that jumps out at me is 189 Git sources that SWH does not
have.  Usually they have basically all of the non-recursive Git sources.
It’s something to look into.

I also took a quick peek at the 1.9K “unknown” tar-gz sources.  About
39% percent of them are old Rust crates.  It’s a known problem with
Disarchive.  However, 42% of them are old Bioconductor packages.  They
seem to be lost.  It looks like Bioconductor now stores multiple package
versions per Bioconductor version [2], but before version 3.15 that was
not the case.  As an example, take “ggcyto” from Bioconductor 3.10 [3].
We packaged version 1.14.0, and then at some point Bioconductor 3.10
switched to version 1.14.1.  We packaged that, too, but now 1.14.0 is
gone.  I know it’s been discussed before, but I can’t remember what the
conclusion was.  Are these just gone forever?  I’m doing another pass
through all of them and recovering a few from the bordeaux substitute
server, but only a handful.

[2] https://bioconductor.org/packages/3.15/bioc/src/contrib/Archive/DiffBind/
[3] https://bioconductor.org/packages/3.10/bioc/html/ggcyto.html

That’s all for now.  Enjoy the update and the script!


-- Tim

;;; specification-to-swhids.scm
;;; Copyright © 2023 Timothy Sample 
;;;

Re: The  Shepherd gets a service collection

[Adam Faiz]

I imagine we could develop more convenient services like this, such as 
basic command scheduler similar to the ‘at’ command, and a syslogd

implementation.  The latter could be nice for a couple of reasons:
logging would happen from the start and till the end (an improvement
over the external syslogd process), and it could let us provide a nicer
user interface to view logs (taking inspiration from that of
‘journalctl’).

Thoughts?  Ideas?I don't think a command scheduler should be added to the 
Shepherd, isn't that what mcron does?

If mcron has any deficiencies for being used as an `at` command, then that can 
be improved.

Regarding syslogd, I think a better approach is to tell the services to send 
their output to stdout and stderror,
so that logs don't depend on a separate logging service in the first place.
Per-service logging is already implemented in the Shepherd, but could be 
streamlined to have a default logs directory:
https://skarnet.org/software/s6/s6-log.html#loggingchain

I'm already happy with zless/any GUI log viewer, but it would be great to have 
a nicer user interface to view logs nonetheless.

Towards generalized testing (Was: Caching test results separately?)

[Development of GNU Guix and the GNU System distribution.]

Hi Josselin,

On Mon, Mar 13, 2023 at 3:21 PM Josselin Poiret  wrote:
>
> I would really like for tests to move out of build phases
>
> That would require a huge change to Guix though

Fortunately, there is a great precedent for that work in Debian. It is
called autopkgtest. [1]

One package I maintained there, called Lintian, had a comprehensive
test suite that mimicked large parts of Debian including building
packages. When I moved the tests out of the build process, it
shortened the build time from six hours to seven minutes.

That was on Gitlab with external runners provided by a donor.

The key to understanding Debian's autopkgtest process is that, while
the test suites are distributed together with the sources, they
actually test the *installed* versions of the software (which are
loaded separately). That crucial distinction catches important
user-facing errors that were previously undetectable, such as
installation in the wrong path.

We already have many good starting points in Guix. Some truly great
work has been done. [2]

There are also two SRFIs [3][4] although I am personally biased toward
TAP (the "Test-Anything Protocol") [5] and already used that in one of
my Guile modules. [6]

Either way, such testing efforts will make our distribution even
better, especially as we support additional architectures. I would be
eager to cooperate if there is an extra seat at the table.

Kind regards,
Felix

[1] https://people.debian.org/~eriberto/README.package-tests.html
[2] https://luis-felipe.gitlab.io/guile-proba/manual/
[3] https://srfi.schemers.org/srfi-64/srfi-64.html
[4] https://srfi.schemers.org/srfi-78/srfi-78.html
[5] https://codeberg.org/ft/guile-tap
[6] 
https://codeberg.org/lechner/guile-naptcha/src/branch/history/scm/test/naptcha/pseudo-random.scm

Re: Caching test results separately?

[Ryan Prior]

--- Original Message ---
On Monday, March 13th, 2023 at 10:21 PM, Josselin Poiret  
wrote:

> But I would really like for tests to move out of build phases

I've mentioned this previously in IRC as well. Fundamentally, it strikes me as 
wrong that a change which only affects tests, leaving the installed package the 
same byte for byte, should require a full package re-build.

An improvement would be to treat tests, and their pass/fail status, are 
metadata separate from package builds. In Eelco's thesis, testing is described 
as a major motivator for nix: but in his construction, nix's role is to produce 
package builds, and then you run tests on the outputs to decide whether to 
promote a build into production or not. We should use Guix to create testing, 
staging and production deployments, not to gatekeep and prevent packages with 
failing tests from building.

Many packages I've created or upgraded had some or all tests disabled because 
they require a network stack, eg a client library for postgres requires a 
postgres server. This should be easy to model using Guile and Guix. I would 
love to contribute to development in this direction.

Ryan

Re: Caching test results separately?

[Josselin Poiret]

Hi,

"Dr. Arne Babenhauserheide"  writes:

> Hi,
>
> A large part of the build time is being consumed by tests … could we
> separate those phases so a package whose tests succeeded once does not
> have to be rebuild just because its package got garbage collected?
>
> (⇒ keep the test result (boolean) longer than the build result)
>
> Keeping just a boolean that says whether the tests for a given hash
> succeeded could make that pretty cheap and it could avoid half the build
> time for packages that got garbage collected before.

As it stands it's really not possible, as

1) testing is part of the build process itself and
2) we can't look-up any stateful info like this from the building
process (of course!)

But I would really like for tests to move out of build phases, the
advantages would be twofold: less build time for a lot of packages, and
better environment management for tests (we could rely on better
interaction with other packages, more complicated mocking, use linux
namespaces to their fullest extent, etc.).  That would require a huge
change to Guix though, so it's more of a dream than anything concrete.

Best,
-- 
Josselin Poiret


signature.asc
Description: PGP signature

Re: The  Shepherd gets a service collection

[indieterminacy]

Thanks!


PS: It just occurred to me that we might as well rename the new
(shepherd service …) hierarchy to (shepherd sheep …) or even
(shepherd  …).


Its a shame that Directed Acyclic Graphs are part of Guix verbiage,
as otherwise Id be suggesting the name 'dags'.

(Dags are the yellowed clumps of wool that cluster around their 
netherregions.
There is a antipodean phrase, 'rattle yer dags!', to mean 'hurry up!', 
given

the clumps bouncing around when sheep move quickly.

--
Jonathan McHugh
indieterminacy@libre.brussels

Caching test results separately?

[Dr. Arne Babenhauserheide]

Hi,

A large part of the build time is being consumed by tests … could we
separate those phases so a package whose tests succeeded once does not
have to be rebuild just because its package got garbage collected?

(⇒ keep the test result (boolean) longer than the build result)

Keeping just a boolean that says whether the tests for a given hash
succeeded could make that pretty cheap and it could avoid half the build
time for packages that got garbage collected before.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de


signature.asc
Description: PGP signature

Re: bug#61894: [PATCH RFC] Team approval for patches

[Peter Polidoro]

There is a phenomenon in manufacturing quality control where 
sometimes adding inspectors decreases the number of defects that 
get past inspection unnoticed, because one inspector catches a 
defect that another inspector missed, but other times the number 
of unnoticed defects actually goes UP, presumably because if 
inspectors know others are also looking for defects, they, perhaps 
subconciously, think they do not need to look as carefully, 
because another inspector will catch whatever they miss. One 
inspector looking carefully can be better than two inspectors 
looking less carefully.


It would be nice if packages that pull from a "trusted source" and 
that need only a bump in the version number and hash could be 
approved by only one person or, more ideally, zero people, if it 
could be tested and automated somehow. Although perhaps that would 
always be a security risk.


Is there documentation or a roadmap somewhere online for people 
new the community who submit patches, but someday aspire to arise 
to committer status? The roadmap might be a list of books to read, 
tutorials to complete, packages to create, in order to learn 
enough to be able to help with the committer shortage?

The  Shepherd gets a service collection

[Ludovic Courtès]

Hello Guix!

I pushed some changes yesterday that confirm that the Shepherd paves the
way for init system innovation, synergistic cross-domain fertilization,
and delimited continuations:

  
https://git.savannah.gnu.org/cgit/shepherd.git/log/?id=31d21fa083872d500c016b6b3b2587d25510702d

  31d21fa * Add REPL service.
  cd6f3fb * comm: Add 'open-server-socket'.
  c64804f * Add resource monitoring service.

These new services are built into shepherd, allowing users to control it
and to fiddle with it.  The REPL is functional but of course a bit
bumpy: you’d better know what you’re doing.

I imagine we could develop more convenient services like this, such as a
basic command scheduler similar to the ‘at’ command, and a syslogd
implementation.  The latter could be nice for a couple of reasons:
logging would happen from the start and till the end (an improvement
over the external syslogd process), and it could let us provide a nicer
user interface to view logs (taking inspiration from that of
‘journalctl’).

Thoughts?  Ideas?

Ludo’.

PS: It just occurred to me that we might as well rename the new
(shepherd service …) hierarchy to (shepherd sheep …) or even
(shepherd  …).

Re: Using Guix inside a Guix container

[Simon Tournier]

Hi Konrad,

On sam., 18 févr. 2023 at 10:21, Konrad Hinsen  
wrote:

> Both examples are about composing tools freely, without worrying if they
> use Guix internally or now.

Thanks for explaining and the use-cases.  Well, maybe a discussion for
gwl-devel… well, I have started one. :-)


Cheers,
simon

Re: State of core-updates

[Simon Tournier]

Hi,

On ven., 10 mars 2023 at 15:58, Andreas Enge  wrote:

> let me start with a call for help! I realise that it takes me about one
> week and something close to 100GB on my poor 2-core laptop to rebuild
> the bulk of core-updates up to the packages in my profile, and that is not
> sustainable. It also forces me to do a "guix gc" between two runs, with
> the danger of either doing it too late and having to restart the builds
> (lived experience, one week lost), or losing and having to recompile
> store items that effectively have not changed.

I sympathize.  That’s one of the reason [1] why I have not checked out
core-updates yet. :-)

1: https://yhetil.org/guix/86v8k57vey@gmail.com

> So it would be nice if someone could set up a more complete job for
> core-updates on cuirass or QA, 

[...]

> Here is my eclectic selection of packages I would add to the job:

[...]

> Please suggest more leaf packages that exercise your favourite missing
> language or application domain!

I agree.  It could be helpful if Berlin or Bordeaux could build some
manifest for core-updates.  And then, once the manifest builds, we could
add some packages and repeat.

It would avoid that we all build the same things; worse, that each of us
burn many CPU just for knowing what fails.

Well, maybe it could be helpful if now Berlin or Bordeaux starts to
build etc/release-manifest.scm.  WDYT?

Cheers,
simon