Re: Upgrading Shepherd services

[Development of GNU Guix and the GNU System distribution.]

Hi Attila,

On Thu, May 23 2024, Attila Lendvai wrote:

> maybe it fails for you due to some missing modules that are available
> in my test env?

Thanks for trying that out locally!  Still no go here.  I can restart
the upgraded services and trigger the timers, but my system won't boot.

> this below is with my shepherd branch

I see some services starting but no errors on the console.  Also, there
is absolutely nothing in /var/log/messages.  Would it help to diagnose
it using your Shepherd branch?

Kind regards
Felix

Re: How to write a compute-file producing utf8 output?

[Development of GNU Guix and the GNU System distribution.]

Hi Tomas,

On Thu, May 23 2024, Tomas Volf wrote:

> As you can see, ? does not equal 猫.

Maybe it helps you that this file works:

(use-modules (gnu packages base)
 (guix gexp))

(computed-file
 "utf8-display"
 #~(with-output-to-file #$output
 (λ ()
   (use-modules (ice-9 binary-ports)
(rnrs bytevectors))
   (put-bytevector (current-output-port)
   (string->utf8 "猫\n")

Diagnosing encoding issues can be tricky.  Therefore:

$ guix shell xxd -- xxd 
/gnu/store/qqljfpzks62kph7qqdv26aj76r2q14nz-utf8-display
: e78c ab0a

The present issue reminds me---and perhaps you---of
https://issues.guix.gnu.org/69381.

Kind regards
Felix

Re: Upgrading Shepherd services

[Attila Lendvai]

hi Felix,


> Here is a small one for not booting, although the service activation
> during 'guix deploy' succeeds.
>
> Please try the Guix timer below with the Shepherd development branch.
> My equipment does not boot when the apparently erroneous (actions ...)
> field in the shepherd-service record is present.


i cannot reproduce this.

maybe it fails for you due to some missing modules that are available in my 
test env?

this below is with my shepherd branch, but later i double checked with vanilla 
'devel', and it works the same.

# herd trigger garbage-collector
Triggering timer.
#

herd[210]: [debug] Got a reply, processing it
shepherd[1]: [debug] fork+exec-command for (guix gc --free-space=1G), user #f, 
group #f, supplementary-groups (), log-file #f
shepherd[1]: [debug] exec-command for (guix gc --free-space=1G), user #f, group 
#f, supplementary-groups (), log-file #f, log-port #
shepherd[1]: Timer 'garbage-collector' spawned process 212.
shepherd[1]: [debug] query-service-controller; message status, service 
#< provision: (garbage-collector) requirement: (guix
shepherd[1]: [debug] query-service-controller; message running, service 
#< provision: (garbage-collector) requirement: (gui
shepherd[1]: [guix] guix gc: already 30082.59 MiBs available on /gnu/store, 
nothing to do
shepherd[1]: Process 212 of timer 'garbage-collector' terminated with status 0 
after 1 seconds.

HTH,

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“There are two ways to be fooled. One is to believe what isn't true; the other 
is to refuse to believe what is true.”
— Søren Kierkegaard (1813–1855)

Re: [shepherd] several patches that i deem ready

[Attila Lendvai]

hi Ludo,


> nevertheless, i'll rebase my work on the devel branch eventually. it
> will be a lot of pain in itself, but if i need to reimplement/rebase
> stuff by hand anyway, then i'll try to further sort the commits in a
> least-controversial order.


i've rebased my commits on top of the devel branch, and in the process i've 
reordered them into a least controversial order for your cherry-picking 
convenience:

https://codeberg.org/attila-lendvai-patches/shepherd/commits/branch/various

i just started a wave of deeper testing after the rebase, so the more complex 
commits may change, but those need further work/negotiation anyway.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“We have been to the moon, we have charted the depths of the ocean and the 
heart of the atom, but we have a fear of looking inward to ourselves because we 
sense that is where all the contradictions flow together.”
— Terence McKenna (1946–2000)

Postmortem of service downtime

[Ludovic Courtès]

From Sunday May 19th to Tuesday may 21st, for about 36h,
bayfront.guix.gnu.org, the machine behind many services went down:

  https://lists.gnu.org/archive/html/info-guix/2024-05/msg0.html

Affected web sites and services included:

  guix.gnu.org
  bordeaux.guix.gnu.org
  logs.guix.gnu.org
  hpc.guix.info
  foundation.guix.info
  packages.guix.gnu.org
  qa.guix.gnu.org

Here’s the series of events that led to this:

  • The machine had not been rebooted for 7 months and needed to be
rebooted to run a newer version of Shepherd (it was on 0.10.2, which
had a bug regarding replacements that is fixed in newer versions:
).

  • The machine did not reboot.  There’s no IPMI (this fully free system
we acquired some years ago did not support it), so all we have is a
remote-controlled power controller that allows us to turn it on and
off.  This had no effect though: the machine didn’t come back.

Fellow hackers of Aquilenet, the non-profit ISP that rents the bay
in the data center where bayfront is, are looking into setting up
serial console access to the machine for us.

  • We (Andreas and myself) scheduled an intervention in the data center
where it is, in Bordeaux (France), and could only get there on
Tuesday morning.

  • The machine was failing to boot because of an error in the Shepherd
config (unbound variable), now fixed:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=97a31249793b8af9923f915140a6732539e9d2a3

The underlying problem is that an error in a non-essential service
would prevent the machine from booting.  This issue is being tracked
here:

  https://issues.guix.gnu.org/71144

Such errors can be detected by testing the config in ‘guix system
vm’, at the cost of extra time for sysadmins.

  • Pulling and reconfiguring the machine was extremely slow.  This is
in part due to spinning disks, and in part due to the fact that we
had to pull the right commit that would allow us to not rebuild
Linux-libre locally (substitutes for the latest upgrade, from
Monday, were unavailable; also we had to pass
--substitute-urls=https://hydra-guix-129.guix.gnu.org in lieu of the
default https://bordeaux.guix.gnu.org, which was unavailable).

A large part of the slowness was due to ‘guix substitute’ reading
all the 300K+ entries from /var/guix/substitute/cache and deleting
them, one by one (this took several minutes).  Chris had mentioned
that performance issue in the past; it’s not much of a problem on
one’s laptop with an SSD, but it’s clearly a problem here where
there are more entries than usual.  We should at least drastically
reduce the TTL of cache entries.

  • qa-frontpage failed to build when we first reconfigured the machine,
so we commented it out.  This is now fixed:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=3fecb1e8fdea65a7440fec403c1c52da197b5dfe

  • guix-packages-website (the server behind packages.guix.gnu.org)
still refuses to start with an Artanis error:

  https://issues.guix.gnu.org/71138

Ludo’, on behalf on the emergency rescue^W^W sysadmin team.


signature.asc
Description: PGP signature

How to write a compute-file producing utf8 output?

[Tomas Volf]

Hello Guix,

I am currently struggling when trying to create a computed-file producing
internationalized content.  Simple reproducer is:

  (use-modules (gnu packages base)
   (guix gexp))

  (computed-file
   "utf8-display"
   #~(with-output-to-file #$output
   (λ ()
 (display "猫\n"

This, when executed and printed, gives following result:

  /tmp $ guix build -e '(load "/tmp/yy.scm")'
  /gnu/store/fyr9h82v9hhbjm4f7lqsdnhs4c45sn3y-utf8-display
  /tmp $ cat /gnu/store/fyr9h82v9hhbjm4f7lqsdnhs4c45sn3y-utf8-display
  ?

As you can see, ? does not equal 猫.  I would expect the root cause to be the
same as #66777.

Any ideas what can I do about this?  Or, even better, why this does not just
work out of the box?

Thanks for your help,
Tomas

--
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: Shepherd log rotation service

[Development of GNU Guix and the GNU System distribution.]

Hi Ludo'

On Sat, May 18 2024, Ludovic Courtès wrote:

> I’ve just pushed a simple log rotation service for Shepherd:

Sorry to double up on my messages.  I might further rename the somewhat
generic sounding %default-calendar-event to %default-rotation-event, or
perhaps even %default-rotation-calendar-event.

Also, the #:compression parameter does not appear to be passed on to
rotate-service-logs here [1] meaning it'll always use 'gzip.

Kind regards
Felix

[1] 
https://git.savannah.gnu.org/cgit/shepherd.git/tree/modules/shepherd/service/log-rotation.scm?h=devel&id=0484726801c2b5c1b7deecb9a96054c2c510ac71#n204