Re: The Shepherd on Fibers

[Philip McGrath]

On 3/28/22 20:14, Philip McGrath wrote:
I'm not sure how useful any of that is, but take it for whatever it's 
worth. My overall impression is that the Shepherd on Fibers is a big 
step in the right direction. Congrats on the great work!


-Philip



P.S.: If you can get access to a copy of John Reppy's book "Concurrent 
Programming in ML" (Cambridge University Press, 1999, reprinted with 
corrections in 2007), chapter 7 presents an extended example of a 
software build system. In addition to the other interesting features of 
the problem domain, it provides an example of controlling external 
programs in Concurrent ML style.

Re: The Shepherd on Fibers

[Philip McGrath]

Hi,

On 3/23/22 18:36, Ludovic Courtès wrote:

Hello Guix!

I have pushed a ‘wip-fibers’ branch of the Shepherd:

   https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers

The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.



Exciting!

As I wrote in 
, 
"I haven't programmed with it in Guile at all, but, from my experience 
using Racket's Concurrent ML system, I think it is the right foundation 
for a supervisor daemon."


I still haven't programmed with Guile Fibers, and looking briefly at the 
diff of the "wip-fibers" branch was the first time I'd looked at the 
Shepherd's implementation, but I had a couple thoughts.


First,


Fibers is used in a single-threaded fashion, which is the main
constraint for shepherd since it forks.  That also means that fibers
cannot be preempted, so it’s fully cooperative scheduling.



Would it be feasible for shepherd *not* to fork? Or only to fork in a 
way that cooperates with fibers?


Obviously forking is pretty ubiquitous, but I the 2019 paper "A fork() 
in the road"[1] fairly convincing in its argument that



fork has lost its classic simplicity; it today impacts all the
other operating system abstractions with which it was once
orthogonal. Moreover, a fundamental challenge with fork is
that, since it conflates the process and the address space in
which it runs, fork is hostile to user-mode implementation
of OS functionality, breaking everything from buffered IO
to kernel-bypass networking. Perhaps most problematically,
fork doesn’t compose—every layer of a system from the kernel
to the smallest user-mode library must support it.


I consider a Concurrent ML system a "user-mode implementation of OS 
functionality": indeed, an early version of Racket's thread system 
(where thread = fiber) is one of the examples in Flatt, Findler, 
Krishnamurthi, & Felleisen, "Programming Languages as Operating Systems 
(or Revenge of the Son of the Lisp Machine)" (ICFP 1999)[2].


More concretely, preemption is a big benefit of fibers.

Racket's approach also addresses this part:


There’s one catch: Fibers is currently Linux-only.  The good news is
that work has been done to port it to other kernels via libevent².
Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd.



Racket has a cross-platform C library, "rktio"[3], for accessing 
os-specific functionality. It was refactored into its current form in 
2017 as an early step toward Racket-on-Chez: while it happens to provide 
exactly the functionality Racket needs, it no longer is specific to 
Racket or any particular implementation thereof. That includes 
everything needed to implement the Concurrent ML system and nonblocking 
IO on a variety of OSes and kernels.


In particular, it implements—IIUC primarily in 
"rktio_process.c"—abstractions (over `fork` or alternatives) to start a 
new process running something, with environment, stdin, stdout, and 
stderror wired up ports in the sense of 
`current-{input,output,error}-port`, and use the Concurrent ML system to 
monitor its exit status, send it signals, etc. The Racket-level API is 
documented at [4].


I spend as little time as possible with these C-level sorts of issues, 
and I particularly don't know about special considerations for PID 1, 
but I hope there would be some way to meet Shepherd's needs without 
interfering with Fibers.


Second, I'm a little uneasy about `unwind-protect`:

```
+(define-syntax-rule (unwind-protect body ... conclude)
+  "Evaluate BODY... and return its result(s), but always evaluate CONCLUDE
+before leaving, even if an exception is raised.
+
+This is *not* implemented with 'dynamic-wind' in order to play well with
+delimited continuations and fibers."
+  (let ((conclusion (lambda () conclude)))
+(catch #t
+  (lambda ()
+(call-with-values
+(lambda ()
+  body ...)
+  (lambda results
+(conclusion)
+(apply values results
+  (lambda args
+(conclusion)
+(apply throw args)
```

though maybe it's used only internally and doesn't have to deal with the 
general case: I'm not an expert by any means, but my understanding (from 
a Racket mailing list thread at [5], continued at [6]) is that dealing 
with the general case may be something of an open research question.


As an example of the sort of problem, what if the `body`s evaluate 
normally, but an exception is raised in the dynamic extent of 
`(conclusion)`, causing `(conclusion)` to run again? One possible 
mitigation would be to `set!` a local variable before the normal 
`(conclusion)` and check it in the exception handler.


More generally, of course, `body ...` could escape by some mechanism 
other than a raising an exception.


If you were writing Racket, I would recommend 
`(call-with-continuation-barrier (λ () body ...))`—logically, `body ...` 
isn't re-entrant anyway—but I see 

Re: Hardened toolchain

[Development of GNU Guix and the GNU System distribution.]

Yes it would be easier to add the hardening flags to gcc directly, I just 
wasn't sure whether the maintainers would be open to the idea.

Since the default gcc toolchain version is still on gcc 10, the hardening flags 
could be added to gcc 11. Then the upgrade from gcc toolchain 10 to 11 can 
benefit from the hardening flags, and "only" 1 world rebuild is needed.

Mar 28, 2022, 03:17 by maxim.courno...@gmail.com:

> Hi,
>
> Maxime Devos  writes:
>
>> zimoun schreef op ma 21-03-2022 om 14:34 [+0100]:
>>
>>> > * gcc can be compiled with `--enable-default-ssp --enable-default-
>>> > pie`
>>> > to enforce ssp and pic
>>>
>>> You wrote [1]:
>>>
>>> --8<---cut here---start->8---
>>> (define-public gcc
>>>   (package
>>>     (inherit gcc)
>>>     (arguments
>>>  (substitute-keyword-arguments (package-arguments gcc)
>>>  ((#:configure-flags flags
>>>    `(append (list "--enable-default-ssp" "--enable-default-pie")
>>>     ,flags)))
>>> --8<---cut here---end--->8---
>>>
>>
>> I think it would be a lot simpler to just add this to the 'standard'
>> gcc configure flags, in (gnu packages gcc), given that probably the
>> idea is to do this hardening for all packages?  Needs a world-rebuild
>> though.
>>
>
> +1.  The whole distribution can probably benefit from this hardening.
>
> Maxim
>

Improving importers best investment for growing gnu/packages/

[Brendan Tildesley]

On 25/3/22 18:05, Paul Alesius wrote:


Rust analyzer is a language server for the Rust programming language.

I've tried to produce a patch for the latest version, and it has
probably hundreds of dependencies that need to be updated.


[...]


I'd suggest that a Guix package for rust-analyzer is not needed,
especially due to the excessive time required to update its package
definition and all of the vendored dependency crates, and focus should
instead be on rust and rust-cargo.


I understand how defeating it feels when looking at the countless hundreds of
dependencies that get pulled in, however, I believe this can be solved by
improving the importers in Guix.  I think it requires too much manual labour to
maintain packages curretly, particularly when refresh -u will only update the
package version and not dependencies.

Therefore I'm working on making use of
https://github.com/rust-lang/crates.io-index  to fully import base definitions
all required crates rather than pulling metadata from the internet with each 
refresh.

Maintaining rust packages then becomes mostly
1. Refresh generated package definitions
2. In a separate file edit any hacks/fix needed argument fields etc
3. Edit cargo-build-system to account for the most common hacks like setting
LIBCLANG_PATH.

It should be easy for a guix user to get all the latest crates that they would
get using cargo.

I think importers can be taken to the next level by going beyond metadata and
actually inspecting the contents of source files. For example grepping build
files it can be determined if pkg-config is used and add it to inputs. an index
of .pc files can automatically detect and add the inputs a package is most
likely looking for. One complaint about Guix is that typically just running
./configure; make on a project will not work, but if a guix can have a Universal
Importer where running guix wave-magic-wand in a directory can inspect files,
determine the build system, required inputs etc and build a program, would that
not be wonderful for programmers? Working on KDE made me feel like I was wasting
too much time manually adding inputs and would be better of writing an importer.

Using rust in a hypothetical, imagine if a rust developer that is otherwise
uninterested in Guix, they just want to build their software and distribute
actually chooses to use Guix not due to idealism but because it:
A. Handles their rust dependecies at least as well as cargo.
B. Handles their non-rust dependencies even better.
C. Improved Guix pack/build can produce distributable packages, stripped of
uneeded files that actually runs on other distributions.

Would that not be awesome?

Re: better error messages through assertions

[Philip McGrath]

Hi,

On 3/7/22 05:13, Ludovic Courtès wrote:

Hi Philip,

Philip McGrath  skribis:


Racket's state-of-the-art contract system has many features and nuances. I *do
not* think anyone should try to implement them all in one fell swoop. I'm
hoping there's a way to implement your simple assertions with only a modest
amount of overhead that will provide the right base on which to grow the rest
of a contract system. In the short term, the advantage over:


 (assert-type (listof service?) services
  "SERVICES must be a list of  values.")


is that you don't have to write error messages by hand.

You need two types of values:

  1. Contracts, recognized by `contract?`; and
  2. Blame objects, recognized by `blame?`.


[...]

Thanks for the explanation and references!  I had briefly looked at
Racket’s contract API in the past but your message gave a clearer view
of how this all fits together.



I'm glad this is something Guix people are interested in!


I would love to have contracts in Guix, even very rudimentary contracts. If
it's something the community more generally would be interested in, I'd be
glad to help as much as I can.


It’d be great to benefit from your expertise here.  Like you wrote, I
think we should start with a simple contract system, certainly simpler
than Racket’s, and build from there.

If you’re willing and able to spend time prototyping this, that’s great.
:-)



I'm interested in putting together a prototype.

I've taken my own suggestion and asked the Racket community for more 
advice: 
https://racket.discourse.group/t/advice-on-implementing-a-contract-system/832


To quote the end of my last message there,


The tl;dr of all that is that `(guix records)` seems to ultimately call for 
"indy-dependent" contracts[1].

On the one hand, the distinction between "indy-dependent" `->i`[2] and 
"lax-dependent" `->d`[3] is exactly the sort of hard-learned lesson that I hope the Guix 
community can draw from Racket's decades of experience.

On the other hand, I'm increasingly intrigued by the idea of starting with 
forms along the lines of `invariant-assertion`[4] and `struct-guard/c`[5] and 
truly sticking to flat contracts to start with, leaving all the higher-order 
complexity for another day.


I'm thinking that a reasonable place to start might be to implement a 
`contract->sanitizer` form that would allow using contracts to create 
sanitizers, ideally with no changes to `(guix records)`.


In addition to the questions about contract system design, I realized I 
have a few questions about Guix/Guile that would be relevant when 
starting a prototype.


What is the preferred mechanism for exceptions? I know about:

  * (rnrs exceptions)
  * (ice-9 exceptions)
  * (srfi srfi-34)
  * (srfi srfi-35)

and IIRC I've seen more than one of them used in the Guix codebase.

Likewise, what record system should I use? I think the answer should 
*not* be (guix records): instead, I think (guix records) should 
eventually use (guix contracts). But should I use:


  * (rnrs records syntactic)
  * (rnrs records procedural)
  * (srfi srfi-9)
  * (oop goops)

Of those, I'm most familiar with R6RS records. I know (guix records) is 
implemented on top of (srfi srfi-9), though I vaguely remember some 
discussion about potentially changing that.


Also, I don't know much about how the "abi" aspect of (guix records) 
works and what types of changes there would trigger rebuilds. (Though, 
again, I hope no changes would be needed for the proof-of-concept phase.)


Finally, when I looked again at the example at the top of this thread:

On 2/14/22 17:32, Ricardo Wurmus wrote:

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure struct-vtable: Wrong type argument in position 1 (expecting 
struct):
--8<---cut here---end--->8---

As you can probably tell easily by looking at this message, the
“service” field of the operating system configuration looked something
like this:

  (services (append (list a b c %desktop-services) #;oops))

instead of this

  (services (append (list a b c) %desktop-services))

This is because INSTANTIATE-MISSING-SERVICES — and FOLD-SERVICES, and
many more — assumes that it is only passed a plain list of services.  It
then proceeds to call SERVICE-KIND on what may or may not be a service.


Another problem here seems to be the fault of (srfi srfi-9). For example:

```
$ guile
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> ,use (srfi srfi-9)
scheme@(guile-user)> (define-record-type container (make-container 
contents) container? (contents container-contents))

scheme@(guile-user)> (container-contents '())
ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In 

Re: Fwd: [Libre-soc-dev] CLEAR, Open Source FPGA

[jbranso]

March 28, 2022 1:02 PM, "Tobias Platen"  wrote:

> If they succeed the free toolchain can be packaged in guix.

I am not so well versed in hardware...Why is this exciting?

It only has 3kb of RAM.  This could power an mp3 player?   maybe?

Fwd: [Libre-soc-dev] CLEAR, Open Source FPGA

[Tobias Platen]

If they succeed the free toolchain can be packaged in guix.
--- Begin Message ---
https://groupgets.com/campaigns/1003-clear-the-open-source-fpga-asic-by-chipignite

please do share that widely, there are a few days to go before they reach 200.  
both the HDL and the toolchain are Libre, they deserve to succeed.

l.

___
Libre-soc-dev mailing list
libre-soc-...@lists.libre-soc.org
http://lists.libre-soc.org/mailman/listinfo/libre-soc-dev
--- End Message ---

Re: using srfi-189 in (gnu services configuration)

[Maxime Devos]

Attila Lendvai schreef op ma 28-03-2022 om 14:35 [+]:
> this is a follow up to: using an SRFI that is not available in Guile
> 
> https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00249.html
> 
> let me summarize the discussion, and with that my argument why i'd
> like to use srfi-189 in the configuration code:
> 
> - sometimes we need to be able to unambiguously distinguish whether a
>   config field value has been specified by the user or not. [...]
> 
>   in the current setup, simply specifying a default value would make
>   it impossible to distinguish, because [...]
> 
> - the current code uses the symbol 'DISABLED

It's a bit of a distraction to the discusses issue, but in Guile
Scheme, symbols are case-sensitive, so (not (eq? 'disabled 'DISABLED)).

>  as a special field value
>   to signify that the field has not been set (i.e. what Nothing would
>   mean if we used srfi-189). it is rather confusing, because many
>   config fields are boolean fields, where 'DISABLED sounds like a
>   valid off value. it is also prone for clashes with user specified
>   values.
> 
> - the current codebase also uses 'UNDEFINED as yet another special
>   marker. once i understood, but unfortunately, i have forgotten what
>   for since then... looks like only as a marker in the macro for the
>   situation when no default value form has been specified for a
>   field's definition.
> 
> - using symbols as markers for special values is a bad idea, because
>   the user may specify a field type to be SYMBOL?, which wouldn't
>   error when the value is 'DISABLED.
> 
> - we can't use Guile's *UNSPECIFIED* for this, because the underlying
>   record implementation of Guile uses it for pretty much the same
>   thing, and it errors whenever this value is encountered in a
>   record's field.

This does not appear to be true, at least for (srfi srfi-9) records:
the following code can put *unspecified* in Guile records without any
errors:

(use-modules (srfi srfi-9))
(define-record-type 
 (make-foobar foo) foo? (foo foobar-foo))

(pk 'foobar (make-foobar *unspecified*))
;;; (foobar #< foo: #>)

Anyway, even if *unspecified* causes problems, this can be resolved by
introducing a new constant like *unspecified* or the symbol 'disabled',
but without the potential confusion with a symbol.  E.g.:

(define-values (*unset-configuration-value* unset-configuration-value?)
  (let ()
(define-record-type 
  (*make-unset-configuration-value*) unset-configuration-value?
  unset-configuration-value?)
(values (*make-unset-configuration-value*)
unset-configuration-value?)))

srfi-189 is also an option, but it seems to me that Haskell-style
Maybe/Just/None that would require lots of wrapping and unwrapping
which seems a bit tedious to me -- doable and definitely an option, but
potentially tedious.

Additionally, for your Swarm example, would something like the
following work:

   ;; defined in (gnu services cryptocurrencies) or such
   (define swarm-testnet
 (swarm-instance
   (bootstrap-peers (list "x.y.z.w" "::::4"))
   (foo-rate 1.5+2i)
   ...))
   (define swarm-mainnet [...])

   (swarm-configuration
 (ethereum-account ...)
 (port 12345) ; default: 54321
 ;; If the user known what they are doing, they can override
 ;; 
 (swarms (list swarm-testnet swarm-mainnet)))

?  This way, the well-known swarms 'testnet' and 'mainnet' do not have
to be special-cased.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

using srfi-189 in (gnu services configuration)

[Attila Lendvai]

this is a follow up to: using an SRFI that is not available in Guile

https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00249.html

let me summarize the discussion, and with that my argument why i'd
like to use srfi-189 in the configuration code:

- sometimes we need to be able to unambiguously distinguish whether a
  config field value has been specified by the user or not.

  the reason is that in some situations setting a config value by the
  user is actually an error; e.g. when field A's value is derived from
  field B's value, but only sometimes, depending on the actual value
  of B.

  in the current setup, simply specifying a default value would make
  it impossible to distinguish, because by the time the code of the
  service is executed, the default value is already written into the
  field.

- the current code uses the symbol 'DISABLED as a special field value
  to signify that the field has not been set (i.e. what Nothing would
  mean if we used srfi-189). it is rather confusing, because many
  config fields are boolean fields, where 'DISABLED sounds like a
  valid off value. it is also prone for clashes with user specified
  values.

- the current codebase also uses 'UNDEFINED as yet another special
  marker. once i understood, but unfortunately, i have forgotten what
  for since then... looks like only as a marker in the macro for the
  situation when no default value form has been specified for a
  field's definition.

- using symbols as markers for special values is a bad idea, because
  the user may specify a field type to be SYMBOL?, which wouldn't
  error when the value is 'DISABLED.

- we can't use Guile's *UNSPECIFIED* for this, because the underlying
  record implementation of Guile uses it for pretty much the same
  thing, and it errors whenever this value is encountered in a
  record's field.

- i see only one way to implement this in the current setup that may
  be doable: use DEFINE-RECORD* (already a loss of many features of
  CONFIGURATION), use thunked fields, and squeeze the logic into the
  default thunk of every field separately.

  at least in my case, it would force a rather unnatural shape on the
  code. understanding the code would be only possible if the reader
  has a proper understanding of thunked fields and what is executed
  when -- which is arguably a harder requirement than grasping Maybe
  and Nothing.

- srfi-189's Maybe and Nothing may come useful in other parts of the
  Guix codebase.

the Maybe and Nothing types/abstranctions implement a solution exactly
for this problem: the ability to detect and deal with the/a special
Nothing value.

the first stage of this adventure, namely adding guile-srfi-189 to the
packages, has been merged.

now, the second stage is going to be a non-trivial task for me,
therefore before i venture into incorporating the use srfi-189 into
the configuration codebase, and before we can see what the actual
implementation looks like, i'd like to ask the maintainers to speak up
if either:

 1) they have been convinced that this may actually turn out
well, or

 2) if they still have strong feelings against this venture, and would
probably oppose the use of sfri-189, regardless of the qualities
of the resulting patch.

any feedback is appreciated,

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Learning without thinking is useless. Thinking without learning is dangerous.”
— Confucius (551–479 BC), 'The Analects'

Re: Hardened toolchain

[zimoun]

Hi,

On Sun, 27 Mar 2022 at 23:17, Maxim Cournoyer  wrote:
> Maxime Devos  writes:

>> I think it would be a lot simpler to just add this to the 'standard'
>> gcc configure flags, in (gnu packages gcc), given that probably the
>> idea is to do this hardening for all packages?  Needs a world-rebuild
>> though.
>
> +1.  The whole distribution can probably benefit from this hardening.

(Parenthesis, the initial question is about how to create a custom gcc,
somehow whatever the options are about, and my answers are in this
direction and not in supporting directly in Guix some variants or even
create a new upstream .  To me, that “a lot simpler” is orthogonal. :-)
Closing parenthesis.)


Yes, for sure, it can be a good idea to follow the “Arch Linux” hardened
flags.  The two question I have are:

 1. Is it well-supported for cross-compiling?

 2. Do we introduce the hardened flags for compiling the hardened
 compiler?  Other said, at which bootstrap level in the chain do we
 introduce these hardened options?


Cheers,
simon