Re: better error messages through assertions

[Ricardo Wurmus]

Philip McGrath  writes:

> As a Racketeer, I think you're half way to reinventing contracts.

Yes, I was in fact thinking of contracts, but shied away from mentioning
them :)  The reason is that I think we can cover a lot of distance with
just a few simple assertions to avoid plowing ahead on bad arguments.

-- 
Ricardo

Re: better error messages through assertions

[Philip McGrath]

Hi,

On 2/14/22 17:32, Ricardo Wurmus wrote:

As you can probably tell easily by looking at this message, the
“service” field of the operating system configuration looked something
like this:

   (services (append (list a b c %desktop-services) #;oops))

instead of this

   (services (append (list a b c) %desktop-services))

This is because INSTANTIATE-MISSING-SERVICES — and FOLD-SERVICES, and
many more — assumes that it is only passed a plain list of services.  It
then proceeds to call SERVICE-KIND on what may or may not be a service.

I think we should add simple type checks, something like this:

   (define (listof pred)
 (lambda (thing)
  (and (list? thing) (every pred thing
   …
   (define (assert-type type-check thing message)
 (or (false-if-exception (type-check thing))
 (report-error (G_ "type error: …\n" message

   ;; Use ASSERT-TYPE in an example procedure.
   (define (do-something-with-services services)
 (assert-type (listof service?) services
  "SERVICES must be a list of  values.")

 ;; Do things…
 (map service-kind services))

What do you think?  There are many different ways of implementing this
(a new variant of DEFINE that also accepts a type declaration, an assert
like above, a fancier assert that composes a helpful error message by
itself, a separate type declaration that is looked up only when the
corresponding procedure is called in a certain context, etc), but I’d
first like to know if there is consensus that we want something like
this.



As a Guix user and contributor, I would love better error messages.

As a Racketeer, I think you're half way to reinventing contracts.

In particular, since the operating system services field is thunked, 
this example already points to the desirability of higher-order contracts.


Using "contracts" need not initially involve all the bells and whistles 
of Racket's contract library. For example, higher-order contracts can be 
implemented with `lambda` in the absence of special runtime support for 
chaperones and impersonators, as illustrated in [1]. But the Racket 
community has accumulated a great deal of both theoretical insight and 
implementation experience in many subtle corners of this problem: I hope 
Guix can build on that experience.


-Philip

[1]: https://docs.racket-lang.org/guide/Building_New_Contracts.html

Re: Missing dependency for emacs-magit

[Liliana Marie Prikler]

Hi Zelphir,

Am Dienstag, dem 15.02.2022 um 09:45 + schrieb Zelphir Kaltstahl:
> Hello Liliana,
> 
> On 2/14/22 20:33, Liliana Marie Prikler wrote:
> > Hi Zelphir,
> > 
> > Am Montag, dem 14.02.2022 um 18:41 + schrieb Zelphir Kaltstahl:
> > > [...]
> > I think we need some multi-level printf debugging here.
> > 
> > First, after  `source "${GUIX_PROFILE}/etc/profile"`, is
> > EMACSLOADPATH
> > correctly pointing to the share/emacs/site-lisp of
> > ${GUIX_EXTRA_PROFILES}/emacs-test-profile?
> > 
> > Second, before and after executing your init.el snippet, what are
> > the
> > contents of load-path?
> > Do they contain /gnu/store/SOME_LONG_HASH-magit-MAGIT_VERSION?
> > 
> > Cheers
> 
> To simplify things, I created a shell script, which does the calls:
> 
> 
> #!/usr/bin/env bash
> 
> set -Eeuxo pipefail
> 
> # source the environment
> GUIX_PROFILE="${GUIX_EXTRA_PROFILES}/emacs-test-profile"
> . "${GUIX_PROFILE}/etc/profile"
> printf "emacs load path: %s\n" "${EMACSLOADPATH}"
> 
> # run emacs
> env XMODIFIERS='' emacs
> 
> 
> This outputs the following:
> 
> 
> [...]
> 
> 
> Magit line is:
> 
> 
> "/gnu/store/f0461m96rhnpkmhjlj06yz058pqyj02d-emacs-magit-
> 3.3.0/share/emacs/site-lisp/magit-3.3.0"
> 
Okay, this means that subdirs.el is correctly expanded.  The next
question regards autoloads.  Is (featurep 'magit-autoloads) t or nil?

> Seems to be the same as without running the `init.el` file regarding
> magit at least. This makes sense, as I don't add things to the load-
> path in the whole `init.el` file, except for:
> 
> 
> (defconst xl-home "~")
> (defconst xl-guix-profiles (concat xl-home "/" ".guix-extra-
> profiles"))
> (defconst xl-emacs-profile (concat xl-guix-profiles "/" "emacs-test-
> profile"))
> (defconst xl-site-lisp-dir (concat xl-emacs-profile "/"
> "share/emacs/site-lisp/"))
> (add-to-list 'load-path
>   xl-site-lisp-dir)
> 
That snippet should not be needed.  As can be seen from EMACSLOADPATH,
xl-site-lisp-dir should already be in load-path.

> (Not sure if defconst is the best thing to use, but emacs was
> complaining when using setq, that I was assigning to variables, which
> are free, and those strings are actually constant, so I figured that
> would be appropriate to do and emacs does not complain about it any
> longer.)
Let-binding would be nicer, but this is not the coding academy awards,
so anything goes.

Cheers

Re: Dropping gzip-compressed substitutes

[Christopher Baines]

Mathieu Othacehe  writes:

>> I like Chris Baines’ idea of decoupling nar distribution from nar
>> building.  If we want to keep nars long enough so that ‘time-machine’ is
>> usable, then storage requirements will keep growing.
>>
>> Perhaps that means we can regularly copy nars “elsewhere” for long-term
>> storage, using nar-herder, rsync, or whatever.  The machine that stores
>> nars long-term has low requirements compared to the build farm because
>> we don’t need to trust it for anything other than storage.  If that
>> makes things easier (and financially viable), a VPS is good enough.
>
> Sure, the VPS would also allow us to have a less European-centric
> hosting. I did not follow closely the development of the
> nar-herder. Chris what improvements this tool would bring compared to a
> rsync based approach?

There's some discussion of this in the README [1].

1: https://git.cbaines.net/guix/nar-herder/about/

I think the short answer for the moment though is the nar-herder doesn't
do anything that you couldn't do with rsync.

I jumped straight in with Guile+SQLite rather than using rsync+files
because I think the performance for various operations will scale better
this way, and it'll lead on to more advanced functionality, like doing
GC like operations, metrics and tagging nars.


signature.asc
Description: PGP signature

Re: Guix Data Service client module

[Christopher Baines]

Ludovic Courtès  writes:

>> The only thing I can see that's required before merging though is the
>> exports. I'm now thinking about this kind of thing (getting data out of
>> the data service) in the context of patch/branch review.
>
> I think there’s a couple of issues that would be nice to address in the
> JSON API of the data service.
>
> First, it’s unversioned, which will make it hard to maintain things
> going forward.  How about adding, say, “/v1” to URL paths, similar to
> what SWH does?

I think that's a good idea.

> Second, there are places where I found inconsistencies or redundancy in
> the API.  For instance there are several JSON schemas for things called
> “branches” (see the FIXME in there).

Sounds like something to investigate as well.

> The API to access package version history, which I think lots of users
> are interested in, is not intuitive IMO:
>
> scheme@(guile-user)> (define s (open-data-service 
> "https://data.guix.gnu.org;))
> scheme@(guile-user)> (car (package-versions (lookup-package s "emacs")))
> $20 = #< string: "27.2" branches: (#< name: "master" 
> repository-id: 1>)>
> scheme@(guile-user)> (car (package-version-history s (car 
> (package-version-branches $20)) "emacs"))
> $21 = #< version: "27.2" first-revision: #< 
> commit: "cc33f50d0e2a7835e99913226cb4c4b0e9e961ae" date: # second: 54 minute: 30 hour: 20 day: 25 month: 3 year: 2021 zone-offset: 0>> 
> last-revision: #< commit: 
> "de38ccf2e0bb2fd21393925c296b65dca7499bd3" date: # 37 minute: 48 hour: 13 day: 4 month: 2 year: 2022 zone-offset: 0>>>
>
> That said, I don’t have any suggestion on this one.
>
> I also wonder if there’s a way to obtain a commit range for a given
> package version, directly, without having to browse the list returned by
> ‘package-version-history’?

I think the "easy" option is to just add API endpoints for useful
queries, like the one you suggest above.

Though, there's probably some way of providing greater access to the
data. For example, if the data service could populate a tripplestore
with information, you could use SPARQL to query that. I think there's
also other query languages like GraphQL that are designed to address
this kind of problem.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Guix Scripts and Unit Testing

[Phil]

This is largely resolved now - turns out my code sucked rather than needing
to do something clever with Guix module importing, which is what I
suspected ;-)

The only compromise I'm making now is that any complicated manifest for
example, I am having to split into a trivial manifest script calling into a
library that I can then unit test - so it's not quite as good as having a
meta switch for manifests or repl scripts, but it's good enough.

On Sun, 13 Feb 2022 at 20:58, Phil  wrote:

> I have a theory about what is causing this - I think it's me not thinking
> about deferred evaluation carefully enough.
>

This is true, but my second post was still missing the point!


>
> What I think is happening is that the propagated-inputs in my newly
> created test-package will have their evaluation delayed until the build
> occurs in the daemon.  I presume at this point my local library I've used
> add-to-load-path to include in the manifest will no longer be accessible
> and hence the eval failure being returned.
>
>
>
The problem was actually much easier to solve than I had originally
over-thought - simply by making my-package- with-my-gurobi a procedure
which returns the package rather than having it defined as a variable that
contains a package all the problems go away apart from a handful of imports
required in the library script.  The my-gurobi-package can also be nested
inside the let statement, and the whole procedure added to the library.

(define (my-package-with-my-gurobi)
  (let* ((test-package-string (get-env-var-or-prompt "GUIX_TEST_PACKAGE"))
 (test-package (specification->package test-package-string))
 (my-gurobi-package ((transform) (specification->package
"gurobipy"
(format #t "~%Setting Test Package: ~a~%" test-package-string)
(package/inherit test-package
 (propagated-inputs
  `(("gurobipy" ,my-gurobi-package) ;; add my new gurobi
,@(alist-delete "gurobipy"
(package-propagated-inputs test-package))) ;; remove the original
gurobi and splice

Re: Assisting reviewing & committing with tags?

[Maxime Devos]

Bengt Richter schreef op di 15-02-2022 om 13:23 [+0100]:
> Hi guix,
> 
> It sounds like a good idea, but ISTM we don't need yet another markup syntax
> if emacs org mode already defines a useful standard that can be adopted.
> 
> The advantage to org mode style [0] -- in commit commentary as well as tags
> would be its scrapability -- i.e., ease of writing an extractor/formatter
> for handy report snippets/pages and web stuff etc., whether implemented
> using foreign shell or within guix.

FYI, I think you responded to the wrong thread.  This thread was about
additional usertags in debbugs for reviewing in Guix, not about markup
languages or org mode.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Investigating a reproducibility failure

[Bengt Richter]

Hi,

On +2022-02-05 15:12:28 +0100, Ludovic Courtès wrote:
> Konrad Hinsen  skribis:
> 
> > There is obviously a trade-off between reproducibility and performance
> > here.
>

I suspect what you really want to reproduce is not verbatim
code, but the abstract computation that it implements,
typically a digitally simulated experiment?

Thus far, "show me the code" is the usual way to ask someone
what they did, and guix makes is possible to answer in great
detail.

But what is really relevant if you are helping a colleague
reproduce e.g. a monte-carlo simulation experiment computing
pi by throwing random darts at a square, to draw a graph
showing convergence of statistically-computed pi on y-axis
vs number of darts thrown on x-axis?

(IIRC pi should be hits within inscribed circle / hits in
1x1 square)

Well, ISTM you can reproduce this experiment in any language
and method that does the abtract job.

The details of Fortran version or Julia/Clang or guile
pedigree only really come into play for forensics looking
for where the abstract was implemented differently.

E.g., if results were different, were the x and y random
numbers displacing the darts within the square really
uniform and independent, and seeded with constants to ensure
bit-for-bit equivalent computations?

How fast the computations happened is not relevant,
though of course nice for getting work done :)

> I tried hard to dispel that belief: you do not have to trade one for the 
> other.
> 
> Yes, in some cases scientific software might lack the engineering work
> that allows for portable performance; but in those cases, there’s
> ‘--tune’.
> 
>   
> https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/
> 
> We should keep repeating that message: reproducibility and performance
> are not antithetic.  And I really mean it, otherwise fellow HPC
> practitioners will keep producing unverifiable results on the grounds
> that they cannot possibly compromise on performance!
>

Maybe the above pi computation could be a start on some kind
of abstract model validation test? It's simple, but it pulls
on a lot of simulation tool chains. WDYT?

> Thanks,
> Ludo’.
> 

-- 
Regards,
Bengt Richter

Re: Assisting reviewing & committing with tags?

[Bengt Richter]

Hi guix,

It sounds like a good idea, but ISTM we don't need yet another markup syntax
if emacs org mode already defines a useful standard that can be adopted.

The advantage to org mode style [0] -- in commit commentary as well as tags
would be its scrapability -- i.e., ease of writing an extractor/formatter
for handy report snippets/pages and web stuff etc., whether implemented
using foreign shell or within guix.

[0] http://xahlee.info/emacs/emacs/emacs_org_markup.html

On +2022-02-02 08:58:18 -0500, Maxim Cournoyer wrote:
> Hi,
> 
> Leo Famulari  writes:
> 
> > On Sun, Jan 09, 2022 at 11:54:25AM +0100, Maxime Devos wrote:
> >> Hi guix reviewers and committers,
> >> 
> >> WDYT of tagging reviewed patches that look good with a usertag,
> >> e.g. 'reviewed-looks-good':
> >> 
> >> https://debbugs.gnu.org/cgi/pkgreport.cgi?tag=reviewed-looks-good=guix
> >> 
> >> then if a committer doesn't have much time to review and hence doesn't
> >> subscribe to guix-patches@, but they do trust the reviewer, they can visit
> >> that URL to look for reviewed patches that can be applied.
> >> 
> >> There could also be a tag 'reviewed-looks-good2' if the patch appears ok
> >> to two reviewers, or a 'reviewed-needs-work', etc.
> >
> > This is a great idea. I guess we will need to adjust the software that
> > runs issues.guix.gnu.org to make use of it, but in the meantime you
> > should keep using this tag. Thanks!
> 
> I like it as well.
> 
> Maxim
> 

-- 
Regards,
Bengt Richter

Re: Dropping gzip-compressed substitutes

[Mathieu Othacehe]

Hey Ludo,

> As discussed on IRC, I’m skeptical about this because:
>
>   1. It requires the development and testing of a custom tool that’s
>  easy to get wrong—e.g., it removes a gzipped nar for something that
>  had nothing but gzip available, etc.
>
>   2. That code would have to run with privileges that give it access to
>  the signing key on berlin.
>
>   3. Those 6.5 TB are an initial constant factor; growth of the storage
>  requirements going forward probably matters more and
>   will give us more flexibility
>  on that.

While those are valid points, we need to keep in mind that it is
important that we manage to move the store to the new SSD array quite
quickly to start GCing again.

If we cannot manage to remove those gzip nars then, I see only two
alternatives:

* Host the nars on the HDD array only.

* Host the nars elsewhere, on a VPS as you are proposing.

> I like Chris Baines’ idea of decoupling nar distribution from nar
> building.  If we want to keep nars long enough so that ‘time-machine’ is
> usable, then storage requirements will keep growing.
>
> Perhaps that means we can regularly copy nars “elsewhere” for long-term
> storage, using nar-herder, rsync, or whatever.  The machine that stores
> nars long-term has low requirements compared to the build farm because
> we don’t need to trust it for anything other than storage.  If that
> makes things easier (and financially viable), a VPS is good enough.

Sure, the VPS would also allow us to have a less European-centric
hosting. I did not follow closely the development of the
nar-herder. Chris what improvements this tool would bring compared to a
rsync based approach?

Thanks,

Mathieu

Re: Missing dependency for emacs-magit

[Zelphir Kaltstahl]

Hello Liliana,

On 2/14/22 20:33, Liliana Marie Prikler wrote:

Hi Zelphir,

Am Montag, dem 14.02.2022 um 18:41 + schrieb Zelphir Kaltstahl:

Hi Michael!
What I mean by that is the following in `init.el`:

(setq guix-package-enable-at-startup t)
(require 'guix-init nil t)

I have a separate profile, which I created for Emacs and Emacs packages
from Guix:

$ guix package --manifest="${GUIX_EXTRA_PROFILES}"/emacs-test-profile-
manifest.scm --profile="${GUIX_EXTRA_PROFILES}"/emacs-test-profile
$ GUIX_PROFILE="${GUIX_EXTRA_PROFILES}/emacs-test-profile"; source
"${GUIX_PROFILE}/etc/profile"
$ emacs

Where the `manifest.scm` contains:

[...]

The channel is currently:

[...]

When I run `emacs` with that profile, I get the error I described
earlier.

I think we need some multi-level printf debugging here.

First, after  `source "${GUIX_PROFILE}/etc/profile"`, is EMACSLOADPATH
correctly pointing to the share/emacs/site-lisp of
${GUIX_EXTRA_PROFILES}/emacs-test-profile?

Second, before and after executing your init.el snippet, what are the
contents of load-path?
Do they contain /gnu/store/SOME_LONG_HASH-magit-MAGIT_VERSION?

Cheers


To simplify things, I created a shell script, which does the calls:


#!/usr/bin/env bash

set -Eeuxo pipefail

# source the environment
GUIX_PROFILE="${GUIX_EXTRA_PROFILES}/emacs-test-profile"
. "${GUIX_PROFILE}/etc/profile"
printf "emacs load path: %s\n" "${EMACSLOADPATH}"

# run emacs
env XMODIFIERS='' emacs


This outputs the following:


+ GUIX_PROFILE=/home/user/.guix-extra-profiles/emacs-test-profile
+ . /home/user/.guix-extra-profiles/emacs-test-profile/etc/profile
++ export 
PATH=/home/user/.guix-extra-profiles/emacs-test-profile/bin:/home/user/.cargo/bin:/home/hans/.config/guix/current/bin:/home/user/.config/guix/current/bin:/home/user/.config/guix/current/sbin:/home/user/.config/guix/current/bin:/home/user/.local/bin:/home/user/.guix-profile/bin:/home/user/.config/guix/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/user/.local/bin
++ 
PATH=/home/user/.guix-extra-profiles/emacs-test-profile/bin:/home/user/.cargo/bin:/home/hans/.config/guix/current/bin:/home/user/.config/guix/current/bin:/home/user/.config/guix/current/sbin:/home/user/.config/guix/current/bin:/home/user/.local/bin:/home/user/.guix-profile/bin:/home/user/.config/guix/current/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/user/.local/bin
++ export 
INFOPATH=/home/user/.guix-extra-profiles/emacs-test-profile/share/info:/home/user/.config/guix/current/share/info:/home/user/.guix-profile/share/info:/home/user/.config/guix/current/share/info:
++ 
INFOPATH=/home/user/.guix-extra-profiles/emacs-test-profile/share/info:/home/user/.config/guix/current/share/info:/home/user/.guix-profile/share/info:/home/user/.config/guix/current/share/info:
++ export 
EMACSLOADPATH=/home/user/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp:/home/user/.guix-profile/share/emacs/site-lisp
++ 
EMACSLOADPATH=/home/user/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp:/home/user/.guix-profile/share/emacs/site-lisp
++ export 
GUILE_LOAD_COMPILED_PATH=/home/user/.guix-extra-profiles/emacs-test-profile/lib/guile/3.0/site-ccache:/home/user/.guix-extra-profiles/emacs-test-profile/share/guile/site/3.0:/home/user/.guix-profile/lib/guile/3.0/site-ccache:/home/user/.guix-profile/share/guile/site/3.0
++ 
GUILE_LOAD_COMPILED_PATH=/home/user/.guix-extra-profiles/emacs-test-profile/lib/guile/3.0/site-ccache:/home/user/.guix-extra-profiles/emacs-test-profile/share/guile/site/3.0:/home/user/.guix-profile/lib/guile/3.0/site-ccache:/home/user/.guix-profile/share/guile/site/3.0
++ export 
GUILE_LOAD_PATH=/home/user/.guix-extra-profiles/emacs-test-profile/share/guile/site/3.0:/home/user/.guix-profile/share/guile/site/3.0
++ 
GUILE_LOAD_PATH=/home/user/.guix-extra-profiles/emacs-test-profile/share/guile/site/3.0:/home/user/.guix-profile/share/guile/site/3.0
+ printf 'emacs load path: %s\n' 
/home/user/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp:/home/user/.guix-profile/share/emacs/site-lisp
emacs load path: 
/home/user/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp:/home/user/.guix-profile/share/emacs/site-lisp
+ env XMODIFIERS= emacs


So the $EMACSLOADPATH before starting Emacs is:


/home/user/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp:/home/user/.guix-profile/share/emacs/site-lisp


About load-path inside Emacs: I think I can get the value before loading 
`init.el` with `emacs -Q` (?):


("~/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp/"
 "/home/user/.emacs.d/elpa/transient-0.3.7"
 "/home/user/.guix-extra-profiles/emacs-test-profile/share/emacs/site-lisp"
 
"/gnu/store/7nd7zr9f3wzkvkcgvc821j7jmnd3x67l-emacs-auto-complete-1.5.1/share/emacs/site-lisp/auto-complete-1.5.1"

Re: better error messages through assertions

[Maxime Devos]

Ricardo Wurmus schreef op ma 14-02-2022 om 23:32 [+0100]:
> I think we should add simple type checks, something like this:
> [...]
> 
> What do you think?  There are many different ways of implementing this
> (a new variant of DEFINE that also accepts a type declaration, an assert
> like above, a fancier assert that composes a helpful error message by
> itself, a separate type declaration that is looked up only when the
> corresponding procedure is called in a certain context, etc), but I’d
> first like to know if there is consensus that we want something like
> this.

Seems nice.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part