from:"Christopher Baines"


Jack Hill  writes:

> On Mon, 20 Dec 2021, zimoun wrote:
>
>> Hi,
>>
>> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès  wrote:
>>
>>> Regarding nar-herder, I think it’d be nice to have a solution to
>>> mirroring in Guix proper, developed similarly to other components,
>>> because it could be a fairly central tool.
>>>
>>> ‘guix publish’ is probably not extensible enough to support that, but we
>>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>>
>> Why not extend “guix archive”?
>
> I'm quite interested in learning more and potentially trying out the
> nar-herder! Some thoughts that I'd like to add to the design space:

Apologies for the slow reply, it's great that you're interested!

> I think it would be great if one of the pastures to which we herd the
> nars would be a free and open source software mirror site. In my
> experience, these are usually some static web hosting in front of a
> large disk with a place to run scripts to sync the content. A database
> server may not be available. I'd like to support this use case because
> I think it is a great way to build bridges to the communities who run
> or gather around these mirrors.

I think there's a general discrepancy between how Guix works and how
mirror sites generally work, but there are probably ways of bridging
that gap. Maybe all the nars for the latest release could be mirrored
for example, and the nar-herder could probably help with that.

> I'd also like the ability fetch nars directly from the local-to-me
> mirror rather than having them be proxied through a far way server.

I think setting up some mirrors closer to the people that use Guix is
now easier to do with the help of the nar-herder.

> One of the things that I really like and find empowering about Guix is
> that the developer/system administration tools are as available, easy
> to use, and convenient as the every day tooling. To the extent
> possible, I think that we should strive to make our syncing/mirroring
> solution practical to run for local, small setups, and not require
> project-scale infrastructure or coordination between many programs
> that are not captured in a Guix service.

Indeed, and this is something to strive for in the design.


signature.asc
Description: PGP signature

Re: extend ’guix archive’?

zimoun  writes:

> On Mon, 20 Dec 2021 at 23:07, Ludovic Courtès  wrote:
>
>> Regarding nar-herder, I think it’d be nice to have a solution to
>> mirroring in Guix proper, developed similarly to other components,
>> because it could be a fairly central tool.
>>
>> ‘guix publish’ is probably not extensible enough to support that, but we
>> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.
>
> Why not extend “guix archive”?
>
> However, without all the details so my remark is totally naive, I miss
> what “nar-herder” is doing that “guix archive”+rsync+“guix publish” is
> not doing – other said I miss why another SQL database is required to
> serve stuff from one place to another.  I have read README but I did not
> get the point.

Apologies, I missed replying to this at the time.

Using an SQL database (sqlite) isn't required in my opinion, but I do
like that aspect of the design.

You could for example keep the narinfo's as files on the disk, and then
use rsync say to copy them between machines to setup mirrors. I think
this would perform worse compared to storing the narinfos in a database
when initially setting up a mirror. With a database, you only have to
download one large file, whereas with individual narinfo files, you have
to download lots of individual small files. Storing all the narinfo
files individually also generally takes up more space than storing them
in a database, since there's an overhead associated with each file on
the filesystem.

Additionally, the database is used to do extra things on top of just
storing the narinfos. The references from the narinfos are stored
separately to facilitate doing GC like removal of the
nars. Additionally, there's a way to tag nars, something which I was
thinking of using to allow selectively removing some nars which only
need to be stored for a short time.

https://git.cbaines.net/guix/nar-herder/tree/nar-herder/database.scm#n74

I hope that makes some sense,

Chris

signature.asc
Description: PGP signature

nar-herder design retrospective

Hi!

I rushed the nar-herder [1] in to existence back in December, to address
the buildup of nars on bayfront by moving the nars to another machine
with more space to store them.

1: https://git.cbaines.net/guix/nar-herder/about/

This was something I was planning for a while though, I sent an email to
guix-devel outlining some points of the design and aims around a year
ago [2].

2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

The nar-herder is currently in a stable state. It's packaged for guix
and there's a service to use it. There are probably some bugs, but I
think I've fixed the important initial ones.

I covered some information about the deployment of the nar-herder in
this email back in December [3] and that led to some really good
replies, I'll copy/paste some of the important bits below.

3: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html

https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00196.html:

> Regarding nar-herder, I think it’d be nice to have a solution to
> mirroring in Guix proper, developed similarly to other components,
> because it could be a fairly central tool.

> Usually I’m the one asking for blog posts :-), but I’d really like us as
> a project to collectively engage on the topic before we publicize this
> specific approach.

https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00201.html:

> Why not extend “guix archive”?
> 
> However, without all the details so my remark is totally naive, I miss
> what “nar-herder” is doing that “guix archive”+rsync+“guix publish” is
> not doing – other said I miss why another SQL database is required to
> serve stuff from one place to another.  I have read README but I did not
> get the point.

https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00204.html:

> I'm quite interested in learning more and potentially trying out the
> nar-herder! Some thoughts that I'd like to add to the design space:
> 
> I think it would be great if one of the pastures to which we herd the
> nars would be a free and open source software mirror site. In my
> experience, these are usually some static web hosting in front of a
> large disk with a place to run scripts to sync the content. A database
> server may not be available. I'd like to support this use case because I
> think it is a great way to build bridges to the communities who run or
> gather around these mirrors.
> 
> I'd also like the ability fetch nars directly from the local-to-me
> mirror rather than having them be proxied through a far way server.
> 
> One of the things that I really like and find empowering about Guix is
> that the developer/system administration tools are as available, easy
> to use, and convenient as the every day tooling. To the extent
> possible, I think that we should strive to make our syncing/mirroring
> solution practical to run for local, small setups, and not require
> project-scale infrastructure or coordination between many programs
> that are not captured in a Guix service.

So, currently the nar-herder can be used to move nars between machines,
which is what I wanted the initial implementation to be capable of. This
functionality should be sufficient for operating mirrors, although I'm
not aware of any being setup yet. I'd also like to be able to get
metrics about nar requests, but this isn't supported yet.

I'm all for having a solution to mirroring in Guix itself, although I
don't have a plan for this. Maybe the nar-herder could just be moved in
to Guix, maybe with a different name? Any other ideas?

I think moving the nar-herder repository on to Savannah is probably a
good thing to do regardless.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Guix Data Service client module


Ludovic Courtès  writes:

> Hello Guix!
>
> Here’s a client module for the Guix Data Service, allowing you to access
> a subset of the Guix Data Service interfaces from the comfort of your
> REPL.
>
> I had it sitting in my source tree for a while and Chris sent me an
> impressive shell one-liner that made me want to try from Scheme:
>
>   wget
> "https://data.guix-patches.cbaines.net/revision/47f85c53d954f857b45cebefee27ec512d917484/lint-warnings.json?locale=en_US.UTF-8=input-labels=linter=message=location;
> -O - | jq -r '.lint_warnings | .[] | .package.name' | sort | uniq | wc
> -l
>
> Turns out we can do the same in two long lines of Scheme!
>
> scheme@(guix data-service)> (define s (open-data-service 
> "https://data.guix-patches.cbaines.net;))
> scheme@(guix data-service)> (length (delete-duplicates (map 
> lint-warning-package (revision-lint-warnings s 
> "47f85c53d954f857b45cebefee27ec512d917484" "input-labels"
> $6 = 3560
>
>
> (That counts the number of packages at that revision that have one or
> more warnings from the new ‘input-labels’ lint checker.)
>
> We can do other things, such as browsing package versions:
>
> scheme@(guix data-service)> (define s (open-data-service 
> "https://data.guix.gnu.org;))
> scheme@(guix data-service)> (package-version-branches (car (package-versions 
> (lookup-package s "emacs"
> $9 = (#< name: "master" repository-id: 1>)
> scheme@(guix data-service)> (package-version-history s (car $9) "emacs")
> $10 = (#< version: "27.2" first-revision: #< 
> commit: "cc33f50d0e2a7835e99913226cb4c4b0e9e961ae" date: # second: 54 minute: 30 hour: 20 day: 25 month: 3 year: 2021 zone-offset: 0>> 
> last-revision: #< commit: 
> "364b56124b88398c199aacbfd4fdfc9a1583e634" date: # 31 minute: 16 hour: 13 day: 27 month: 6 year: 2021 zone-offset: 0>>> 
> #< version: "27.1" first-revision: #< 
> commit: "36a09d185343375a5cba370431870f9c4435d623" date: # second: 52 minute: 16 hour: 4 day: 28 month: 8 year: 2020 zone-offset: 0>> 
> last-revision: #< commit: 
> "ac29d37e2ffd7a85adfcac9be4d5bce018289bec" date: # 2 minute: 36 hour: 17 day: 25 month: 3 year: 2021 zone-offset: 0>>> 
> #< version: "26.3" first-revision: #< 
> commit: "43412ab967ee00789fe933f916d804aed9961c57" date: # second: 29 minute: 36 hour: 3 day: 30 month: 8 year: 2019 zone-offset: 0>> 
> last-revision: #< commit: 
> "bf19d5e4b26a2e38fe93a45f9341e14476ea5f82" date: # 19 minute: 50 hour: 21 day: 27 month: 8 year: 2020 zone-offset: 0>>> 
> #< version: "26.2" first-revision: #< 
> commit: "5069baedb8a902c3b1ea9656c11471658a1de56b" date: # second: 8 minute: 46 hour: 22 day: 12 month: 4 year: 2019 zone-offset: 0>> 
> last-revision: #< commit: 
> "02c61278f1327d403f072f42e6b92a1dc62fc93a" date: # 35 minute: 44 hour: 0 day: 30 month: 8 year: 2019 zone-offset: 0>>> 
> #< version: "26.1" first-revision: #< 
> commit: "897f303d2fa61497a931cf5fcb43349eb5f44c14" date: # second: 47 minute: 31 hour: 7 day: 1 month: 1 year: 2019 zone-offset: 0>> 
> last-revision: #< commit: 
> "ee6c4b62b88640f3828cf73a30377124e16cb95f" date: # 51 minute: 8 hour: 20 day: 12 month: 4 year: 2019 zone-offset: 0>>>)
>
> Now all we need to do is plug it into the right tools and enjoy!

Thanks for writing this Ludo, sorry it's taken so long for me to have a
look.

I've had a little play around with it locally, and it seems to work
well.

I added some exports (included below) so that I could more easily use
the module.

Maybe open-data-service could have the url default to
"https://data.guix.gnu.org;.

The only thing I can see that's required before merging though is the
exports. I'm now thinking about this kind of thing (getting data out of
the data service) in the context of patch/branch review.

Thanks,

Chris



  #:export (repository?
repository-id
repository-label
repository-url
repository-branches

branch?
branch-name
branch-repository-id

package-version?
package-version-string
package-version-branches

package?
package-name
package-versions

revision?
revision-commit
revision-date

build?
build-server-id
build-id
build-time

channel-instance?
channel-instance-system
channel-instance-derivation
channel-instance-builds

lint-warning?
lint-warning-package
lint-warning-package-version
lint-warning-message
lint-warning-location

open-data-service

lookup-package
lookup-repository
package-version-history
revision-channel-instances
revision-lint-warnings))


signature.asc
Description: PGP signature

Re: Having a package browser on guix.gnu.org

2022-01-09 Thread Christopher Baines


Ludovic Courtès  writes:

> Luis Felipe  skribis:
>
>> But I wonder if it is possible now to make the packages part use any of the 
>> Postgres databases that already exist and allow traditional search without 
>> JavaScript...
>
> Former Outreachy intern Danjela Lura, together with Chris Baines, had
> started developing a JS-free package browsing interface:
>
>   https://lists.gnu.org/archive/html/guix-devel/2020-07/msg00050.html
>
> Chris, could you tell us what the status is and what’s missing before we
> can use it on the web site?

I think it's unchanged for a while now, I've attached what's probably
the latest source code (there's not much to it).

I think the remaining work is to settle on a direction in terms of the
design and how to integrate it in to the website, and then deploy
something for real. Personally, I'm in favour of a packages.guix.gnu.org
domain which hosts the search page plus the package pages, and stopping
generating package pages with haunt.

I don't really have the time to try and move this forward myself, but I
can try and support others.

#!/usr/local/bin/guile -s
!#
(use-modules (web server)
 (web request)
 (web response)
 (web uri)
 (sxml simple)
 (web client)
 (rnrs bytevectors)
 (srfi srfi-11)
 (srfi srfi-1)
 (ice-9 match)
 (json)
 (texinfo)
 (texinfo plain-text)
 (apps aux strings)
 (apps base templates theme)
 (apps base utils)
 (apps base types)
 (apps base templates components))

(define (templatize title body)
  `(html (head (title ,title))
 (body ,@body)))

(define* (respond #:optional body #:key
  (status 200)
  (title "Packages")
  (doctype "\n")
  (content-type-params '((charset . "utf-8")))
  (content-type 'text/html)
  (extra-headers '())
  (sxml (and body (templatize title body
  (values (build-response
   #:code status
   #:headers `((content-type
. (,content-type ,@content-type-params))
   ,@extra-headers))
  (lambda (port)
(if sxml
(begin
  (if doctype (display doctype port))
  (sxml->xml sxml port))

(define (search-packages-page request body)
  (define uri-value
(let ((uri (request-uri request)))
  (if (eqv? #f (uri-query uri))
  ""
  (second
   (string-split
(uri-query
 uri)
#\=)

  (define response
(let-values
(((response-object body)
  (http-request
   (string-append

"http://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/packages.json?locale=en_US.utf8_query=;
uri-value 
"=version=synopsis=description_name=_results=30") 
#:method 'GET)))
  (json-string->scm
   (utf8->string body

  (respond
`((link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/package.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/item-preview.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/page.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/elements.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/common.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/messages.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/navbar.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/breadcrumbs.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/buttons.css;)))
  (link (@ (rel "stylesheet") (href 
"http://guix-website-test.cbaines.net/static/base/css/footer.css;)))
  (link (@ (rel "stylesheet") (href 
"https://stackpath.bootstrapcdn.com/bootstrap/3.4.1/css/bootstrap.min.css;)
   (integrity 
"sha384-HSMxcRTRxnN+Bdg0JdbxYKrThecOKuH5zCYotlSAcp1+c8xmyTe9GYg1l9a69psu")
   (crossorigin "anonymous")))

  ,(navbar #:active-item "packages/search")

  (div (@ (class "package-info"))
   (div (@ (class "search-container")
   (style "display: block; text-align: center;"))
(h1 "Packages")
(form (@ (style "display: inline-block; margin-right auto; 
text-align: left"))
  (input (@ (type "text")
(placeholder "Search packages")
(name "search")))

Re: Mid-December update on bordeaux.guix.gnu.org

2022-01-06 Thread Christopher Baines


Ludovic Courtès  writes:

>> However, due to the time spent not building things, the backlog is
>> longer than usual, and the substitute availability (especially for
>> x86_64-linux and i686-linux) is lower than usual.
>
> Yeah, ‘guix weather coreutils’ finds nothing on bordeaux.guix right now.

I've been so slow in replying to this email that the situation has
actually improved now, substitute availability for x86_64-linux is
around ~70% and still slowly rising.

>> ** Space issues and the nar-herder
>>
>> bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
>> bayfront was getting a little scarce. This week I started rolling out
>> the nar-herder [2], a utility I've been thinking about for a while. This
>> has enabled moving nars off of bayfront on to another machine which I've
>> confusingly named lakefront.
>
> Woow, neat!
>
> Regarding nar-herder, I think it’d be nice to have a solution to
> mirroring in Guix proper, developed similarly to other components,
> because it could be a fairly central tool.
>
> ‘guix publish’ is probably not extensible enough to support that, but we
> could make it a new ‘guix mirror’ or ‘guix sync’ or whatever command.

I think having something in Guix would be good too.

I still view the nar-herder as a bit wierd in terms of the collection of
concerns, mixing mirroring in with moving nars around between machines
and I'm hoping to add some metrics functionality soon as well.

>> The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
>> storage across 2 hard drives. When a nar is requested from bayfront, it
>> will check it's local storage, and serve it from there if it exists,
>> otherwise it will forward the request to lakefront. There might be a
>> slight drop in the speed you can download nars, but apart from that this
>> change shouldn't be visible.
>
> Excellent, thanks for acting this promptly as problems crop up!
>
> I think we should make sure this is funded by the project and that
> credentials are shared.  As discussed before, I think it’s best to keep
> things tidy in that respect, in the spirit of building and maintaining
> this collectively.

That would be good, I can start a thread on guix-sysadmin about this.

>> Going forward, it would be good to have an additional full backup of the
>> nars that bayfront can serve things from, to provide more
>> redundancy. I'm hoping the nar-herder will also enable setting up
>> geographically distributed mirrors, which will hopefully improve
>> redundancy further, and maybe performance of fetching nars too.
>>
>> Once I've had a chance to neaten up the code a little, I'll get a Guix
>> package and service written, plus I'll draft a Guix blog post about the
>> nar-herder.
>
> Usually I’m the one asking for blog posts :-), but I’d really like us as
> a project to collectively engage on the topic before we publicize this
> specific approach.

Sure, I also still need to post patches for the Guix service, which I'll
try to do soon.

>> That means there's the following currently running build agents (by
>> architecture):
>>
>>  - x86_64-linux + i686-linux (3 machines):
>>- 4 core Intel NUC (giedi)
>>- Max 16 cores for 1 concurrent build on bayfront
>>- 32 cores on milano-guix-1 (slow storage though)
>>  - aarch64-linux + armhf-linux (2 machines)
>>- 16 core HoneyComb LX2 (hatysa)
>>- 4 core Overdrive (monokuma)
>>  - powerpc64le-linux (1 machine)
>>- 64 core machine (polaris)
>
> This is looking pretty nice!  I’m also all for streamlining machine
> handling, both administratively (in maintenance.git) and financially.

I think there was some discussion previously about guix-europe buying
the HoneyComb board, which I can probably restart. It would be good to
also sort out better hosting for the powerpc64le-linux machine.

I'll try to put some time in to organising things in maintenance.git.

>> Ironically, I think that the most under-resourced area is x86_64-linux +
>> i686-linux. bayfront is restricted in what it can do since it also runs
>> the coordinator, and things go badly if the machine gets
>> overloaded. bayfront and milano-guix-1 both have hard drive storage,
>> which also can slow them down when building things (especially
>> milano-guix-1).
>>
>> If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
>> keep up, it would be good to make a plan to add capacity. I think even a
>> single high core count x86_64-linux machine with performant storage
>> would make a big difference.
>
> Yes, we should do that, we still have funds for it.

Great :)

>> ** IPv6 not supported (yet)
>>
>> I was slow to notice, but bordeaux.guix.gnu.org isn't available over
>> IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
>> to address this, but I haven't worked out quite how to yet.
>
> Should be almost done now, as discussed on IRC today.  \o/

Indeed, I think this is working now.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Solstice infrastructure hackathon

2021-12-16 Thread Christopher Baines

Ludovic Courtès  writes:

> We were unlucky enough that it happened days after the other build farm,
> bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped,
> right before the big merge—so it doesn’t have substitutes for current
> master.

Builds effectively stopped on the 29th of November, which is more than a
few days I'd say, although this is maybe not the biggest issue. Since
the build coordinator instance behind bordeaux.guix.gnu.org wasn't
building things from core-updates-frozen prior to the merge, even if
builds hadn't stopped due to the space issues on bayfront, it still
wouldn't have had many substitutes.

As part of testing patches and branches [1], I think it would be good to
get builds for things like core-updates-frozen happening, that will
hopefully improve the substitute availability from bordeaux.guix.gnu.org
on average.

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg1.html

>   • Add DNS redundancy for guix.gnu.org so it can point to one of two
> hosts (need to figure out certbot challenges so both machines can
> update their certificates).

This (in general) is something I'm interested in working out, since
it'll be helpful for setting up mirrors for substitutes as well (in the
case where you want the mirrors to respond to one common DNS name with
working TLS).

>   • Come up with a plan to add disks to the RAID array on bayfront, the
> head node of bordeaux.guix.gnu.org.

The space issue on bayfront that led to builds not happening has now
been effectively resolved (see [2]). There's definitely lots of tidying
up to do, but I think the situation for storing the nars is much better
now.

2: https://lists.gnu.org/archive/html/guix-devel/2021-12/msg00140.html

That's not to say there's not something to be gained by upgrading the
bayfront hardware, some SSD storage would be ideal to speed up the
coordinator and builds.

>   • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
> plain rsync or .

I'm interested in getting bordeaux.guix.gnu.org in to a state where
there's less of a discrepancy in performance depending on where in the
world it's accessed from. I'm assuming there is some difference in the
performance, which is definitely an assumption to check, which is one
part of the problem. If it turns out there are some gains to be had, the
next step is investigating how this could be approached. Mirrors plus
GeoIP based DNS is the approach I currently have in mind.

Anyway, even if there isn't a meaningful performance difference, maybe
it's worth setting up distributed mirrors for reliability.

>   • Have a documented procedure to set up substitute mirrors, such as
> the one in .cn (I can’t find the URL), ideally with plain rsync
> access.

Getting the nar-herder in to a state where other people might be able to
use it is definitely on my list of things to do. I'm assuming here that
it's something that people might want to use, and again that's probably
worth investigating. If it turns out that people just want to use rsync,
it's probably worth assisting with getting that kind of setup working.

> Who’s in?  :-)

Not sure how much time I'll have, but I'll try to be around :)

Thanks,

Chris

signature.asc
Description: PGP signature

Re: Mid-December update on bordeaux.guix.gnu.org

2021-12-16 Thread Christopher Baines


zimoun  writes:

> Hi Chris,
>
> On Thu, 16 Dec 2021 at 00:20, Christopher Baines  wrote:
>> zimoun  writes:
>
>>> Do you think that Bordeaux could run
>>>
>>><https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>
>>
>> The Guix Build Coordinator just builds derivations. I haven't got it to
>> build a manifest before, but that's possible I guess.
>
> I am not sure to understand since Cuirass also builds derivations and
> the purpose of this source-manifest.scm is to let Cuirass ingest all
> sources,
>
> <https://ci.guix.gnu.org/jobset/source>
>
>> I think it's unnecessary though, since I believe derivations for all
>> origins of all packages are already being built, that happens through
>> just asking the coordinator to build derivations for all packages, you
>> don't need to specify "source" derivations separately.
>
> Your assumption is wrong, IMHO.  We have many failed examples and at the
> end Ludo wrote this source-manifest.scm to be sure that all is ingested
> for sure.

What assumption?

I believe the reason the source-manifest.scm thing is useful to use with
Cuirass is that it has something to do with Cuirass copying the outputs
from those builds back to where they're served from, and maybe
registering GC roots as well. I could be wrong though.

I'm saying that this additional thing is unnecessary when using the Guix
Build Coordinator to build packages, since at least with the
configuration for bordeaux.guix.gnu.org, it'll build the sources, store
and serve them without any extra effort.

>>> ?  Having a redundancy about all origins would avoid breakage.  For
>>> instance, because Berlin was down yesterday morning, “guix pull” was
>>> broken because the missing ’datefuge’ package – disappeared upstream.
>>
>> I would hope that bordeaux.guix.gnu.org has a substitute for that, could
>> you check the derivation against data.guix.gnu.org, and see if there's a
>> build? Use a URL like:
>>
>>   
>> https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv
>>
>> There is one issue though, bordeaux.guix.gnu.org doesn't provide content
>> addressed files in the same way guix publish does. I hope to add that
>> through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
>> hopefully be added to the list of content addressed mirrors:
>>
>>   https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368
>>
>> That would mean that the bytes for a tar archive for example would be
>> available by the sha256 hash, not just as a nar. I'm not sure to what
>> extent this would help, but it's probably useful.
>
> Thanks for explaining the details.  From a pragmatical point of view as
> end-user, “guix pull” must Just Work whatever the plumbing.
>
> For instance, some of us spent energy to “evangelize” in scientific
> communities how Guix is awesome.  Next morning, people give a look and
> bang!  Because a tiny and short outage.  Obviously, “shit happens”(*)
> and it is really really really sparse but too late the damage is there.
>
> My feeling here is that both build farms work independently instead of
> coordinate the usage of resources and exploit this strength to have two.

I too want to coordinate, although I think that having two independent
build farms is actually good for reliability.

>>> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
>>> solves the issue but from my understanding, it will improve at least
>>> performance delivery.  Well, it appears to me worth to give a try.
>>>
>>> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
>>> 3: 
>>> <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
>>> 4: <https://yhetil.org/guix/87tvju6145@gnu.org/>
>>> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
>>> 6: <https://yhetil.org/guix/87d0my1380@gmail.com/>
>>
>> Effectively this is moving towards building a CDN. With the nar-herder,
>> you could deploy reverse proxies (or edge nodes) in various
>> locations. Then the issue just becomes how to have users use the ones
>> that are best for them. This might require doing some fancy stuff with
>> GeoIP based DNS, and somehow sharing TLS certificates between the
>> machines, but I think it's quite feasible.
>
> Considering the human resource vs the money resource, it appears to me
> better to invest the human energy into things that do not

Re: Mid-December update on bordeaux.guix.gnu.org

2021-12-15 Thread Christopher Baines

zimoun  writes:

> Hi Chris,
>
> Thanks for the update.  And for the all work. :-)
>
>
> On Wed, 15 Dec 2021 at 16:48, Christopher Baines  wrote:
>
>> In summary, the space issue I mentioned in the previous update has
>> effectively been addressed. All the paused agents are now unpaused and
>> builds are happening again.
>
> The timing had almost been perfect. ;-)
>
>
> Well, as discussed on Sept., one concern I have is about “long-term
> storage” – where long-term is not well-defined and storage either.
>
> Do you think that Bordeaux could run
>
><https://git.savannah.gnu.org/cgit/guix.git/tree/etc/source-manifest.scm>

The Guix Build Coordinator just builds derivations. I haven't got it to
build a manifest before, but that's possible I guess.

I think it's unnecessary though, since I believe derivations for all
origins of all packages are already being built, that happens through
just asking the coordinator to build derivations for all packages, you
don't need to specify "source" derivations separately.

> ?  Having a redundancy about all origins would avoid breakage.  For
> instance, because Berlin was down yesterday morning, “guix pull” was
> broken because the missing ’datefuge’ package – disappeared upstream.

I would hope that bordeaux.guix.gnu.org has a substitute for that, could
you check the derivation against data.guix.gnu.org, and see if there's a
build? Use a URL like:

https://data.guix.gnu.org/gnu/store/vhj3gg00hzqfi8lazr3snb9msr4a3q6l-datefudge_1.23.tar.xz.drv

There is one issue though, bordeaux.guix.gnu.org doesn't provide content
addressed files in the same way guix publish does. I hope to add that
through the nar-herder, and once that's added, bordeaux.guix.gnu.org can
hopefully be added to the list of content addressed mirrors:

  https://git.savannah.gnu.org/cgit/guix.git/tree/guix/download.scm#n368

That would mean that the bytes for a tar archive for example would be
available by the sha256 hash, not just as a nar. I'm not sure to what
extent this would help, but it's probably useful.

>> In general this is an important step in being more flexible where the
>> nars are stored. There's still a reliance on storing pretty much all the
>> nars on a single machine, but which machine has this role is more
>> flexible now. I think this architecture also makes it easier to break
>> the "all nars on a single machine" restriction in the future as well.
>
> IIUC the design, if the proxy server is lost, then it is easy to replace
> it.  Right?

I guess so, the nar-herder helps with managing the data at least which
makes setting up new or replacement servers easier.

> I remember discussions about CDN [2,3,4,5,6].  I do not know if it
> solves the issue but from my understanding, it will improve at least
> performance delivery.  Well, it appears to me worth to give a try.
>
>
> 2: <https://lists.gnu.org/archive/html/guix-devel/2016-03/msg00312.html>
> 3: 
> <https://yhetil.org/guix/KBlEbLsxWu2Kmv5RvS2dHXDleGAyyz9WEA0T6wQ1QArc0mjkol-1W5Vv66D9oauvQ5l6WYTaJ86Ckxjc8YS_2pn2NN1M_L8RJUsIBmmFeqE=@protonmail.ch/>
> 4: <https://yhetil.org/guix/87tvju6145@gnu.org/>
> 5: <https://lists.gnu.org/archive/html/guix-devel/2018-12/msg00192.html>
> 6: <https://yhetil.org/guix/87d0my1380@gmail.com/>

Effectively this is moving towards building a CDN. With the nar-herder,
you could deploy reverse proxies (or edge nodes) in various
locations. Then the issue just becomes how to have users use the ones
that are best for them. This might require doing some fancy stuff with
GeoIP based DNS, and somehow sharing TLS certificates between the
machines, but I think it's quite feasible.

>> Going forward, it would be good to have an additional full backup of the
>> nars that bayfront can serve things from, to provide more
>> redundancy. I'm hoping the nar-herder will also enable setting up
>> geographically distributed mirrors, which will hopefully improve
>> redundancy further, and maybe performance of fetching nars too.
>
> To me, one first general question about backup coordination is to define
> a window for time:
>
>  - source: forever until the complete fallback to SWH is robust;
>  - all the substitutes to run “guix time-machine --commit=<> -- help ”
>for any commit reachable by inferior: forever;
>  - package substitute: rule something.

The idea I've been working with so far is simply to store everything
that's built, forever.

Currently, that amounts to 561,043 nars totaling ~2.5TB's.

How feasible this is depends on a number of factors, but I don't have
any reason to think it's not feasible yet.

> Thanks for taking care about redundancy and reliance of CI.

There's not a relationship to continuous integration yet, although

Mid-December update on bordeaux.guix.gnu.org

2021-12-15 Thread Christopher Baines

Hey!

I sent out the last update 3 weeks ago [1].

1: https://lists.gnu.org/archive/html/guix-devel/2021-11/msg00154.html

In summary, the space issue I mentioned in the previous update has
effectively been addressed. All the paused agents are now unpaused and
builds are happening again.

However, due to the time spent not building things, the backlog is
longer than usual, and the substitute availability (especially for
x86_64-linux and i686-linux) is lower than usual.

I've also noticed that bordeaux.guix.gnu.org doesn't work over IPv6, and
I want to fix that soon.

** Space issues and the nar-herder

bordeaux.guix.gnu.org wasn't building things for 2 weeks as the space on
bayfront was getting a little scarce. This week I started rolling out
the nar-herder [2], a utility I've been thinking about for a while. This
has enabled moving nars off of bayfront on to another machine which I've
confusingly named lakefront.

The lakefront machine is hosted by Hetzner in Germany, and has 6TB of
storage across 2 hard drives. When a nar is requested from bayfront, it
will check it's local storage, and serve it from there if it exists,
otherwise it will forward the request to lakefront. There might be a
slight drop in the speed you can download nars, but apart from that this
change shouldn't be visible.

The nar-herder is now busy deleting nars on bayfront which are available
on lakefront. Once it's got through the backlog, I'll enable NGinx
caching for the nars on bayfront, which should help improve
performance. I've tested downloading the largest nars (~2GB) though, and
it seems to work fine.

In addition to lakefront, I've also added a 6TB hard drive to hatysa,
the HoneyComb LX2 machine that I host. Like lakefront, it's busy
downloading the nars from bayfront. This will act as a backup in case
lakefront is lost.

In general this is an important step in being more flexible where the
nars are stored. There's still a reliance on storing pretty much all the
nars on a single machine, but which machine has this role is more
flexible now. I think this architecture also makes it easier to break
the "all nars on a single machine" restriction in the future as well.

Going forward, it would be good to have an additional full backup of the
nars that bayfront can serve things from, to provide more
redundancy. I'm hoping the nar-herder will also enable setting up
geographically distributed mirrors, which will hopefully improve
redundancy further, and maybe performance of fetching nars too.

Once I've had a chance to neaten up the code a little, I'll get a Guix
package and service written, plus I'll draft a Guix blog post about the
nar-herder.

2: https://git.cbaines.net/guix/nar-herder/about/

** Build machines and backlog

Because of the 2 weeks of not building anything, there's a significant
backlog of builds to get through, and I'm not including the new builds
from the core-updates-frozen merge here.

As for build machines, milano-guix-1 came back online today, which is
great. I believe harbourfront is still unusable through (broken hard
drive).

That means there's the following currently running build agents (by
architecture):

 - x86_64-linux + i686-linux (3 machines):
   - 4 core Intel NUC (giedi)
   - Max 16 cores for 1 concurrent build on bayfront
   - 32 cores on milano-guix-1 (slow storage though)
 - aarch64-linux + armhf-linux (2 machines)
   - 16 core HoneyComb LX2 (hatysa)
   - 4 core Overdrive (monokuma)
 - powerpc64le-linux (1 machine)
   - 64 core machine (polaris)

Ironically, I think that the most under-resourced area is x86_64-linux +
i686-linux. bayfront is restricted in what it can do since it also runs
the coordinator, and things go badly if the machine gets
overloaded. bayfront and milano-guix-1 both have hard drive storage,
which also can slow them down when building things (especially
milano-guix-1).

If we (as a project) want bordeaux.guix.gnu.org to have the capacity to
keep up, it would be good to make a plan to add capacity. I think even a
single high core count x86_64-linux machine with performant storage
would make a big difference.

** IPv6 not supported (yet)

I was slow to notice, but bordeaux.guix.gnu.org isn't available over
IPv6 yet, since bayfront doesn't seem to have IPv6 connectivity. I want
to address this, but I haven't worked out quite how to yet.

Please let me know if you have any comments or questions!

Chris


signature.asc
Description: PGP signature

Re: Update on bordeaux.guix.gnu.org

2021-12-03 Thread Christopher Baines

Ricardo Wurmus  writes:

> Ludovic Courtès  writes:
>
>>> The disk space usage trend is pretty much
>>> linear, so if things continue without any change, I think it will
>>> be
>>> necessary to pause the agents within a month, to avoid filling up
>>> bayfront entirely.
>>
>> Ah, bummer.  I hope we can find a solution one way or another.
>> Certainly we could replicate nars on another machine with more disk,
>> possibly buying the necessary hardware with the project funds.
>
> Remember that I’ve got three 256G SSDs here that I could send to
> wherever bayfront now sits.  With LLVM or a RAID configuration
> these could just be added to the storage pool — if bayfront has
> sufficient slots for three more disks.

While it would be nice for bayfront to have an SSD, it might actually be
more valuable to use those for some of the machines that do more of the
building.

harbourfront currently has a broken hard drive (I believe), and
milano-guix-1 has some slow hard drives that impede it building
things. I've CC'ed Andreas as I think he knows more about harbourfront,
and I'll follow up about milano-guix-1 off list.

signature.asc
Description: PGP signature

Re: Update on bordeaux.guix.gnu.org

2021-12-03 Thread Christopher Baines


Ludovic Courtès  writes:

> Christopher Baines  skribis:
>
>> I've been doing some performance tuning, submitting builds is now more
>> parallelised, a source of slowness when fetching builds has been
>> addressed, and one of the long queries involved in allocating builds has
>> been removed, which also improved handling of the WAL (Sqlite write
>> ahead log).
>>
>> There's also a few new features. Agents can be deactivated which means
>> they won't get any builds allocated. The coordinator now checks the
>> hashes of outputs which are submitted, a safeguard which I added because
>> the coordinator now also supports resuming the uploads of outputs. This
>> is particularly important when trying to upload large (> 1GiB) outputs
>> over slow connections.
>>
>> I also added a new x86_64 build machine. It's a 4 core Intel NUC that I
>> had sitting around, but I cleaned it up and got it building things. This
>> was particularly useful as I was able to use it to retry building
>> guile@3.0.7, which is extremely hard to build [2]. This was blocking
>> building the channel instance derivations for x86_64-linux.
>>
>> 2: 
>> https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-guile-3.0.7.drv
>
> Neat!  (Though I wouldn’t say building Guile is “extremely hard”,
> especially on x86_64.  :-))  The ability to keep retrying is much
> welcome.

To rephrase, I found it extremely hard to get that particular Guile
derivation to build successfully, it failed to build 12 times, and only
succeeded when I added new hardware to attempt on (I'm guessing the
particular issue I was encountering was exacerbated by more cores).

Unfortunately, I also think that you finding it easy to build actually
contributes to the problem here, since it makes finding and addressing
issues like this harder.

>> Space is running out on bayfront, the machine that runs the coordinator,
>> stores all the nars and build logs, and serves the substitutes. I knew
>> this was probably going to be an issue, bayfront didn't have much space
>> to begin with, but I had hoped I'd be further forward in developing some
>> way to allow moving the nars around between multiple machines, to remove
>> the need to store all of them on bayfront. I have got a plan, there's
>> some ideas I mentioned back in February [4], but I haven't got around to
>> implementing anything yet. The disk space usage trend is pretty much
>> linear, so if things continue without any change, I think it will be
>> necessary to pause the agents within a month, to avoid filling up
>> bayfront entirely.
>
> Ah, bummer.  I hope we can find a solution one way or another.
> Certainly we could replicate nars on another machine with more disk,
> possibly buying the necessary hardware with the project funds.

Since this email got a bit delayed when I sent it, things have moved on
a bit now.

90% disk usage was the threshold I had in mind for bayfront, and that's
now pretty much been reached so I've paused all the agents. My plans for
how to address this have also developed a bit as well, but it's still
going to take a month at least to get things going again.

Chris


signature.asc
Description: PGP signature

Update on bordeaux.guix.gnu.org

2021-11-24 Thread Christopher Baines

Hey!

It's been 3 months since I sent the last update [1]. This email was
meant to go out on Friday, but it seems opensmtpd was broken on my
machine, so it got stuck.

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg00075.html

First, some good things:

I've been doing some performance tuning, submitting builds is now more
parallelised, a source of slowness when fetching builds has been
addressed, and one of the long queries involved in allocating builds has
been removed, which also improved handling of the WAL (Sqlite write
ahead log).

There's also a few new features. Agents can be deactivated which means
they won't get any builds allocated. The coordinator now checks the
hashes of outputs which are submitted, a safeguard which I added because
the coordinator now also supports resuming the uploads of outputs. This
is particularly important when trying to upload large (> 1GiB) outputs
over slow connections.

I also added a new x86_64 build machine. It's a 4 core Intel NUC that I
had sitting around, but I cleaned it up and got it building things. This
was particularly useful as I was able to use it to retry building
guile@3.0.7, which is extremely hard to build [2]. This was blocking
building the channel instance derivations for x86_64-linux.

2:
https://data.guix.gnu.org/gnu/store/7k6s13bzbz5fd72ha1gx9rf6rrywhxzz-guile-3.0.7.drv

On the related subject of data.guix.gnu.org (which is the source of
derivations for bordeaux.guix.gnu.org, as well as a recipient of build
information), there have been a couple of changes. There was some web
crawler activity that was slowing data.guix.gnu.org down significantly,
NGinx now has some rate limiting configuration to prevent crawlers
abusing the service. The other change is that substitutes for the latest
processed revision of master will be queried on a regular basis, so this
page [3] should be roughly up to date, including for ci.guix.gnu.org.

3:
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-substitute-availability

Now for some not so good things:

Submitting builds wasn't working quite right for around a month, one of
the changes I made to speed things up led to some builds being
missed. This is now fixed, and all the missed builds have been
submitted, but this was more than 50,000 builds. This, along with all
the channel instance derivation builds that can now proceed mean that
there's a very large backlog of x86 and ARM builds which will probably
take at least another week to clear. While this backlog exists,
substitute availability for x86_64-linux will be lower than usual.

Space is running out on bayfront, the machine that runs the coordinator,
stores all the nars and build logs, and serves the substitutes. I knew
this was probably going to be an issue, bayfront didn't have much space
to begin with, but I had hoped I'd be further forward in developing some
way to allow moving the nars around between multiple machines, to remove
the need to store all of them on bayfront. I have got a plan, there's
some ideas I mentioned back in February [4], but I haven't got around to
implementing anything yet. The disk space usage trend is pretty much
linear, so if things continue without any change, I think it will be
necessary to pause the agents within a month, to avoid filling up
bayfront entirely.

4: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Thanks,

Chris

signature.asc
Description: PGP signature

Re: data.guix.gnu.org problem with revision 33d2574

2021-09-24 Thread Christopher Baines


Ludovic Courtès  writes:

> Christopher Baines  skribis:
>
>> guix-data-service: computing the derivation-file-name for mips64el-linux
>> Computing Guix derivation for 'mips64el-linux'...
>
> [...]
>
>> @ build-started
>> /gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv
>> - mips64el-linux
>> /var/log/guix/drvs/l8//9jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv.bz2
>> 28674
>> @ build-started
>> /gnu/store/ibk4hi6nxsh2j0v69i5fc55lw923s98z-gcc-4.8.2.tar.xz.drv -
>> mips64el-linux
>> /var/log/guix/drvs/ib//k4hi6nxsh2j0v69i5fc55lw923s98z-gcc-4.8.2.tar.xz.drv.bz2
>> 28680
>> Downloading 
>> https://ci.guix.gnu.org/nar/f2j6pi0d18pbz35ypflp61wzhbfcr8dp-linux-libre-4.14.67-gnu.tar.xz...
>> . linux-libre-4.14.67-gnu.tar.xz 94.7MiB 0B/s 00:00 [ ]
>> 0.0%. linux-libre-4.14.67-gnu.tar.xz 94.7MiB 62KiB/s 00:00 [ ]
>> 0.0%@ unsupported-platform
>> /gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv
>> mips64el-linux
>> while setting up the build environment: a `mips64el-linux' is
>> required to build
>> `/gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv',
>> but I am a `x86_64-linux'
>> builder for 
>> `/gnu/store/l89jsjd4881jv5pgc06z76vfw02hr7xs-guile-bootstrap-2.0.drv' failed 
>> with exit code 1
>
> It seems to be working as expected, no?  Computing the derivation for
> mips64el-linux entails building mips64el-linux derivations (due to
> grafts), and that fails.

Well, it's subtle that things are quite wrong. There's this bit, which
looks fine:

  Authenticating channel 'guix', commits 9edb3f6 to 33d2574 (1 new commits)...

Later on though, this is one of the clearer signs that whatever being
processed doesn't actually correspond to that revision:

  warning: no (guix lint) module found


signature.asc
Description: PGP signature

Re: OBS substitutes

2021-09-21 Thread Christopher Baines


zimoun  writes:

> I am confused because using Guix fb32a38, I get:
>
> $ guix weather obs
> computing 1 package derivations for x86_64-linux...
> looking for 1 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   0.0% substitutes available (0 out of 1)
>   unknown substitute sizes
>   0.0 MiB on disk (uncompressed)
>
>   0.0% (0 out of 1) of the missing items are queued
>   at least 1,000 queued builds
>   powerpc64le-linux: 999 (99.9%)
>   i686-linux: 1 (.1%)
>   build rate: .00 builds per hour
>   powerpc64le-linux: 0.00 builds per hour
>   x86_64-linux: 0.00 builds per hour
>   i686-linux: 0.00 builds per hour
>   aarch64-linux: 0.00 builds per hour
> looking for 1 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   100.0% substitutes available (1 out of 1)
>   4.0 MiB of nars (compressed)
>   15.2 MiB on disk (uncompressed)
>   (continuous integration information unavailable)
>
>
> well, I do not understand why it is not on Berlin.  But why not.  Then,
>
> $ guix build obs -n
> The following derivations would be built:
>/gnu/store/iz6yxr0jz312xz6494zc9wwv3w2lxzs5-obs-27.0.1.drv
>/gnu/store/s4yg1apy5m3l2scqbd2v27bcs07c1b5j-obs-27.0.1.tar.xz.drv
>/gnu/store/i9r2gwiqqr3ggyf63ad3k6ardiv68w1c-obs-27.0.1-checkout.drv
>
>
> Hum, I am confused why Guix wants to build from source when it is
> available on Bordeaux.  I have checked my config, but why not if I
> miss something.  Explicitly,
>
> $ guix build obs --substitute-urls="https://bordeaux.guix.gnu.org;
> The following derivations will be built:
>/gnu/store/iz6yxr0jz312xz6494zc9wwv3w2lxzs5-obs-27.0.1.drv
>/gnu/store/s4yg1apy5m3l2scqbd2v27bcs07c1b5j-obs-27.0.1.tar.xz.drv
>/gnu/store/i9r2gwiqqr3ggyf63ad3k6ardiv68w1c-obs-27.0.1-checkout.drv
> building 
> /gnu/store/i9r2gwiqqr3ggyf63ad3k6ardiv68w1c-obs-27.0.1-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to 
> `/gnu/store/v6f44zccwh9z5zk3pjlywjybbi8n2hjh-tar-1.32/bin:/gnu/store/ncydgq2znms5n1d2k5yqshhf58nsixwv-gzip-1.10/bin:/gnu/store/i8h2pcxqdq07ijm3ibkka8f4smn1w48v-bzip2-1.0.8/bin:/gnu/store/9860f1abqj8wjjnwl8a9v54pdcc3bhgf-xz-5.2.4/bin:/gnu/store/60g7r3l01fd7c58yjbm6krgcwj1jkpwg-file-5.38/bin:/gnu/store/n4n560pfvvw50a9369axw5vj5rrqfj1n-diffutils-3.7/bin:/gnu/store/cd5qf3kcnlq35p9k392pjdpdzpsnds70-patch-2.7.6/bin:/gnu/store/hic7snhayfl7m6cpfqqr73nmm19bpqkg-findutils-4.7.0/bin:/gnu/store/swqdvwri9dbv6zssg6v0by7l05hd6wxp-gawk-5.0.1/bin:/gnu/store/ishk7fswcs4gkwcp8mh788z4mvvl9bxh-sed-4.8/bin:/gnu/store/bhs4rj58v8j1narb2454raan2ps38xd8-grep-3.4/bin:/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/bin:/gnu/store/hm40bxnv8jxmbc1lpb7zfimii4xm9m81-make-4.3/bin:/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin:/gnu/store/mpa04aq8lblbcviyxywxcsb1zbi0mf39-ld-wrapper-0/bin:/gnu/store/m1z7cdbqsqyp9xnjw5cvlb4a7gkcg3m4-binutils-2.34/bin:/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/bin'
> hint: Using 'master' as the name for the initial branch. This default branch 
> name
> hint: is subject to change. To configure the initial branch name to use in all
> hint: of your new repositories, which will suppress this warning, call:
> hint: 
> hint: git config --global init.defaultBranch 
> hint: 
> hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
> hint: 'development'. The just-created branch can be renamed via this command:
> hint: 
> hint: git branch -m 
> Initialized empty Git repository in 
> /gnu/store/86x4q8w68984r6n4j0wl00zw1zq8xy96-obs-27.0.1-checkout/.git/
>   C-c C-c
>
> Why?  How is it possible that “guix weather” reports that the
> substitute is available but then “guix build” does not fetch it?

guix weather just looks whether there's a substitute.

When you run guix build, that substitute will only be used if it's
signed by a key that is in your ACL. My guess here is that this isn't
the case.

On Guix Systems, if the substitute keys are left as the default, for
recent revisions of Guix, the bordeaux.guix.gnu.org signing key will be
included in the ACL by default. With Guix on foreign distributions, it
needs adding manually.

I think there's some room for improvement in the UI/UX here, in terms of
telling users that there are substitutes available, if they trust a
specific key (all the relevant information is in the narinfo).


signature.asc
Description: PGP signature

Re: guix weather -m etc/sources-manifest.scm and CI?

2021-09-21 Thread Christopher Baines

zimoun  writes:

>> As for keeping build results, everything that's ever been built for
>> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
>> still around. In some ways, this is because deleting things is a bit
>> more difficult, as the files aren't in the store, you can't just do a
>> guix gc.
>>
>> However, I do want to try to never delete anything. That's going to
>> require a bit of work as the local storage on the bayfront machine will
>> be all used up at some point, but I do have the beginnings of a plan to
>> avoid this an keep building things.
>
> From my understanding, the project does not have a lot of resources;
> especially about storage.  And my question is: do we have a “coherent”
> plan between the machines behind Bordeaux and the ones behind Berlin?

Not that I'm following at least. The machines behind
bordeaux.guix.gnu.org were just ones that I could find to use when the
plan of dpeloying behind ci.guix.gnu.org was looking less and less
likely.

> I mean, IMHO, it does not seem a correct usage of project resource to
> back twice all from v1.0 to now (maybe from earlier to later).  And I
> also have in mind energy consumption which is becoming more and more a
> precious value.
>
> For instance, we could imagine the both build farms build and challenge
> the outputs.  Then, on Berlin the identical outputs are deleted after
> say 6 months and only live on Bordeaux.  Bordeaux being the backup
> time-machine.  That’s only an idea to open the discussion. :-) Not a
> concrete plan.

Some duplication of stored data is going to be necessary for redundancy
I think, for bordeaux.guix.gnu.org I was thinking of at least trying to
store all the substitute data in two different places.

As you've mentioned, building the same things multiple times is also
valuable for testing reproduciblity.

> Moreover, Disarchive is also something that should be added; probably to
> machines behind Bordeaux in the above plan.

I haven't been following Disarchive too closely. I think I remember
thinking it might be interesting for it to listen to data from
data.guix.gnu.org, to find out about new source tarballs (which isn't
really possible yet).

> From my point of view, what is really missing are the derivations for
> *all* the commits; for example module-import.drv, guix-33bc3fb2a.drv,
> guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv,
> guix-packages-modules.drv, guix-system-tests-modules.drv,
> guix-packages-base-modules.drv, etc.  And the backup time-machine should
> systematically build them for all the reachable commits.

If you're just talking about the derivations, data.guix.gnu.org does
store these derivations for the commits it processes (which are just the
states of the master branch, and that data is also missing for some
older commits that were processed a while ago).

If you're talking about substitutes for the outputs of these
derivations, the situation on bordeaux.guix.gnu.org isn't great for
x86_64-linux currently, since building guile@3.0.7 is blocking building
these things for recent revisions [1]. The underlying reason is a
problem with the testsuite I think, but it's hard to work around for
some other reasons as well. Once it can be built, all the blocked builds
will be attempted too.

1: 
https://data.guix.gnu.org/gnu/store/5z950sh1wpi59mfh53qdjdn8qnlcwp8g-guile-3.0.7.drv

I am interested in what can be done to support using guix time-machine
though, I think there's a range of things like storing substitutes,
testing things better (so they don't start failing to build in the
future), and helping users find the right commits to use.

Chris

signature.asc
Description: PGP signature

Re: data.guix.gnu.org problem with revision 33d2574

2021-09-21 Thread Christopher Baines


zimoun  writes:

> Hi Chris,
>
> On Fri, 17 Sep 2021 at 18:42, Christopher Baines  wrote:
>> Christopher Baines  writes:
>>
>>> In summary: data.guix.gnu.org is going to be down while I attempt to
>>> restore a backup from earlier this week.
>>
>> The restore was actually quicker than I expected, only taking ~2 hours,
>> so data.guix.gnu.org is back now.
>
> Cool!  Thank you!
>
>> It's still going to take some time to catch up and process recent
>> revisions. I also forgot about the build data being sent over from
>> bordeaux.guix.gnu.org, so I'll need to work out some way to replay the
>> events since Monday so that data.guix.gnu.org isn't missing that
>> information. Until that point, the build information won't be correct/up
>> to date.
>
> A question. :-) The “Builds” field shows bordeaux.guix.gnu.org and not
> ci.guix.gnu.org, right?  Does data.guix.gnu.org still process Berlin?

Correct, the version of Cuirass currently deployed on ci.guix.gnu.org
doesn't support sending data to the Guix Data Service.


signature.asc
Description: PGP signature

Re: data.guix.gnu.org problem with revision 33d2574

2021-09-17 Thread Christopher Baines

Christopher Baines  writes:

> In summary: data.guix.gnu.org is going to be down while I attempt to
> restore a backup from earlier this week.

The restore was actually quicker than I expected, only taking ~2 hours,
so data.guix.gnu.org is back now.

It's still going to take some time to catch up and process recent
revisions. I also forgot about the build data being sent over from
bordeaux.guix.gnu.org, so I'll need to work out some way to replay the
events since Monday so that data.guix.gnu.org isn't missing that
information. Until that point, the build information won't be correct/up
to date.

signature.asc
Description: PGP signature

Re: guix weather -m etc/sources-manifest.scm and CI?

2021-09-17 Thread Christopher Baines


zimoun  writes:

> Playing with the new 'etc/sources-manifest.scm', using fb32a38, I get:
>
> $ guix weather -m ~/src/guix/guix/etc/source-manifest.scm
> computing 16,831 package derivations for x86_64-linux...
> looking for 16,831 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   74.6% substitutes available (12,556 out of 16,831)
>   at least 65,367.1 MiB of nars (compressed)
>   81,988.1 MiB on disk (uncompressed)
>   0.095 seconds per request (1,606.8 seconds in total)
>   10.5 requests per second
>
>   0.0% (0 out of 4,275) of the missing items are queued
>   5 queued builds
>   aarch64-linux: 4 (80.0%)
>   powerpc64le-linux: 1 (20.0%)
>   build rate: .00 builds per hour
>   powerpc64le-linux: 0.00 builds per hour
>   aarch64-linux: 0.00 builds per hour
>   i686-linux: 0.00 builds per hour
>   x86_64-linux: 0.00 builds per hour
> looking for 16,831 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   99.8% substitutes available (16,804 out of 16,831)
>   62,195.0 MiB of nars (compressed)
>   108,212.7 MiB on disk (uncompressed)
>   0.049 seconds per request (829.2 seconds in total)
>   20.3 requests per second
>   (continuous integration information unavailable)
>
>
> The questions are:
>
> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
> (I guess the missing ones on bordeaux are corner cases as icecat, 
> linux-libre).

bordeaux.guix.gnu.org is hopefully only missing substitutes for things
where there's actually an issue building them. It's possible not to
guess though and instead ask guix weather what is missing.

I used time-machine as well for the latest commit processed by
data.guix.gnu.org, just so that the list doesn't include builds which
haven't started yet.

→ guix time-machine --commit=33bc3fb2a5f30a6e21f1b8d6d43867d921bd951c -- 
weather --substitute-urls=https://bordeaux.guix.gnu.org --display-missing -m 
./etc/source-manifest.scm
computing 16,850 package derivations for x86_64-linux...
looking for 16,850 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  99.8% substitutes available (16,822 out of 16,850)
  62,377.9 MiB of nars (compressed)
  108,593.9 MiB on disk (uncompressed)
  0.160 seconds per request (4.5 seconds in total)
  6.3 requests per second
  (continuous integration information unavailable)

Substitutes are missing for the following items:
  /gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz
 
i686-linux
  /gnu/store/9fgglvi13vhpc63knf15dipzmmck6ia9-mini-os-git-checkout  
 
i586-gnu
  /gnu/store/pb5jmi9zalg6xylzsjmrskwxs0kar97l-fossil-src-2.11.tar.gz
 
armhf-linux
  /gnu/store/8s5d2hjd6nhzf6m9dlxrykk8ijkf62pi-texlive-marginnote-51265-checkout 
 
armhf-linux
  /gnu/store/cp3ka40bhb28rrmyj4mzf9xjhi0ssxjx-CombBLAS_beta_16_2.tgz
 
x86_64-linux
  /gnu/store/f1h94axw82id0k8c2lippg73sqlibqs8-dovecot-trees-2.1.0.tar.gz
 
aarch64-linux
  /gnu/store/f12l7dg2z1xyf2wcw9g3v7jwpgd8m5zv-tla2tools-1.8.0-checkout  
 
i686-linux
  /gnu/store/53j9996hdgnmhgzswjjggdz9wnv29p5b-jpegsrc.v9d.tar.gz
 
i586-gnu
  /gnu/store/80s1m4q2hnjfbqzw3fhywvsyim2b00cd-gcc-4.7.4.tar.bz2 
 
x86_64-linux
  /gnu/store/csv5xca0p8w5jqqx53szy2dja8lwxma2-unicode-blocks.txt
 
i686-linux
  /gnu/store/xn62dzq9hw3qnvmbxyxjkvhlacs72rz7-canada1500.zip
 
armhf-linux
  /gnu/store/6vqin3by3nkn0sxhgwnzi9l7gflpfw1q-gcc-vc4-6.5.0-checkout
 
x86_64-linux
  
/gnu/store/zxfkf2bzq7pp7nhmbdgzvmjp0iv46wds-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout
i686-linux
  
/gnu/store/g98l0m8qsfdqybak2jz3ma2miv9bki4j-emacs-danneskjold-theme-0.0.0-2.e4d1f2c-checkout
   x86_64-linux
  /gnu/store/bznj81ls02r7kwld6338dhba0pql7nik-Rserve_1.8-6.tar.gz   
 
i686-linux

Re: Update on bordeaux.guix.gnu.org

2021-08-18 Thread Christopher Baines

Vagrant Cascadian  writes:

> On 2021-08-18, Christopher Baines wrote:
>> Around 2 months ago, bordeaux.guix.gnu.org came in to existence [1][2].
>>
>> 1: 
>> https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/
>> 2: https://lists.gnu.org/archive/html/guix-devel/2021-06/msg00044.html
>>
>> This took work I'd done on providing substitutes back in 2020 and
>> attempted to bring benefits from that to normal users of Guix.
>>
>> Unfortunately, I don't really know if this has been much of a
>> success.
>
> I can definitely speak from experience the likelihood of actually
> getting substitutes for any aarch64 linux-libre* packages has gone from
> "rarely" to "usually", so it has definitely been a huge improvement for
> aarch64, where building locally on most aarch64 hardware is very slow.

That's good to hear. Unfortunately I only got around to enabling
armhf-linux builds last week, but now there should be good substitute
availability for armhf-linux as well.

> Does it have faster build machines? Does it keep retrying until it
> succeeds? Building linux-libre* packages for ci.guix.gnu.org usually
> timeout when building the source due to long periods with no output...

Currently for arm builds, there's a Overdrive machine (monokuma) and a
Honeycomb machine. I would guess that the Honeycomb machine is faster,
but I haven't really compared the performance.

I do have the guix-daemon timeouts set to pretty high values (max silent
time of 12 hours, timeout of 24 hours) which is probably necessary for
the linux-libre* packages.

There's also automatic retries, a failed build will be retried
twice. That's more relevant for things that sometimes fail to build
though, or fail to build on particular hardware, rather than timeouts.

signature.asc
Description: PGP signature

Update on bordeaux.guix.gnu.org

2021-08-18 Thread Christopher Baines

Hey!

Around 2 months ago, bordeaux.guix.gnu.org came in to existence [1][2].

1: 
https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/
2: https://lists.gnu.org/archive/html/guix-devel/2021-06/msg00044.html

This took work I'd done on providing substitutes back in 2020 and
attempted to bring benefits from that to normal users of Guix.

Unfortunately, I don't really know if this has been much of a
success. While it should be possible to track requests for substitutes
to roughly see if anyone is making use of them, this is something I
haven't been doing yet.

In terms of the substitute availability stats, I think it's delivered
the expected benefits. I recently enabled armhf-linux builds, so now
substitute availability for the following 5 architectures should be
good:

 - x86_64-linux
 - i686-linux
 - aarch64-linux
 - armhf-linux
 - powerpc64le-linux

You can use guix weather to check the stats yourself, or look at [3] for
an overview (ignore the ci.guix.gnu.org numbers, as they're not
currently up to date).

3: 
https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-substitute-availability

There's still some issues holding substitute availability back. The Guix
Build Coordinator still has issues building things that it can't garbage
collect. The majority of the issues though are actual problems, like
broken fixed output derivations, or generally broken packages.

The next steps in my mind remain roughly the same as they were 2 months
ago:

 - It would be good to have something to provide more visibility in to
   the Guix Build Coordinator as well as the submitting of the builds

 - Supporting performant mirroring would be great, and I have some ideas
   of how to go about this

 - I did previously have some success building things for the Hurd [4],
   and it would be great to try and replicate this on
   bordeaux.guix.gnu.org

 - data.guix.gnu.org performance in processing new revisions is a
   limiting factor, so improving this would be helpful

 - Having aggregate statistics on use of substitutes (splitting out
   machines in the build farm) would be good for assessing use and
   changes in use

 - More hardware would be good for build throughput and redundancy. For
   example, there's currently only two ARM build machines linked up, and
   I host both of them.

4: https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00074.html

If you're interested in getting involved, or have any comments or
questions, please just let me know!

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Hundreds of failed build on master following git package update

2021-08-17 Thread Christopher Baines

br...@waegenei.re writes:

> Today I pushed 2 commits¹ related to the package to master, mainly to
> update it to 2.33.0. But since then https://data.guix.gnu.org/ went
> offline and Cuirass' evaluation #4930² has 556 failed build. Newly
> failed builds have unhelpful logs, ending with "cannot build missing
> derivation [...]"³.
>
> Before pushing those commits, I build git and git-minimal successfully
> on x86_64 and assessed the total impacted package for this update to
> be inferior to 300. Did I do something wrong with the update commit?
> Or is it an unrelated issue?

Hey,

Thanks for noticing that data.guix.gnu.org was down. I've had a look,
and the machine that runs it ran out of disk space.

I'm not quite sure why it ran out of disk space, I suspect it was maybe
excessive PostgreSQL logs... anyway, I'll get things going again and
keep an eye on it.

Once data.guix.gnu.org has caught up it will be easier to see what the
impact of this change is. I don't have any reason to think it was wrong
to make these changes.

Thanks,

Chris

signature.asc
Description: PGP signature

Re: Project direction with testing changes (branches and patches)

2021-08-11 Thread Christopher Baines


Ludovic Courtès  writes:

> Hi Chris,
>
> Christopher Baines  skribis:
>
>> I was thinking of using Cuirass for building derivations when testing
>> patches, but I gave up on that approach back in 2019 after trying to use
>> it (I discussed trying to use it here [1]).
>>
>> 1: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00010.html
>>
>> I was specifically thinking about testing patches when I initially
>> designed both the Guix Data Service and Guix Build Coordinator. For me
>> at least, the focus has been on this direction for the last ~3 years.
>>
>> I realise that Cuirass now has some of the functionality that the Guix
>> Data Service was written to provide, like tracking all
>> packages/derivations in each revision. But from my perspective, Cuirass
>> still lacks key things like being able to compare two revisions and
>> tracking lint warnings.
>
> Cuirass has always had to track packages/derivations in each revision;
> it’s been this way from day 1 when it started more or less as a
> revisited Hydra, which did exactly this.

I know it tracks some derivations against a revision, but it has only
been tracking all of the derivations associated with each revision since
this change [1].

1: 
https://git.savannah.nongnu.org/cgit/guix/guix-cuirass.git/commit/?id=bba1311478a50c837a8c70a556d308ca32ead816

>> There's also things like testing derivations on different hardware,
>> regularly testing fixed output derivations and automatically retrying
>> failed builds that I think the Guix Build Coordinator is better setup to
>> do compared to Cuirass.
>>
>> But this feedback is why I started this thread. I don't see the same
>> option as was found for improving substitutes by setting up a new
>> substitute server using the Guix Data Service and the Guix Build
>> Coordinator. There's a much stronger need to have one approach as a
>> project for testing changes, and if using the Guix Data Service and Guix
>> Build Coordinator isn't looking like a convincing option at this point,
>> that's better to know now, compared to later when more time and effort
>> has been put in.
>
> I can sympathize with the bitter feeling.  I do think though that we
> must work collectively; to me it’d be a problem if misplaced competition
> were to prevent us from moving forward.
>
> Several concrete incremental steps were proposed in this thread and
> earlier.  Instead of trying to provide a definite answer as to whether
> the grand plan you propose is a convincing option at this point, I’d
> like us to collect the low-hanging fruits, in an opportunistic way.  :-)
>
> Several easy hacks have been proposed in this thread and before: custom
> web/CLI views for the Data Service, Cuirass APIs to spin up specs on the
> fly, Data Service integration in Mumi/Gitile, Cuirass notifications sent
> to the Data Service, etc.  None of these is impressive in itself, but
> each of these can be a step making our hacker lives better, IMO.
>
> WDYT?

I don't perceive this as bitterness, just pragmatism.

I think I have an answer now as to whether there's consensus on making
use of the Guix Data Service and Guix Build Coordinator for testing
changes. Knowing there's some objections is useful when considering the
risk and managing my own expectations.

I still personally think that the general direction this work is going
is a good one. There's still some areas of uncertianty, but there's
definitely some stuff that can be done to move forward and more
discussion that can be had.

Thanks for your suggestions on next steps, I still need to get my own
thoughts in order. As you alluded to, I do like to have a plan in mind.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Project direction with testing changes (branches and patches)

2021-08-10 Thread Christopher Baines

Mathieu Othacehe  writes:

>> I think trying to change up how branches (staging/core-updates) are
>> tested is a good place to start. The concrete change I'm proposing is to
>> use an instance of the Guix Data Service plus an instance of the Guix
>> Build Coordinator to do the testing and builds, rather than Cuirass on
>> ci.guix.gnu.org which is the current approach.
>>
>> The main advantages of that would be the comparison support from the
>> Guix Data Service, and the build performance and reliability that the
>> Guix Build Coordinator brings. The main disadvantage is probably the
>> lack of an admin like interface similar to that of Cuirass (I think this
>> can be remedied in the medium term though).
>
> We indeed desperately need some more automation. For each new patch
> series, it would be great to have the following information:
>
> * Status of the linter.
> * Status of the depending derivations.
> * Status of the unit tests (in the tests/ directory).
> * Status of the system tests (in the gnu/tests/ directory).
>
> I would like to stay focused on the existing, well adopted solutions and
> build upon them.  With Cuirass we already have most of the machinery to
> provide those information.

I was thinking of using Cuirass for building derivations when testing
patches, but I gave up on that approach back in 2019 after trying to use
it (I discussed trying to use it here [1]).

1: https://lists.gnu.org/archive/html/guix-devel/2019-03/msg00010.html

I was specifically thinking about testing patches when I initially
designed both the Guix Data Service and Guix Build Coordinator. For me
at least, the focus has been on this direction for the last ~3 years.

I realise that Cuirass now has some of the functionality that the Guix
Data Service was written to provide, like tracking all
packages/derivations in each revision. But from my perspective, Cuirass
still lacks key things like being able to compare two revisions and
tracking lint warnings.

There's also things like testing derivations on different hardware,
regularly testing fixed output derivations and automatically retrying
failed builds that I think the Guix Build Coordinator is better setup to
do compared to Cuirass.

But this feedback is why I started this thread. I don't see the same
option as was found for improving substitutes by setting up a new
substitute server using the Guix Data Service and the Guix Build
Coordinator. There's a much stronger need to have one approach as a
project for testing changes, and if using the Guix Data Service and Guix
Build Coordinator isn't looking like a convincing option at this point,
that's better to know now, compared to later when more time and effort
has been put in.

Thanks,

Chris

signature.asc
Description: PGP signature

Re: Project direction with testing changes (branches and patches)

2021-08-09 Thread Christopher Baines


Ludovic Courtès  writes:

> Christopher Baines  skribis:
>
>> So, I think I've recently switched to thinking about the problem as one
>> of testing changes, rather than just testing patches. Since both patch
>> series, and branches are used to propose changes, I think this makes
>> sense.
>>
>> In abstract, when testing a change, I would break down the problem as
>> follows:
>>
>>   - You need to work out what's affected by the change, so that you can
>> assess the impact
>>
>>   - Once you know what's effected, you can then build those
>> packages/system tests/... and compare the build statuses and outputs
>> against some baseline
>>
>>   - Then there's the general UI component, ideally a first time
>> contributor would be able to take advantage of automatic feedback
>> about a patch they submit. There's multiple other groups of users
>> though, like patch reviewers, and committers for example.
>
> Makes sense to me.
>
> I agree that the first problem, seeing what’s affected by a change, is
> solved, but it’s still hard to get that info.  I think we could have a
> special “skin” for the Guix Data Service to make it easier for people to
> view specifically this information.  IMO the current UI has the upside
> that it’s generic and exposes all the available information, but it has
> the downside that it’s generic and exposes all the available
> information.  :-)
>
> Or we could extend Julien’s Gitile¹ to include links from commits to
> lists of changed packages.
>
> The UI doesn’t have to be a web UI actually; we could use the Data
> Service client interface at
> <https://lists.gnu.org/archive/html/guix-devel/2021-06/msg00228.html>
> and write a new CLI, Emacs mode (similar to ‘M-x build-farm’), or
> something.
>
> ¹ https://git.lepiller.eu/gitile

Indeed, the technical side of detecting changes is solved, but I agree
that there's work needed on making that information useful.

>> I think the first two sub-problems are effectively solved. The Guix Data
>> Service is able to determine the changes between two revisions (assuming
>> it's processed them). The Guix Build Coordinator can then be used to
>> build the relevant packages/system tests, and report that information
>> back to the Guix Data Service.
>>
>> The UI part is much less certain, I've done some work with Patchwork,
>> and I do have some ideas in mind, but there's still more thinking and
>> work to do in this area.
>>
>> Before pressing on though, I think it would be good to know if this is a
>> viable direction?
>
> I think we desperately need more automation, even more than when you
> started working on this!
>
> I think a first step could be to make info from the Guix Data Service
> more readily available, as suggested above.  And from there we could
> address #2 and #3.
>
> The Patchwork instance you maintain at
> <https://patches.guix-patches.cbaines.net/project/guix-patches/list/>
> does a large part of what we want, though the UI is not my favorite I
> must say.  ;-)  I wonder if we could again make minimal changes to Mumi
> so that it includes links to the relevant bits at the Data Service.
> That’d make it more readily available.  WDYT?

I think using Patchwork or Mumi is viable, it would probably not be
great to use both in the long term. The most useful thing for me would
be to pick an approach.

Patchwork is closer in terms of features I think, since it has an API
for patch series and checks. I know mumi gained the ability to generate
mbox files for patch series now though.

I think the requirements in terms of Mumi would be to have some way of
querying for patch series to test, or at least all/recent patch
series. I suppose something could just ask for the 1st or latest series
each time there's an email to a issue, that might be the simplest
approach. Then there's the bits you describe about showing relevant
information about whatever tests take place. I guess Mumi would need an
API or some way to find out about that information, and then display
it. Maybe that could happen in the form of emails with machine+human
readable data that Mumi could interpret.

Unfortunately I don't know much about the Mumi internals. Ricardo, are
you able to comment on the feasibility and whether this direction would
make sense?

>> Currently, there's no automated testing of patches, and testing of
>> branches is limited to the information that Cuirass provides on failed
>> builds. What I'm proposing for the future is: using the Guix Data
>> Service together with the Guix Build Coordinator to analyse the effects
>> of changes, whether that be from a patch series or a branch. I realis

Re: #f as a package description, gnu: Add rocminfo.

2021-08-09 Thread Christopher Baines


Lars-Dominik Braun  writes:

> Hi Christopher,
>
>> Anyway, I wouldn't like for this change to lower the standard though,
>> it's currently the only package in Guix with an invalid description (as
>> far as I'm aware), is there some reason why it doesn't have one?
> it simply fell through the cracks[1]. Commit
> 0a379de3249d5e9ff66fb404f7e5aa8ce2cb3d24 adds a proper descripton.
>
> Sorry for the trouble,

No problem, thanks for fixing it.

> [1] Unfortunately I cannot run `guix lint` on an entire git changeset,
> so instead I have to check each package by hand and I probably missed
> rocminfo. I wish someday we can have a branch/pull-request-based
> workflow with automated CI checks (linting, `guix pull`, signature
> verification, …) *before* merging to master.

This is something I've put some time in to, but it needs some general
support before I'm confident that the things I have in mind can become
reality. I recently started a thread on guix-devel [1] about what I have
in mind, so please comment if you're interested in this area.

1: https://lists.gnu.org/archive/html/guix-devel/2021-08/msg1.html


signature.asc
Description: PGP signature

#f as a package description, gnu: Add rocminfo.

2021-08-08 Thread Christopher Baines


guix-comm...@gnu.org writes:

> lbraun pushed a commit to branch master
> in repository guix.
>
> commit 91ce17a53236578f8055a2588460047741983925
> Author: Lars-Dominik Braun 
> AuthorDate: Fri Aug 6 08:29:12 2021 +0200
>
> gnu: Add rocminfo.
>
> * gnu/packages/rocm.scm (rocminfo): New variable.
> ---
>  gnu/packages/rocm.scm | 21 +
>  1 file changed, 21 insertions(+)
>
> diff --git a/gnu/packages/rocm.scm b/gnu/packages/rocm.scm
> index 07e395c..0cf5a34 100644
> --- a/gnu/packages/rocm.scm
> +++ b/gnu/packages/rocm.scm
> @@ -304,3 +304,24 @@ allows runtimes to work on Windows as well as on Linux 
> without much effort.")
>  (description "OpenCL 2.0 compatible language runtime, supporting offline
>  and in-process/in-memory compilation.")
>  (license license:ncsa)))
> +
> +(define-public rocminfo
> +  (package
> +(name "rocminfo")
> +(version %rocm-version)
> +(source (origin
> +  (method git-fetch)
> +  (uri (git-reference
> +(url "https://github.com/RadeonOpenCompute/rocminfo.git;)
> +(commit (string-append "rocm-" version
> +  (file-name (git-file-name name version))
> +  (sha256
> +   (base32
> +"0pcm308vwkjrwnrk507iya20mkil8j0vx699w9jk2gas4n4jvkcz"
> +(build-system cmake-build-system)
> +(arguments `(#:tests? #f)) ; No tests.
> +(inputs `(("rocr-runtime" ,rocr-runtime)))
> +(home-page "https://github.com/RadeonOpenCompute/rocminfo;)
> +(synopsis "ROCm Application for Reporting System Info")
> +(description #f)
> +(license license:ncsa)))

guix lint notes:
gnu/packages/rocm.scm:309:2: rocminfo@4.3.0: invalid description: #f

I noticed this because it broke an assumption about descriptions in the
Guix Data Service [1].

1: https://data.guix.gnu.org/revision/e81cf4e79a6e297db0ae2a9c39eab495e7e204f0

As this is now part of the history, I'll fix the Guix Data Service to be
able to cope when the description is #f.

Anyway, I wouldn't like for this change to lower the standard though,
it's currently the only package in Guix with an invalid description (as
far as I'm aware), is there some reason why it doesn't have one?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Adding Substitute Mirrors page to installer

2021-08-01 Thread Christopher Baines

raid5atemyhomework  writes:

> In any case, it looks to me that bordeaux is already in
> `%default-substitute-mirrors`, which this patch uses, so it should get
> included anyway as a fallback in case the SJTU mirror is not available
> or something.  So maybe the patch is OK as-is?

I haven't looked at the changes here, but given bordeaux.guix.gnu.org is
a default substitute server, I don't think it'll need special handling.

signature.asc
Description: PGP signature

Project direction with testing changes (branches and patches)

2021-08-01 Thread Christopher Baines

Hey,

This is sort of a followup to [1], at least I think that's the last main
email I sent out about testing changes (although I didn't use that
term). I did also send out some notes from the Guix Day event back in
February 2021 though [2].

1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00583.html
2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00125.html

Back in early 2020, I managed to start work on the Guix Build
Coordinator [3]. That was meant to enable running reliable and
performant substitute servers, but also meant to enable the kind of
testing and quality assurance work that I had been thinking about,
mostly through the perspective of testing patches.

3: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html

Getting the benefits to users didn't go as smoothly as I'd hoped, but
since bordeaux.guix.gnu.org [4] launched back in June, there's a chance
that the work on the Guix Build Coordinator has benefited users of Guix
through improved substitutes.

4: 
https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

As I said in [1], I did do some work last year to use the Guix Build
Coordinator for testing patches and branches. Unfortunately the setup
I'm using is currently not operating, I was having issues with running
out of disk space on the main server, and I haven't got around to
spending the time/money to resolve that.

I want to get another iteration of the patch testing setup working, but
recent experiences with working on providing substitutes has made me
think that discussing the direction with maintainers and as a project is
almost more important.

So, I think I've recently switched to thinking about the problem as one
of testing changes, rather than just testing patches. Since both patch
series, and branches are used to propose changes, I think this makes
sense.

In abstract, when testing a change, I would break down the problem as
follows:

  - You need to work out what's affected by the change, so that you can
assess the impact

  - Once you know what's effected, you can then build those
packages/system tests/... and compare the build statuses and outputs
against some baseline

  - Then there's the general UI component, ideally a first time
contributor would be able to take advantage of automatic feedback
about a patch they submit. There's multiple other groups of users
though, like patch reviewers, and committers for example.

I think the first two sub-problems are effectively solved. The Guix Data
Service is able to determine the changes between two revisions (assuming
it's processed them). The Guix Build Coordinator can then be used to
build the relevant packages/system tests, and report that information
back to the Guix Data Service.

The UI part is much less certain, I've done some work with Patchwork,
and I do have some ideas in mind, but there's still more thinking and
work to do in this area.

Before pressing on though, I think it would be good to know if this is a
viable direction?

Currently, there's no automated testing of patches, and testing of
branches is limited to the information that Cuirass provides on failed
builds. What I'm proposing for the future is: using the Guix Data
Service together with the Guix Build Coordinator to analyse the effects
of changes, whether that be from a patch series or a branch. I realise
that I've already been experimenting with this, what I'm mostly
referring to here is moving towards this being the documented approach,
maintained by the project, not just me.

So yes, is this something that people want, or don't want? If you're
uncertain and have questions, it would be good to know what those
questions are?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Honeycomb LX2 ARM board, problems booting Guix

2021-06-30 Thread Christopher Baines


Julien Lepiller  writes:

> One way to find the missing modules is to boot another distro (or
> maybe the installer has a lot more modules than the installed system)
> and look at dmesg. You might see something like:
>
> scsi host0: ahci-sunxi
> ata1: SATA max UDMA/133 mmio …
>
> sunxi-mmc 1c12000.mmc: initialized, …
> mmc0: host does not support reading read-only…
>
> (That's from an arm board)
>
> In that case, I added sunxi-mmc, ahci_sunxi and for some reason
> sd_mod. Note that naming is not consistent between guix and dmcsg, so
> check the filename in lib/modules of you linux-libre kernel package.
>
> HTH!

Thanks for the tip, reading the dmesg output for a system that boots was
helpful!


signature.asc
Description: PGP signature

Re: Honeycomb LX2 ARM board, problems booting Guix

2021-06-30 Thread Christopher Baines


Vagrant Cascadian  writes:

> On 2021-06-29, Christopher Baines wrote:
>> I've had a Honeycomb LX2 board for a few weeks now, but I've been
>> struggling to get Guix installed on it.
>>
>> The most success I've had is with an installer image written to either a
>> USB drive or nVME drive. I changed the bootloader to the
>> grub-efi-bootloader, using the efi-raw image type, a couple of other
>> tweaks, but I only get as far as Linux starting Guile and then it
>> promptly erroring [1].
>>
>> 1: https://paste.debian.net/plain/1202864
>>
>> For some reason, /proc/partitions is empty, and I can't see usual things
>> relating to storage devices in /dev. Any ideas how this could happen,
>> what might I be doing wrong, or anything else I could try?
>
> Most likely missing kernel modules from the initrd, missing kernel
> configuration options enabled, and/or missing support in the kernel
> upstream...
>
> Hopefully it's one of the first two; good luck module/option hunting!

Thanks for the tip, I think I've muddled through enabling relevant
things in Linux to get past the partition problem at least.

I've seemingly still got some problem getting in to the
installer/getting a tty over the serial connection, but I'll send
another email about that.


signature.asc
Description: PGP signature

Honeycomb LX2 ARM board, problems booting Guix

2021-06-29 Thread Christopher Baines

Hey,

I've had a Honeycomb LX2 board for a few weeks now, but I've been
struggling to get Guix installed on it.

The most success I've had is with an installer image written to either a
USB drive or nVME drive. I changed the bootloader to the
grub-efi-bootloader, using the efi-raw image type, a couple of other
tweaks, but I only get as far as Linux starting Guile and then it
promptly erroring [1].

1: https://paste.debian.net/plain/1202864

For some reason, /proc/partitions is empty, and I can't see usual things
relating to storage devices in /dev. Any ideas how this could happen,
what might I be doing wrong, or anything else I could try?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Substitutes from bordeaux.guix.gnu.org

2021-06-18 Thread Christopher Baines

Ricardo Wurmus  writes:

> Ludovic Courtès  writes:
>
>> BTW, did I mention we have budget?  :-)  Is there affordable POWER9
>> hardware we could purchase?
>
> Aside from that: do we have rack space for a POWER9 machine?
>
> For a limited time I could probably host it at the MDC data centre,
> but moving rack hardware to some other location is messy, so I think
> it should go straight to its final destination.

It would be good to discuss what the options are here.

The hardware can probably not be described as inexpensive, but it's also
quite capable so not much is needed.

Would trying to acquire a 2U rack mount server, and host it in a proper
data centre be feasible? Because of both electricity usage and heat
generation, it's going to be more difficult to host residentially I
think.

signature.asc
Description: PGP signature

Re: Substitutes from bordeaux.guix.gnu.org

2021-06-18 Thread Christopher Baines


Christopher Baines  writes:

> Christopher Baines  writes:
>
>> While I've been working on the software side of building things for
>> substitutes through the Guix Build Coordinator for over a year now [1],
>> I've only been personally pushing to bring the benefits to Guix users by
>> default for the last few weeks [2].
>>
>> 1: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html
>> 2: https://lists.gnu.org/archive/html/guix-devel/2021-05/msg00020.html
>>
>> The strategy that currently seems most feasible is to add an additional
>> default source of substitutes: bordeaux.guix.gnu.org.
>>
>> I think things are pretty much ready, you can find some information at
>> https://bordeaux.guix.gnu.org/ . If you're up for trying out fetching
>> substitutes from bordeaux.guix.gnu.org, now is a good time to do so, and
>> please report back in terms of how it's working out for you.
>
> The patch to enable bordeaux.guix.gnu.org as a default substitute server
> is available in [1].
>
> 1: https://issues.guix.gnu.org/48435
>
> I've created a wip-bordeaux branch [2] now which includes that patch,
> plus a commit to update the guix package to build guix including it. I
> think this will be helpful when testing the changes.
>
> 2: https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-bordeaux
>
> I've also drafted a blog post, which can be found here [3].
>
> 3: 
> https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/drafts/substitutes-from-bordeaux.guix.gnu.org.md
>
> If I can find the time to test things this week, maybe the changes in
> #48435 can be merged on Friday. If anyone is up for testing things, or
> reading through the blog post, that would be great.

These changes are now in master, and I've published the blog post!

  
https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

As said in the blog post, I regard this as an important milestone. As a
change goes, the impact will be gradual, but I'm hoping in hindsight it
will be a step forward.


signature.asc
Description: PGP signature

Re: RISCV porting effort

2021-06-13 Thread Christopher Baines

Efraim Flashner  writes:

> Last week my HiFive Umatched¹ board came and was quite the object of
> interest at the local computer store as I picked up a case, PSU and
> graphics card.

I also recently received a HiFive Umatched board which I ordered months
ago.

While I don't really have the time right at the moment to actually do
much with it, I've got it connected to the internet and can consider
offering SSH access to anyone who could make use of it for porting Guix
to RISCV.

I brought it with building things for Guix in mind, if people would
prefer it being owned by guix-europe, I'm open to selling it.

signature.asc
Description: PGP signature

Re: Substitutes from bordeaux.guix.gnu.org

2021-06-13 Thread Christopher Baines

Christopher Baines  writes:

> While I've been working on the software side of building things for
> substitutes through the Guix Build Coordinator for over a year now [1],
> I've only been personally pushing to bring the benefits to Guix users by
> default for the last few weeks [2].
>
> 1: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html
> 2: https://lists.gnu.org/archive/html/guix-devel/2021-05/msg00020.html
>
> The strategy that currently seems most feasible is to add an additional
> default source of substitutes: bordeaux.guix.gnu.org.
>
> I think things are pretty much ready, you can find some information at
> https://bordeaux.guix.gnu.org/ . If you're up for trying out fetching
> substitutes from bordeaux.guix.gnu.org, now is a good time to do so, and
> please report back in terms of how it's working out for you.

The patch to enable bordeaux.guix.gnu.org as a default substitute server
is available in [1].

1: https://issues.guix.gnu.org/48435

I've created a wip-bordeaux branch [2] now which includes that patch,
plus a commit to update the guix package to build guix including it. I
think this will be helpful when testing the changes.

2: https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-bordeaux

I've also drafted a blog post, which can be found here [3].

3: 
https://git.savannah.gnu.org/cgit/guix/guix-artwork.git/tree/website/drafts/substitutes-from-bordeaux.guix.gnu.org.md

If I can find the time to test things this week, maybe the changes in
#48435 can be merged on Friday. If anyone is up for testing things, or
reading through the blog post, that would be great.

Thanks,

Chris

signature.asc
Description: PGP signature

Substitutes from bordeaux.guix.gnu.org

2021-06-07 Thread Christopher Baines

Hey,

While I've been working on the software side of building things for
substitutes through the Guix Build Coordinator for over a year now [1],
I've only been personally pushing to bring the benefits to Guix users by
default for the last few weeks [2].

1: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html
2: https://lists.gnu.org/archive/html/guix-devel/2021-05/msg00020.html

The strategy that currently seems most feasible is to add an additional
default source of substitutes: bordeaux.guix.gnu.org.

I think things are pretty much ready, you can find some information at
https://bordeaux.guix.gnu.org/ . If you're up for trying out fetching
substitutes from bordeaux.guix.gnu.org, now is a good time to do so, and
please report back in terms of how it's working out for you.

If you're interested in the infrastructure behind bordeaux.guix.gnu.org,
these are the machines which are currently involved:

 - bayfront
   - running the coordinator
   - serving the substitutes
   - building things for x86_64-linux plus i686-linux

 - harbourfront
   - building things for x86_64-linux plus i686-linux

 - milano-guix-1
   - building things for x86_64-linux plus i686-linux

 - monokuma
   - building things for aarch64-linux

 - polaris
   - sometimes building things for powerpc64le-linux

Just let me know if you have any comments or questions.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-06-07 Thread Christopher Baines

Christopher Baines  writes:

> Christopher Baines  writes:
>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?
>
> It's been a few weeks now, so to summarise, I think only one path
> emerged, and that is to get substitutes from bayfront to users.

More weeks have past, it's taking me longer to get things sorted out
that I'd like, but things are still moving forward.

> Bayfront was already running the Guix Build Coordinator (although only
> for the last month), and it's now caught up to the point where I'm
> seeing similar or better substitute availability percentages for
> x86_64-linux (and powerpc64le-linux) when compared to
> ci.guix.gnu.org. It's also building i686-linux and aarch64-linux things,
> but they're still catching up.

Substitute availability for x86_64-linux and i686-linux should be
roughly comparable to ci.guix.gnu.org.  powerpc64le-linux substitute
availability is OK, and aarch64-linux might even be doing better than
ci.guix.gnu.org somehow.

Other things like armhf-linux and i586-gnu are still very much works in
progress.

> Obviously just having the substitutes doesn't magically get them to
> users, so I've started looking in to the changes to start making that
> happen. Adding the signing key and changing the defaults in a few places
> seems like a good step forward [1].
>
> 1: https://issues.guix.gnu.org/48435

I've gone ahead and put the key in to the Guix git repository [1] and
sent an updated patch for changing various bits of configuration [2].

1: 
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=26499816a973b3aab9aaf8e13b909d0bde4e2dd5
2: https://issues.guix.gnu.org/48435#8

I think the patch still needs a bit more work, mostly to update the
docs. I'll try to work out what needs tweaking in the docs and send a v3
ASAP.

In terms of what to initially change, I'm still not sure if there's
something that needs updating that I'm currently missing, or something
that I'm updating that can be done later.

> Apart from merging the changes in [1], I guess a blog post might be
> useful. Have I missed anything?

I'll start another thread on guix-devel to solicit feedback about
substitutes from bordeaux.guix.gnu.org, I'm not sure what specifically
about, but peoples observations might be helpful when writing a blog
post about this. I'll also try to start drafting a blog post.

What else needs doing to actually get these substitutes to users?

Thanks,

Chris

signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-19 Thread Christopher Baines


Mathieu Othacehe  writes:

> Hey Chris,
>
>> That sounds sensible. On the specific name, given this is just about
>> substitutes, and at least in my opinion has nothing to do with
>> continuous integration, maybe picking just another word would avoid
>> thinking too much, it could be bordeaux, or hippo, or anything
>> really. As you say, stability and not being tied to a particular machine
>> is the important thing.
>
> The substitutes coverage is one indicator to take into account but there
> are many others. For instance, the evaluation speed, the failed
> evaluation count, the average evaluation builds completion time, the
> availability of the connected build machines between other things.

Indeed, and I'm aware that the Guix Data Service, which performs a
similar function to the evaluations in Cuirass, is much slower.

> Deploying a solution that builds substitutes is fine, but as soon as it
> is deployed and accessible to all Guix users, the system administrators
> will have to monitor it and maintain it in the long run.
>
> Having two heterogeneous build infrastructures on two sets of machines,
> providing different metrics will make the update and maintenance of
> those machines harder.
>
> I hear your point about K-out-of-N policy and it also makes sense to
> me. However, we should maybe consider doing it using two similar
> infrastructures.

Indeed. The reality though is that two different approaches have been in
development now for a little over a year, and this is a reflection of
that.


signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-18 Thread Christopher Baines

Ludovic Courtès  writes:

> Hello!
>
> Christopher Baines  skribis:
>
>> Christopher Baines  writes:
>>
>>> Is there still a path to bring some of these benefits to users, and if
>>> so, what things need doing?
>
> [...]
>
>> Obviously just having the substitutes doesn't magically get them to
>> users, so I've started looking in to the changes to start making that
>> happen. Adding the signing key and changing the defaults in a few places
>> seems like a good step forward [1].
>>
>> 1: https://issues.guix.gnu.org/48435
>>
>> I want to push on with this within the next couple of weeks, mostly so I
>> can shift focus to Outreachy and the security related tooling work, but
>> also because I still think this will be a good step forward in terms of
>> substitute availability for users. It's been over a year now since
>> implementation started, so it would be good to actually make a positive
>> difference.
>
> I’m fine with distributing an extra signing key alongside that of
> ci.guix.gnu.org.

Great.

> I’m unsure about having two substitute URLs by default since it adds a
> bit of overhead, though that overhead is only upon cache misses (I have
> that setup on my laptop actually).

All of this work has been built on the assumption that it's possible to
do better in providing substitutes, and anecdotally from the data I've
seen over the last year, that should be possible, even with the limited
hardware (compared to ci.guix.gnu.org) connected to bayfront.

So yes, that's a valid concern, but if all the addition of bayfront does
is make users wait a little longer because of cache misses, it's a sign
that the whole endeavour is not working out.

> It’s also a one-way change: people are likely to keep the defaults
> “forever”.  So we can’t just “experiment” and change our mind later.
> That means we should at least have a DNS entry that’s not tied to a
> particular machine, like ci2.guix.gnu.org or whatever.

That sounds sensible. On the specific name, given this is just about
substitutes, and at least in my opinion has nothing to do with
continuous integration, maybe picking just another word would avoid
thinking too much, it could be bordeaux, or hippo, or anything
really. As you say, stability and not being tied to a particular machine
is the important thing.

signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-18 Thread Christopher Baines

Christopher Baines  writes:

> Is there still a path to bring some of these benefits to users, and if
> so, what things need doing?

It's been a few weeks now, so to summarise, I think only one path
emerged, and that is to get substitutes from bayfront to users.

Bayfront was already running the Guix Build Coordinator (although only
for the last month), and it's now caught up to the point where I'm
seeing similar or better substitute availability percentages for
x86_64-linux (and powerpc64le-linux) when compared to
ci.guix.gnu.org. It's also building i686-linux and aarch64-linux things,
but they're still catching up.

Obviously just having the substitutes doesn't magically get them to
users, so I've started looking in to the changes to start making that
happen. Adding the signing key and changing the defaults in a few places
seems like a good step forward [1].

1: https://issues.guix.gnu.org/48435

I want to push on with this within the next couple of weeks, mostly so I
can shift focus to Outreachy and the security related tooling work, but
also because I still think this will be a good step forward in terms of
substitute availability for users. It's been over a year now since
implementation started, so it would be good to actually make a positive
difference.

There's a few issues still on my mind. Even though the substitute
availability percentages are good when compared to ci.guix.gnu.org, as
bayfront has much less compute power connected, it might not keep up as
well when big sets of changes are merged. I think that's just an
argument for using the build coordinator on berlin and the connected
machines though.

The other thing in comparison to ci.guix.gnu.org is that bayfront only
has ~4TB of storage rather than ~37TB, and given that currently none of
the generated nars are deleted, that will need thinking about in a few
months to avoid running out of space. I've had some plans around this
for a while [1], but they still require implementing.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html

Apart from merging the changes in [1], I guess a blog post might be
useful. Have I missed anything?

Chris

signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-18 Thread Christopher Baines

Ludovic Courtès  writes:

> Hi,
>
> Christopher Baines  skribis:
>
>> If there's effort put in to getting substitutes served from bayfront,
>> why do you suggest not documenting how to get those substitutes in the
>> manual?
>
> Mostly because bayfront is not as powerful as the build farm behind
> ci.guix.

I see that mostly as an argument for trying to bring the benefits of the
Guix Build Coordinator to the substitutes served by ci.guix.gnu.org.

Even with 3 machines for x86_64-linux (2.x given that bayfront is only
lightly used), bayfront has pretty much caught up with ci.guix.gnu.org
in terms of x86_64-linux substitutes, and I expect it to have a higher
substitute availability percentage in the coming days, since it's still
working through the backlog.

Once it is doing better, at least for x86_64-linux substitutes, I see
that as a strong argument to promote it to users (default config change,
blog post, ...).

1:
→ guix describe
Generation 310  May 16 2021 09:06:12(current)
  guix 7c4c781
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 7c4c781aa40c42d4cd10b8d9482199f3db345e1b

→ guix weather --substitute-urls="https://bayfront.guix.gnu.org 
https://ci.guix.gnu.org;
computing 17,423 package derivations for x86_64-linux...
looking for 18,872 store items on https://bayfront.guix.gnu.org...
https://bayfront.guix.gnu.org
  91.6% substitutes available (17,293 out of 18,872)
  37,146.8 MiB of nars (compressed)
  143,978.2 MiB on disk (uncompressed)
  (continuous integration information unavailable)
looking for 18,872 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  92.1% substitutes available (17,389 out of 18,872)
  at least 139,781.5 MiB of nars (compressed)
  35,184,372,238,367.3 MiB on disk (uncompressed)

signature.asc
Description: PGP signature

Re: branch master updated: gnu: Add emacs-wucuo.

2021-05-15 Thread Christopher Baines


guix-comm...@gnu.org writes:

> This is an automated email from the git hooks/post-receive script.
>
> ngz pushed a commit to branch master
> in repository guix.
>
> The following commit(s) were added to refs/heads/master by this push:
>  new bb94429  gnu: Add emacs-wucuo.
> bb94429 is described below
>
> commit bb94429cb776dc658361e9c26f999c9648d0bd86
> Author: Joseph LaFreniere 
> AuthorDate: Mon Oct 19 22:21:04 2020 -0500
>
> gnu: Add emacs-wucuo.
>
> * gnu/packages/emacs-xyz.scm (emacs-wucuo): New variable.
>
> Signed-off-by: Nicolas Goaziou 
> ---
>  gnu/packages/emacs-xyz.scm | 34 ++
>  1 file changed, 34 insertions(+)
>
> diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
> index 7b999a9..8d8b3f9 100644
> --- a/gnu/packages/emacs-xyz.scm
> +++ b/gnu/packages/emacs-xyz.scm
> @@ -131,6 +131,7 @@
>#:use-module (guix build-system trivial)
>#:use-module (gnu packages)
>#:use-module (gnu packages admin)
> +  #:use-module (gnu packages aspell)
>#:use-module (gnu packages audio)
>#:use-module (gnu packages bash)
>#:use-module (gnu packages cmake)
> @@ -9843,6 +9844,39 @@ current frame, disabling the mode line, and adding 
> margins to the buffer that
>  restrict the text width to 80 characters.")
>  (license license:bsd-3)))
>
> +(define-public emacs-wucuo
> +  (package
> +(name "emacs-wucuo")
> +(version "0.2.9")
> +(source
> + (origin
> +   (method git-fetch)
> +   (uri (git-reference
> + (url "https://github.com/redguardtoo/wucuo;)
> + (commit "89b99166768afb811c48a7db7c93c02d51a32b09")))
> +   (file-name (git-file-name name version))
> +   (sha256
> +(base32 "03a6jlbv9axrd9yr0xscq3ni7fipm20ppc51kxy0sn241rplv0pg"
> +(build-system emacs-build-system)
> +(arguments
> + `(#:tests? #tn

The n on this line means that this package can't be built, and also
breaks the linter.

I've removed the n in fd2abc2a51e2cc39ac67dcef1d21a8037147e798.

> +   #:test-command '("make" "test")
> +   #:phases (modify-phases %standard-phases
> +  ;; Set HOME, otherwise tests fail on loading aspell dict.
> +  (add-before 'check 'set-home
> +(lambda _ (setenv "HOME" (getcwd)))
> +(native-inputs
> + ;; For tests.
> + `(("aspell" ,aspell)
> +   ("aspell-dict-en" ,aspell-dict-en)))
> +(home-page "https://github.com/redguardtoo/wucuo;)
> +(synopsis "Fast spell checker for camel case code or plain text")
> +(description
> + "Wucuo provides a spell checker on top of either Aspell or Hunspell, and
> +relies on Flyspell internally.  It operates on the current region or buffer,
> +a file, or a complete directory.")
> +(license license:gpl3+)))
> +
>  (define-public emacs-ido-completing-read+
>(package
>  (name "emacs-ido-completing-read+")


signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-15 Thread Christopher Baines

Ludovic Courtès  writes:

> That all sounds like a plan!
>
> Christopher Baines  skribis:
>
>> There's a few intertangled things, but the main question is, if bayfront
>> can be a source of substitutes, what's the path to actually have that
>> benefit users?
>
> One way it’ll benefit everyone is by feeding build/reproducibility data
> to the Data Service.
>
> Another way can be by advertising bayfront.guix.gnu.org to “insiders”
> (that is, maybe we won’t put it in the manual, but we can suggest this
> option to people reading this list etc.)
>
> Thoughts?

I guess my question was maybe too vague, I was thinking about getting
substitutes served from bayfront to users, and the benefit coming from
that.

In my mind, this would ideally include users who want to enable
substitutes, but don't know much more than that. I've started looking at
the changes this would involve [1].

1: https://issues.guix.gnu.org/48435

If there's effort put in to getting substitutes served from bayfront,
why do you suggest not documenting how to get those substitutes in the
manual?

signature.asc
Description: PGP signature

Re: bug#47897: [PATCH] substitutes: Don't cache negative lookups or transient errors.

2021-05-14 Thread Christopher Baines


Ludovic Courtès  writes:

> Hi,
>
> Christopher Baines  skribis:
>
>>> Now, the penalty it imposes is annoying.  I’ve sometimes found myself
>>> working around it, too (because I knew the server was going to have the
>>> store item sooner than 1h).
>>>
>>> Rather than removing it entirely, I can think of these options:
>>>
>>>   1. Reduce the default negative timeouts.
>>
>> I think reducing it is good, as you say, it's possible to override the
>> default from the server side. Just in case someone wants caching
>> behaviour, it might be worth keeping that functionality at least.
>
> OK, let’s do that.
>
>>>   2. Add an option to ‘guix publish’ (and to the Coordinator?) so they
>>>  send a ‘Cache-Control’ header with the chosen TTL on 404.  That
>>>  way, if the server operator doesn’t mind extra load, they can run
>>>  “guix publish --negative-ttl=0”.
>>
>> That sounds sensible. The Guix Build Coordinator doesn't do any serving,
>> that's left to something else like nginx. For the deployments I maintain
>> though, I don't think I'm setting the relevant headers, but I'll look at
>> changing that.
>
> Cool.
>
>> Going back to the %narinfo-transient-error-ttl, if I'm correct in saying
>> that it's not possible to override that, maybe that should also use the
>> relevant header value if set?
>
> Correct, ‘%narinfo-transient-error-ttl’ cannot be overridden.  We can
> halve it if you think that’s useful, thought when that happens, it means
> something’s wrong with the server (returning 500 or similar).
>
> I’ve sent patches to address this, lemme know what you think!

The patches you've sent look good.


signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-12 Thread Christopher Baines

Christopher Baines  writes:

> Andreas Enge  writes:
>
>> Hello Chris,
>>
>> Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
>>> I think there are some benefits for using the Guix Build Coordinator to
>>> build things for substitutes, and it would be good to work out how to
>>> get those benefits to users of Guix generally.
>>
>> my question is a bit tangential, but how can we get the substitutes that
>> the build coordinator puts on bayfront? I still have bayfront.guix.info
>> in my list of substitute servers, but does it use the same signing key
>> as before? It would be nice to document what a user should do. In my
>> opinion, a second build farm would be a useful thing, in case one server
>> goes down, or needs to be updated, for instance. (And then it allows for
>> reproducibility checks.)
>
> So, bayfront is currently serving substitutes build through the Guix
> Build Coordinator, the signing key is the same as it was before.
>
> It's only been building things at pace since yesterday though, so it'll
> be a few days at least until it's caught up.
>
> I agree that having multiple independent substitute providers would be
> nice, and getting bayfront working well is a step towards that.

Continuing on the subject of bayfront, it seems like the most feasible
way to get substitutes build through the Guix Build Coordinator to users
might be through it. At least there hasn't been any other leads yet from
this thread.

Luckily, from a thread ("Machine things") on guix-sysadmin [1] a number
of months ago, the suggestion of using the Guix Build Coordinator on
bayfront already came up, so it's actually been running for around a
month, and making progress building things for part of that.

1: https://lists.gnu.org/mailman/private/guix-sysadmin/2021-February/003412.html

Getting some benefit from the substitutes will require a few things
though, firstly getting things in order such that a good proportion and
range of substitutes are available, and secondly amending the default
configuration to include bayfront.

On the default configuration point, what are the prerequsites for that?
Beyond having some substitutes to offer, is there any particular
criteria to consider, perhaps about the machines involved?

On the subject of the machines involved, currently there's:

 - x86_64-linux + i686-linux
   - bayfront (coordinator, agent and serving substitutes)
   - harbourfront
   - milano-guix-1
 - aarch64-linux + armhf-linux
   - monokuma

I'd like to get some childhurd VMs running, they can go on the x86_64
machines, and hopefully there will be enough resources to keep up with
x86_64-linux, i686-linux, while doing some i586-gnu builds. If I have
some luck, I might also be able to get some powerpc64 hardware, and I do
have two Raspberry Pi 4's, but they're not running Guix as the OS yet,
so it would be good to work out if they're suitable, or better left to
other tasks.

There's a few intertangled things, but the main question is, if bayfront
can be a source of substitutes, what's the path to actually have that
benefit users?

signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-04 Thread Christopher Baines


Andreas Enge  writes:

> Hello Chris,
>
> Am Sat, May 01, 2021 at 07:56:05PM +0100 schrieb Christopher Baines:
>> I think there are some benefits for using the Guix Build Coordinator to
>> build things for substitutes, and it would be good to work out how to
>> get those benefits to users of Guix generally.
>
> my question is a bit tangential, but how can we get the substitutes that
> the build coordinator puts on bayfront? I still have bayfront.guix.info
> in my list of substitute servers, but does it use the same signing key
> as before? It would be nice to document what a user should do. In my
> opinion, a second build farm would be a useful thing, in case one server
> goes down, or needs to be updated, for instance. (And then it allows for
> reproducibility checks.)

So, bayfront is currently serving substitutes build through the Guix
Build Coordinator, the signing key is the same as it was before.

It's only been building things at pace since yesterday though, so it'll
be a few days at least until it's caught up.

I agree that having multiple independent substitute providers would be
nice, and getting bayfront working well is a step towards that.

> Relatedly, and this may already be documented: How do I know where the
> packages are downloaded from?
> $ guix package -u
> substitute: updating substitutes from 'https://bayfront.guix.info'... 100.0%
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>  imagemagick-6.9.12-4-doc  4.6MiB1.6MiB/s 00:01 [#
>  ]  32.5%
> Before the print format overhaul, the full URL would be printed.

Maybe you can turn on more detailed output? I'm unsure.

> Does the guix daemon complain when substitute URLs do not match any
> authorised key?

I don't think so, it'll just ignore them.


signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-04 Thread Christopher Baines


Ludovic Courtès  writes:

> Christopher Baines  skribis:
>
>> Ludovic Courtès  writes:
>
>> As people are going to Cuirass to look at the build status for packages,
>> system tests and various branches, the problem is similar to that of
>> substitutes. It doesn't matter if the Guix Build Coordinator seems to do
>> some things better, when the question comes up about whether something
>> has built yet, or whether a branch is ready to merge, Cuirass on
>> ci.guix.gnu.org is where people are going, and that seems unlikely to
>> change (at least less likely than the setup for providing substitutes).
>>
>> The Guix Data Service in the center, making it easier to do various
>> things by providing the data, this was the idea when it started, but the
>> reality recently is that strategy hasn't been paying off.
>
> The Data Service provides a wealth of information that’s underused!
>
> I think addressing the interface issue (be it web UI or JSON + CLI) is
> high priority so we can all start taking advantage of the Data Service.
> The current interface is generic enough that it allows you to see
> everything, from lint warnings to version changes to reproducitility
> rates.
>
> We could have interfaces that answer very specific needs:
>
>   • Which packages are broken on x86_64?

While the Guix Data Service can sort of do this (if it has builds data),
I think Cuirass has a page for that now, there's not a single URL for
the latest evaulation, but assuming you click on the latest one, this
page list the failed builds for x86_64:

  https://ci.guix.gnu.org/eval/30959/dashboard

>   • How does branch X compare to branch Y in terms of build failures?
>
>   • Which packages are not reproducible?
>
>   • Which packages are “flaky”?
>
> I know all this information is already available from the web UI, but
> because it’s generic, it can be hard to find out how to answer very
> concrete issues like this.
>
> A QA entry point like you proposed in the thread you mentioned¹ could
> certainly help.  A reproducibility entry point would be nice too.  A
> package browser for guix.gnu.org like the one Danjela worked on would be
> great too, possibly with version browsing facilities.  And Guix Weekly
> News!  And the security tracker!  :-)
>
> ¹ https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00096.html
>
> It seems to me that all that hard work is already done and what I
> describe above are rather low-hanging fruits.
>
> Taking Conway’s law into account, we may find it easier to recruit if as
> much as possible takes place here, things get deployed behind
> *.guix.gnu.org, and relevant bits are made part of Guix proper.  And
> also, we must regularly advertise progress; one blog post in all of the
> Guix Data Service’s lifetime is not enough.  :-)
>
> Thoughts?

This doesn't really relate to the subject of substitutes.

Some of the things you mention do relate to work I'm trying to progress
though. I'm still working on automated patch (plus branch) testing, and
I think having a simple overview of patch+branch states is hopefully
something that I'll get to at some point.

On the subject of the patch testing stuff, that isn't under
.guix.gnu.org and I haven't written a blog post yet. I can't see Cuirass
starting to test patches, but then I wouldn't have predicted it would be
managing builds across multiple machines. Maybe there are some risks
with the patch testing work that I haven't done enough/the right stuff
to mitigate.

I've also got too many things in progress at the moment, with the
combination of work on substitutes (I hope to implement things set out
in [1] some point soon), Outreachy, and the security related work that
I'm trying to start, I need to "finish" some things before starting new
ones or going back to unfinished things.

1: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00104.html


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

2021-05-03 Thread Christopher Baines


Luciana Lima Brito  writes:

> On Sun, 02 May 2021 10:20:56 +0100
> Christopher Baines  wrote:
>
>> I think what things are happening when is relevant, but that's more
>> about understanding the hierarchy, rather than specific start and end
>> times.
>
>> Basically, although using more colours (from a gradient, like white to
>> red) would probably convey more information than just two colours.
>
> I submitted my final application. I added to it some notes on what we
> have been discussing, but I think some specific things could be better
> discussed visually with a mock up.

Great :)

> Don't you think now would be a good time for some new task? :)

As the Outreachy contribution period has now ended, there's no specific
need.

If you really want something to look at, you could investigate the slow
package metadata related query that Canan was looking in to. There's
some details in this email [1] and then later on in that same thread as
well [2].

1: https://lists.gnu.org/archive/html/guix-devel/2021-04/msg00395.html
2: https://lists.gnu.org/archive/html/guix-devel/2021-04/msg00434.html


signature.asc
Description: PGP signature

Re: Bringing substitutes from the Guix Build Coordinator to users

2021-05-03 Thread Christopher Baines


Ludovic Courtès  writes:

> Thanks for your message!  It’s great to see a summary of what’s been
> cooking in the Coordinator and your vision around it.
>
> Christopher Baines  skribis:
>
>> More specifically, while the architecture is similar to daemon
>> offloading, there are some practical advantages. Since the switch to
>> Cuirass remote workers for ci.guix.gnu.org, the advantages of building
>> things across multiple machines in a coordinated manor have also become
>> clear. Additionally, there are things that the Guix Build Coordinator
>> enables, like automated retries, with the option to target specific
>> agents, which can help with avoiding issues with non-reproducible
>> failures, as well as helping to avoid issues when mixing QEMU emulated
>> and native builds.
>
> These are nice examples of current QA blind spots that would be nice to
> address.
>
> [...]
>
>> This is a topic I haven't got directly involved in, until now. I'm just
>> a volunteer, in some ways the most involved I am is that I host an idle
>> ARM build machine, I don't have any particular connection in the default
>> approach to substitutes for users, or the hardware currently
>> involved.
>
> It would be great to have that OverDrive plugged in behind ci.guix
> because we’re still short on CPU power for ARM.  (Not great we had
> forgotten about this one!)
>
> It could be reconfigured with a config similar to
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/overdrive1.scm>,
> with a different host name and IP.  We can discuss the details in a
> separate thread and/or on guix-sysadmin, let me know!

It was/is being discussed, the relevant thread is here:
  https://lists.gnu.org/mailman/private/guix-sysadmin/2021-March/003487.html

>> However, I think some of the stuff I've been working on could be
>> helpful, as I say, I think the Guix Build Coordinator is a step
>> forward in terms of building things for substitutes, and that shows in
>> the substitute availability percentages.
>>
>> Is there still a path to bring some of these benefits to users, and if
>> so, what things need doing?
>
> My understanding is that, to build substitutes, we need more than the
> Coordinator; we also need something to perform evaluations, similar to
> the “top half” of Cuirass and the Data Service, right?  How would you
> envision setting it up?

The queue builds script bundled in with the guix-build-coordinator seems
to work fine, that's how bayfront is currently set up:

  (service guix-build-coordinator-queue-builds-service-type
 (guix-build-coordinator-queue-builds-configuration
  (systems '("x86_64-linux"

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/bayfront.scm#n781

> To me, the way forward is not necessarily substitutes; it could be QA,
> as you showed that it has helpful features.  You already have access to
> bayfront and it’d be great to experiment there.  Do you have ideas on
> how to do make progress on that front?

Ideas yes, and the two things are interrelated, but my perspective at
the moment is that it'll be harder to try and get some value from the
Guix Build Coordinator in terms of quality assurance, as compared to
substitutes.

> Besides, I feel that the Data Service has a huge role to play with a
> variety of applications, including QA, but also way beyond as you showed
> in the blog post¹.  I think it’s time take advantage of it.  :-)
>
> Specifically, as far as QA is concerned, I think we should place the
> Data Service at the center of the stage.  It already does things that
> Cuirass cannot and will not do, notably: linting, challenging substitute
> servers to estimate reproducibility, and recording the history of all
> this.  Couldn’t it gather these other QA metrics mentioned above,
> possibly from a Coordinator-powered bayfront?

To me, linting and substitute comparison are less important quality
assurance things. I think the key issues are around testing packages and
things like system tests.

There's been some progress, albeit slow progress with the patch testing
setup I've been working on, but I've been viewing that as a way to test
branches as well. Given changes happen through patches and branches, I
think this makes sense, and I think the Guix Data Service has some nice
features for helping with this, like comparing two revisions and working
out what breakages have occurred.

However, with patches and branches, this is very much a continuous
integration tooling issue, much more than substitutes. Cuirass is
already building branches, and it's been gaining features previously
found in the Guix Data Service, like tracking the history of which
derivations relate to which revision, and separating

Re: [Outreachy] [Guix Data Service]: Identify the slow parts of process

2021-05-02 Thread Christopher Baines

Canan Talayhan  writes:

>>From this I'm guessing the temp_package_metadata table has only one
>>row. My understanding is that this table would normally have as many
>>rows as packages in the revision of Guix being processed. It might not
>>be possible to reproduce the slowness of the query without more rows.
>
> I've inserted one row just as an example. As you've already said,
> the temp_package_metadata table should have as many rows
> as package_metadata.

"as many rows as packages in the revision of Guix being processed" is
only going to be similar to the number of rows in the package_metadata
table if there's only been one or a number of similar revisions
processed, since the package_metadata table has entires covering all
processed revisions.

> After populated the temp_package_metadata with 500 rows of
> package_metadata, the query takes a long time as we expected.

Great, being able to reproduce the problem in a way that makes trying
things out easy is a good step forward.

I'd pull on this thread further, now you've got a slow query, how can
you make it faster?

> I'm using Flame Graph to visualize the slow paths on the revision part.
> At first, I choose the slow one that I already know.
> However, I can't successfully trigger the slow query following the below step:
>
> * Run the **guix-data-service-process-job** under guix-data-service/scripts
> folder as standalone providing an existing revision on my local db.
>
> Am I on the right path for adding new jobs log to my local db?
>
> In addition, I've successfully generated simple Flame Graph using Linux perf.
> It visualizes only the data that was captured while I'm browsing on the
> Guix Data Service Page. Please find the svg file attached.

If this relates to the query involving the temp_package_metadata table,
I'd focus on analyzing the slow query you're able to execute manually,
rather than processing an entire revision.

If you do however want to add more unprocessed jobs to your local
database, then you can use the
guix-data-service-process-branch-updated-mbox script to do this. It
takes one argument, an mbox file (file containing a bunch of
emails). You can download files by month from here [1], and you'll
probably want the month or next month on from the latest revision your
local database knows about.

1: https://lists.gnu.org/archive/mbox/guix-commits/

signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

2021-05-02 Thread Christopher Baines


Luciana Lima Brito  writes:

> On Sat, 01 May 2021 20:07:56 +0100
> Christopher Baines  wrote:
>
>> Luciana Lima Brito  writes:
>>
>> > For that I propose to build 2 charts, one of the
>> > macro view, what we call "overview first", showing the
>> > sections(processes) and their whole time taken. This way we could
>> > just see what we were aiming for, which is to identify slowness.
>> > The second chart would be what we call "details on demand", in
>> > which we could have the subsections(actions) being shown. To differ
>> > to which section(process) they are bound, we could use two
>> > meaningless alternating colours (just to group the subsections of a
>> > section), and they would follow the same order as the first chart.
>> >
>> > The use of alternating colours could be applied to both charts in
>> > order to make clear the equivalence. Both charts should appear at
>> > the same time, one above the other, to ease comparison.
>>
>> That sounds better, although I think a timeline, similar to what the
>> systemd-analyze example uses [1] might be a more natural
>> representation of the data, colour could then be used to represent
>> relatively how long each part takes.
>>
>> 1: https://lizards.opensuse.org/wp-content/uploads/2012/07/plot001.gif
>
> Here I have two observations to debate:
> 1 - Is the starting and ending time of each process an important
> information to determine its slowness? If this information is not
> necessary, maybe we should avoid the timeline, in order to make the
> chart cleaner. A timeline could impair the comparisons of bars, so I
> would recommend simple bar charts.

I think what things are happening when is relevant, but that's more
about understanding the hierarchy, rather than specific start and end
times.

> 2 - About the colours to represent how long each part takes, I don't
> know if I get it right. Do you mean to have one colour for slow parts
> and other colour to normal parts?

Basically, although using more colours (from a gradient, like white to
red) would probably convey more information than just two colours.


signature.asc
Description: PGP signature

Bringing substitutes from the Guix Build Coordinator to users

2021-05-01 Thread Christopher Baines

Hey,

The Guix Build Coordinator has been around for a while, it's been
available in Guix for more than 6 months now. The setup I've been using
to test the Guix Build Coordinator for building things for substitutes
(guix.cbaines.net) has been running since mid 2020.

Anecdotally, the test setup I've been running (guix.cbaines.net) has for
a while now, generally had higher percentage substitute availability
compared to ci.guix.gnu.org, as reported by guix weather. I think it
might also be building more things since cross derivations are built
plus i586-gnu builds.

I think there are some benefits for using the Guix Build Coordinator to
build things for substitutes, and it would be good to work out how to
get those benefits to users of Guix generally.

More specifically, while the architecture is similar to daemon
offloading, there are some practical advantages. Since the switch to
Cuirass remote workers for ci.guix.gnu.org, the advantages of building
things across multiple machines in a coordinated manor have also become
clear. Additionally, there are things that the Guix Build Coordinator
enables, like automated retries, with the option to target specific
agents, which can help with avoiding issues with non-reproducible
failures, as well as helping to avoid issues when mixing QEMU emulated
and native builds.

Going back to discussions in 2020, I think there was a path for starting
to experiment with the Guix Build Coordinator, trying it on a few
machines in the berlin build farm, but as I say, this was last year and
before any work on implementing similar functionality in Cuirass as far
as I'm aware. Since then, I've also got an instance of the Guix Build
Coordinator running on bayfront with a couple of extra machines also
involved for performing builds, but this is just starting off, and
doesn't bring any benefit in terms of substitutes, at least not yet.

This is a small part of the wider puzzle, when I was designing the Guix
Build Coordinator, it was meant for much more than building things for
substitutes, and those things are also happening. The Guix Build
Coordinator has enabled things like building packages affected by
patches, building fixed output derivations regularly, and will hopefully
enable a whole load of quality assurance things. But my hope was also to
try and tackle building things for substitutes, such that there's not
more software to maintain, and also to inform future work on the
guix-daemon itself, like whether it would be good for it to have an
asynchronous API, and how that might work.

This is a topic I haven't got directly involved in, until now. I'm just
a volunteer, in some ways the most involved I am is that I host an idle
ARM build machine, I don't have any particular connection in the default
approach to substitutes for users, or the hardware currently
involved. However, I think some of the stuff I've been working on could
be helpful, as I say, I think the Guix Build Coordinator is a step
forward in terms of building things for substitutes, and that shows in
the substitute availability percentages.

Is there still a path to bring some of these benefits to users, and if
so, what things need doing?

Thanks,

Chris


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

2021-05-01 Thread Christopher Baines


Luciana Lima Brito  writes:

> On Sat, 01 May 2021 09:16:08 +0100
> Christopher Baines  wrote:
>
>> Currently the timing of various sections of the process includes
>> timing smaller sections, and that may complicate reading the chart,
>> since it won't convey which timed sections include other timed
>> sections. Does that make sense?
>
> Yes, I understand. But just to make sure, you say that the actions we
> see in the logs are actually subsections of a bigger process? The
> problem here would be to clearly mark in the code actions of a same
> process. I'll take this into account on my planning.

Take the lint warnings for example, currently the time to fetch all lint
warnings is timed, but the usage of each individual linter is
timed. Both bits of information are helpful.

> For that I propose to build 2 charts, one of the
> macro view, what we call "overview first", showing the
> sections(processes) and their whole time taken. This way we could just
> see what we were aiming for, which is to identify slowness.
> The second chart would be what we call "details on demand", in which we
> could have the subsections(actions) being shown. To differ to which
> section(process) they are bound, we could use two meaningless
> alternating colours (just to group the subsections of a section), and
> they would follow the same order as the first chart.
>
> The use of alternating colours could be applied to both charts in order
> to make clear the equivalence. Both charts should appear at the same
> time, one above the other, to ease comparison.

That sounds better, although I think a timeline, similar to what the
systemd-analyze example uses [1] might be a more natural representation
of the data, colour could then be used to represent relatively how long
each part takes.

1: https://lizards.opensuse.org/wp-content/uploads/2012/07/plot001.gif

>> Great, this is a good amount of detail.
>
> I'll add this to the plan and to the final application, ok?

Yep.


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

2021-05-01 Thread Christopher Baines


Luciana Lima Brito  writes:

> On Fri, 30 Apr 2021 18:05:15 +0100
> Christopher Baines  wrote:
>
>> >
>> > Task 1: Add instrumentation to identify the slow parts of processing
>> > new revisions:
>> >
>> >   - Implementing a chart over time to identify slow parts:
>> >- The chart should consider two aspects, the time took by
>> >  each specific part, and its name, for identification
>> >   purpose. A bar chart is a good candidate for this task, it is
>> >   simple and can show the whole picture of the time taken by the
>> >  system to process a new revision;
>> >- The bars on the chart should be sorted in order of
>> >   precedence of the process in the X axis, and its height, which
>> >   is determined by the time it takes, measured in the Y
>> >  axis;
>>
>> I'm not sure what you mean by "precedence of the process" here?
>
> As we are concerned only with the time each process takes, and each bar
> on the chart will be presented side by side, the order of the bars would
> be the order that the processes appear. If you prefer, they could be
> sorted alphabetically, or by time taken.

Ok, I think I follow.

Currently the timing of various sections of the process includes timing
smaller sections, and that may complicate reading the chart, since it
won't convey which timed sections include other timed sections. Does
that make sense?

>> >- The charts should work as picture-logs for timing;
>> >- A chart should be generated for each new revision in real
>> >  time; 
>> 
>> It would be good to say explicitly how the chart will be stored, since
>> "the same way as the logs already are" is quite vague.
>
> I misunderstood this point so this part about storing the chart is
> completely messed. But I think now I got it. Let me correct the last
> point and add some more:
>   ...
>   - The time is already being computed for each part and it
> is shown in the logs, the same data can be used to build the charts.
> The data to build the chart for each revision could be
> stored as a new table in the database*, first in parts, for
> when a revision is still being processed, then combining their
> parts when the processing is finished.
>   - The new table on the database should store three information:
> the job_id, the action, and the time taken.
>   - Then, when one wants to see the chart, this table is queried
> and the chart is rendered as html.
>
> * Although the information is already in the log, it is stored as a
>   text, so it is harder to get the names of the actions and the time
>   taken by each, so I think that create a new table, with only these
>   values, is more suitable.

Great, this is a good amount of detail.


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

2021-04-30 Thread Christopher Baines


Luciana Lima Brito  writes:

> Hi,
>
> On Thu, 29 Apr 2021 21:14:10 +0100
> Christopher Baines  wrote:
>
>> Great, can you add more detail to this bit? Given the instrumentation
>> is a really important part, it would be good to have some working
>> ideas for what this chart might look like, and what goes in to making
>> it (like where the data comes from and if and where it's stored).
>
> Task 1: Add instrumentation to identify the slow parts of processing
> new revisions:
>
>   - Implementing a chart over time to identify slow parts:
>   - The chart should consider two aspects, the time took by
> each specific part, and its name, for identification purpose.
> A bar chart is a good candidate for this task, it is simple
> and can show the whole picture of the time taken by the
> system to process a new revision;
>   - The bars on the chart should be sorted in order of precedence
> of the process in the X axis, and its height, which is
> determined by the time it takes, measured in the Y
> axis;

I'm not sure what you mean by "precedence of the process" here?

>   - The charts should work as picture-logs for timing;
>   - A chart should be generated for each new revision in real
> time;
>   - The time is already being computed for each part and it is
> shown in the logs, the same data can be used to build the
> charts. This way data does not need to be stored, because the
> chart can be built in real time, but the chart itself needs
> to be stored, in the same way as the logs already are.

It would be good to say explicitly how the chart will be stored, since
"the same way as the logs already are" is quite vague.

>> I'd try to set out more of a strategy, so what might be causes of
>> slowness, how would you investigate them, and what are common
>> approaches to making those things faster?
>
> Task 2: Improve the performance of these slow parts:
>
>   - Select candidate parts to be analyzed;
>   - Identify the causes of slowness:
>   - If it uses queries, perform query analysis, for example using
> EXPLAIN and ANALYZE, get its statistics and improve them when
> needed;
>   - On the code, investigate the structure, for example, using
> Tracing to discover if a recursion is too long and if it can
> be modified to be tail recursive.
>   - Implement the required improvements.
>
> In fact, I have never performed improvements on queries or code this
> way, but I am studying. Please, tell me if I am missing important
> details. See what you think about that.

Great, this is more like it.


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

2021-04-29 Thread Christopher Baines

lubr...@posteo.net writes:

> Hi,
>
> On Wed, 28 Apr 2021 21:00:44 +0100
> Christopher Baines  wrote:
>
>
>> I think time is the main thing, since that alone will help to identify
>> which are the slow parts.
>
>> Indeed, simplicity is a good thing to keep in mind.
>
> I'll keep in mind what you have said, so here are some ideas to
> break the tasks.
>
> Task 1: Add instrumentation to identify the slow parts of processing
>   new revisions
>
>   - Implementing a chart over time to identify slow parts;

Great, can you add more detail to this bit? Given the instrumentation is
a really important part, it would be good to have some working ideas for
what this chart might look like, and what goes in to making it (like
where the data comes from and if and where it's stored).

>   - Select candidate parts to be analyzed;
>   - Should this part really be taking all this time?
> - Identify if the delay is caused by the code itself or by a
>   query.
>   - Prepare a document organizing where each part should
> be improved.
>
Given these don't contribute to the instrumentation, I would see them as
part of "Task 2".

> Task 2: Improve the performance of these slow parts
>
>   - implementing each improvements.
>
> This Task 2 is really difficult for me to divide in smaller tasks, I'm
> really stuck with ideas.

I'd try to set out more of a strategy, so what might be causes of
slowness, how would you investigate them, and what are common approaches
to making those things faster?

> I took a look at one of the logs as you told me, and I found this parts
> taking a considerable time:

Great :)

> - Computing the channel derivations 636s
> - Acquiring advisory transaction lock:
>   loading-new-guix-revision-inserts 2534s *
> - Getting derivation lint warnings 975s
> - fetching inferior lint warnings 1488s
> - Acquiring advisory transaction lock:
>   loading-new-guix-revision-inserts 3226s *
> - Querying the temp-package-metadata 1949s
> - Fetching inferior package metadata 2476s
>
> To pick this parts is what should be facilitated by the chart?
> So I would perform the analysis of the task 1 and then the ta

I would hope that the "instrumentation" would make finding out what
parts are slowest easier, and making it visual information rather than
text scattered through the long log will hopefully go a long way towards
that.

> By the way, I noticed that the ones I marked with * are performed twice.
> Why is that?

I'd try reading the code to find out. In case you haven't worked with
database locks much before, they are being used in this case to allow
multiple processes to interact with the database to load different
revisions at the same time. The locking is very coarse, so locks can be
held for a very long time, so what you're seeing in the log when it's
saying it's acquiring a lock, and it takes a long time, is that process
was waiting for another process to commit or rollback a transaction in
which it was holding that same lock.

The technical details that are useful to know:

 - The locks here apply within transactions

 - The lock will be released when the transaction commits or rolls back

 - Advisory locks are user defined locks:
 https://www.postgresql.org/docs/13/explicit-locking.html#ADVISORY-LOCKS

signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?


Ludovic Courtès  writes:

> Hi!
>
> Here’s the installation ISO image (with built-in zlib compression):
>
> --8<---cut here---start->8---
> $ ./pre-inst-env guix system image -t iso9660 gnu/system/install.scm
> /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
> $ xz < /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso > /tmp/t.iso.xz
> $ du -h /tmp/t.iso.xz
> 496M  /tmp/t.iso.xz
> $ du -h /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
> 647M  /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
> --8<---cut here---end--->8---
>
> The xz-compressed image is 23% smaller, which is not negligible.
> However, it’s apparently quite unusual to distribute compressed ISOs and
> some services/tools such as libosinfo require plain ISOs (uncompressed).
>
> Should we distribute the installation ISO without xz compression?

Personally I think the ISO is more useful if it's published as such,
rather than an as a xz compressed file.

I believe this will make things like getting Guix through Gnome Boxes
easier, and using Guix at various hosting providers easier.


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks


Luciana Lima Brito  writes:

> On Wed, 28 Apr 2021 19:17:51 +0100
> Christopher Baines  wrote:
>
>> So, there's already some code for timing different parts of the data
>> loading process, if you look in the job output and search for ", took
>> " you should see timings printed out.
>>
>> These timings being printed out does help, but having the information
>> in the log doesn't make it easy to figure out which part is the
>> slowest for example.
>>
>> I'd also not consider this a "one off" thing, the data loading code
>> will continue to change as Guix changes and it's performance will
>> probably change too.
>>
>> I've been wondering about visualisations, I remember systemd had a
>> feature to plot the systems boot as a image which made seeing which
>> parts are slow much easier (here's an example [1]).
>>
>> 1: https://lizards.opensuse.org/wp-content/uploads/2012/07/plot001.gif
>
> This is interesting! In fact, one of the things that attracted me was
> the possibility to work with visualizations, as I saw that on the
> roadmap there is one task related to provide statistics over time).
> My master degree is on Information Visualization, so I would appreciate
> very much if I could help with that.
>
> In this matter, we should determine what else, other than time, would
> be interesting to see. The visualization should be clear enough about
> timing but should also provide information about what could be related
> to the delays, such as size of the queries, complexity, the return it
> gives... So, first I think we should determine what information we want
> to see, then depending on the variables, we choose a suitable way to
> present the visualization.

I think time is the main thing, since that alone will help to identify
which are the slow parts.

> About implementing, I'm kind of new to guile and I never built a
> visualization in guile, so I don't know which libraries it would take to
> build a visual work like that. Depending on what we have,
> interactions could be compromised, and instead we would have to work
> with charts (static visualizations). Can you tell me more about that?

Given the Guix Data Service outputs HTML as well as JSON, it might be
possible to build something with HTML. The package history pages sort of
visualise the data by adding some grey bars to the table [1].

1: http://data.guix.gnu.org/repository/1/branch/master/package/guix

> And one last thing, a visualization can be simple or can be very
> complex.The time for that should be carefully taken into account in
> order to not impair the main goal which is the improvements of the slow
> parts.

Indeed, simplicity is a good thing to keep in mind.

>> > About the improvements on the performance of slow parts, it is a
>> > little bit abstract for me to see now how to break it in smaller
>> > tasks. I do believe that it would require to reformulate some parts
>> > of the queries, and as their result may change a bit, tweaks could
>> > be required on the code too. My point is, how would I propose an
>> > improvement approach if I don't even know what exactly is to be
>> > improved? But I imagine that work on this second task is more
>> > demanding than the first and will take most of the time of the
>> > internship.
>>
>> As I said before, this part is dependent on deciding where the areas
>> for improvement are. Maybe have a look through one of the job logs on
>> data.guix.gnu.org and see if you can spot some slow parts?
>
> I'll look into that and get back to you.

Great.


signature.asc
Description: PGP signature

Re: Outreachy: Timeline tasks

Luciana Lima Brito  writes:

> I was thinking about the timeline of tasks.
>
> The main tasks are:
>
> 1. Add instrumentation to identify the slow parts of processing
>   new revisions
>
> 2. Improve the performance of these slow parts
>
> I'm writing some ideas I have to divide the tasks in small steps, see
> what you think about it.
>
> About the first task I understand that the whole thing starts with
> identifying how the data for new revisions arrives on Guix Data
> Service: the relevant queries and their processing on the code. Based
> on it I would propose start with mapping these queries and their uses,
> so I could run them locally and get their statistics.
>
> Once I get this information I could identify which are the possible
> problematic ones and work on them. If the process is slow but the query
> is not, maybe the problem would be hidden in the code.

So, there's already some code for timing different parts of the data
loading process, if you look in the job output and search for ", took "
you should see timings printed out.

These timings being printed out does help, but having the information in
the log doesn't make it easy to figure out which part is the slowest for
example.

I'd also not consider this a "one off" thing, the data loading code will
continue to change as Guix changes and it's performance will probably
change too.

I've been wondering about visualisations, I remember systemd had a
feature to plot the systems boot as a image which made seeing which
parts are slow much easier (here's an example [1]).

1: https://lizards.opensuse.org/wp-content/uploads/2012/07/plot001.gif

> About the improvements on the performance of slow parts, it is a little
> bit abstract for me to see now how to break it in smaller tasks. I do
> believe that it would require to reformulate some parts of the queries,
> and as their result may change a bit, tweaks could be required on
> the code too. My point is, how would I propose an improvement approach
> if I don't even know what exactly is to be improved? But I imagine that
> work on this second task is more demanding than the first and will take
> most of the time of the internship.

As I said before, this part is dependent on deciding where the areas for
improvement are. Maybe have a look through one of the job logs on
data.guix.gnu.org and see if you can spot some slow parts?

signature.asc
Description: PGP signature

Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons.

Luciana Lima Brito  writes:

> Hi,
>
> On Tue, 27 Apr 2021 21:29:35 +0100
> Christopher Baines  wrote:
>
>
>> Great :) I've tweaked the commit message a little and pushed.
>
> This is really great!!!
>
> I started to look after my final application and I need to see two
> specific things with you to finish filling the forms.
>
> 1 - There is a field in which I should provide more information, if
> required by the community or project's mentor. Are there any questions
> I should answer?

Nope, you can leave that bit.

> 2 - About the timeline: "work with your mentor to provide a timeline of
> the work you plan to accomplish on the project and what tasks you will
> finish at each step".

This is relevant though. I'd say the important bit of what's above is
the breakdown of the work (the tasks) as you see it, rather than any
exact dates in the timeline.

So, have another read of the project description and the main tasks as I
set out. Now is the time to discuss if that approach makes sense, and
once you're happy with the direction, start to split tasks down in to
more concrete parts.

I realise the performance improvements will be dependent on the
discovery work, so this more applies to the approach to work out what
bits are slow. I can try and help with this, but it's not me who's going
to be doing the work, so it's you who needs to be happy with the
proposed approach.

> another thing, what I should be doing now for my next contribution?

Given the final application deadline is quite close, I'd look at that
first.

Chris

signature.asc
Description: PGP signature

Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons.


Luciana Lima Brito  writes:

> On Tue, 27 Apr 2021 19:42:13 +0100
> Christopher Baines  wrote:
>
>> I'd go with this approach, applying the comments I made about the
>> match-lambda bit above in the email I sent a few minutes ago.
>
> I applied all you instructed me. See if it looks better now.

Great :) I've tweaked the commit message a little and pushed.

Chris


signature.asc
Description: PGP signature

Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons.


Luciana Lima Brito  writes:

> On Tue, 27 Apr 2021 13:10:01 +
> Luciana Lima Brito  wrote:
>
>> > Maybe add another procedure that combines group-to-alist but
>> > generates an alist with vectors as the values?
>> > (group-to-alist/vector maybe).
>
> I did it! :)
> It was so much simpler. I created a function
> group-to-alist/vector, based on the previous function, to which I added
> the map I was already using on the controller.scm for data-groups.
>
> (define (group-to-alist/vector process lst)
>   (map
>(match-lambda
>  ((label args ...)
>   `(,label . ,(list->vector args
>(group-to-alist process lst)))
>
> The only change needed on the html.scm is to use vector->list in the
> items, like this
>
> (map
>  (lambda (alist)
>...
>  (or (vector->list items) '()))
>
> Please, tell me what you think is the best, this way or the other that
> I sent earlier with the patch?

Great :)

I'd go with this approach, applying the comments I made about the
match-lambda bit above in the email I sent a few minutes ago.


signature.asc
Description: PGP signature

Re: [Outreachy] [Guix Data Service]: Identify the slow parts of process

Canan Talayhan  writes:

> I am writing to give you an update on the progress that I have made.

Great :)

> I've created a temporary table named temp_package_metadata[1] and
> insert a revision that already in my local database[2]. Then as you
> said I've run the slow query with EXPLAIN ANALYZE. (screenshot is
> attached) I may understand the slow query's working logic.
>
> [1]CREATE TEMPORARY TABLE temp_package_metadata (LIKE package_metadata
> INCLUDING ALL)
>
> [2]INSERT INTO temp_package_metadata (home_page,
> location_id,license_set_id,package_description_set_id,
> package_synopsis_set_id) VALUES ('https://zlib.net/',9,9,2373,1407)

From this I'm guessing the temp_package_metadata table has only one
row. My understanding is that this table would normally have as many
rows as packages in the revision of Guix being processed. It might not
be possible to reproduce the slowness of the query without more rows.

signature.asc
Description: PGP signature

Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons.


Luciana Lima Brito  writes:

> On Mon, 26 Apr 2021 22:21:50 +0100
> Christopher Baines  wrote:
>
>>
>> Rather than writing:
>>
>>   (match-lambda
>> ((alist ...)
>>
>> I'd just use
>>
>>   (lambda (alist)
>>
>> as I think that's equivalent right?
>
> Right, I did this.

Great. Unfortunately I missed a few things similar to this when I last
looked.

For this bit:

  (match-lambda
((label args ...)
 `(,label . ,(list->vector args

Given the thing being matched is a pair in an alist, I'd use (label
. args). I'd also perhaps use cons rather than `(, . ,) for creating the
pair.

>> >> I'd consider these options first probably:
>> >>
>> >>  - Could the data coming from derivation-differences-data have
>> >> vectors where appropriate already? The HTML code would probably
>> >> need to be adjusted, but I think that's fine.
>> >
>> > I tried this for days but with no success. Maybe the only way would
>> > be to tweak group-to-alist, but it touches many places, and I
>> > didn't want to mess with it.
>>
>> Maybe add another procedure that combines group-to-alist but generates
>> an alist with vectors as the values? (group-to-alist/vector maybe).
>
>> I think using let is OK, but I think just unpacking data-groups as
>> you've called it directly in to the alist is fine (so ,@data-groups),
>> rather than picking out the elements. JSON objects are unordered, so
>> the ordering isn't something that really matters.
>>
>> If you do go down this route though, I'd probably add a comment saying
>> what things are being added to the outer most alist, just to make the
>> code quicker to read.
>
> Well, I went down the second route, now I'm calling the ,@data-groups
> and I added a comment explaining its use.
> The main point here is, the code is working and it looks nice, but to
> get the data with the vectors seems to be right too. I'm sending the
> new patch for your review and I'll wait for your call, if you think I
> should try the first route or not.

This looks OK now.

The other thing I forgot to mention last time was using empty objects
(() in the code) for the hash and hash-algorithm for derivations where
those aren't applicable. I'd probably suggest using the symbol null, or
not including those fields in the object.

Hopefully that's all the bits to fix up, apologies for not mentioning
them last time.

Chris


signature.asc
Description: PGP signature

Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons.

2021-04-26 Thread Christopher Baines

Luciana Lima Brito  writes:

> On Mon, 26 Apr 2021 09:15:37 +0100
> Christopher Baines  wrote:
>
>> So, one advantage of alists over lists is that the code is probably
>> less brittle when adding elements say, since code parsing the list
>> will probably break with a new element, but this is probably less
>> likely to happen with an alist.
>>
>> However, this will happen with an alist if match is used to pick
>> elements out. I'd suggest using assq-ref or similar to pluck elements
>> out.
>
> Ok, I changed that on the html.scm.

Great :)

Rather than writing:

  (match-lambda
((alist ...)

I'd just use

  (lambda (alist)

as I think that's equivalent right?

>> I'd consider these options first probably:
>>
>>  - Could the data coming from derivation-differences-data have vectors
>>where appropriate already? The HTML code would probably need to be
>>adjusted, but I think that's fine.
>
> I tried this for days but with no success. Maybe the only way would be
> to tweak group-to-alist, but it touches many places, and I didn't want
> to mess with it.

Maybe add another procedure that combines group-to-alist but generates
an alist with vectors as the values? (group-to-alist/vector maybe).

>>  - Could this be written in a form like:
>>
>>,@(map (lambda (name)
>>...)
>>   '(outputs inputs sources arguments))
>
> This only make sense to me inside render-json (because of the ,@), but I
> think the code would be less clean and "arguments" would appear in a
> different order. What I did was bind the result of a function similar
> to this in the let.

I think using let is OK, but I think just unpacking data-groups as
you've called it directly in to the alist is fine (so ,@data-groups),
rather than picking out the elements. JSON objects are unordered, so the
ordering isn't something that really matters.

If you do go down this route though, I'd probably add a comment saying
what things are being added to the outer most alist, just to make the
code quicker to read.

signature.asc
Description: PGP signature

Re: Outreachy - Guix Data Service: questions about improving the data for derivation comparisons.

2021-04-26 Thread Christopher Baines


Luciana Lima Brito  writes:

> Your advices helped me think more clearly.

Great :)

> There was no need to create or modify structures other than what I was
> already changing. I now return an alist instead of a list on the
> derivation-differences-* functions on comparison.scm (for outputs,
> inputs and sources). It helped to simplify the mapping on
> controller.scm. The changes on html.scm were minimal, basically it is
> matching on pairs, instead of single values.
>
> Two questions:
>
> 1 - The match on the html expects 5 values for "outputs", so I had to
> settle on using empty objects on the JSON, when needed, else it would
> break the match on the html. Is it ok?

So, one advantage of alists over lists is that the code is probably less
brittle when adding elements say, since code parsing the list will
probably break with a new element, but this is probably less likely to
happen with an alist.

However, this will happen with an alist if match is used to pick
elements out. I'd suggest using assq-ref or similar to pluck elements
out.

> 2 - Now on controller.scm "outputs", "inputs", "sources", and even
> "arguments" have the same structure, which is an alist of the form:
>
>   ((base . (...))
>(target . (...))
>(common . (...)))
>
> and I'm using the same map and match-lambda code to process them,
> wouldn't it be reasonable now to make it a local function?

I'd consider these options first probably:

 - Could the data coming from derivation-differences-data have vectors
   where appropriate already? The HTML code would probably need to be
   adjusted, but I think that's fine.

 - Could this be written in a form like:

   ,@(map (lambda (name)
   ...)
  '(outputs inputs sources arguments))


signature.asc
Description: PGP signature

Re: [Outreachy] - Guix Data Service - Set a more informative page title


Canan Talayhan  writes:

> Thanks for your quick response. It helps a lot to me. But still, I
> have some confusion about the reproduction steps. As I understand it,
> I can reproduce the slow query just using the pure SQL queries without
> touching the code for now, right?
>
> Please find my steps below:
>
> 1. CREATE TEMPORARY TABLE temp_package_metadata (LIKE package_metadata
> INCLUDING ALL)
>
> As we expected, it creates a table but has no data. So I want to
> insert some data but colon types are integer and I don't know any
> meaningful data for these colons.

So, the question to ask here is what data would the temporary table
usually contain during loading data for a revision (when the slow query
happens). I gave more specific information about the data in the
temporary table in my previous email.

> 2. CREATE TEMPORARY TABLE temp_package_metadata AS TABLE package_metadata
>
> Then I create a copy of the package_metadata.
>
> 3. EXPLAIN ANALYZE SELECT * FROM temp_package_metadata
> This code generates 155 msec execution time, but I think it should
> take more time.

I don't see why that query should be slower, but that's also not that
relevant since I think the actual slow query in question here is quite
different.

You can find the query in the code by looking at where it does the
timing for "querying the temp_package_metadata".

I've also got information from the PostgreSQL logs, which gives this as
the query:

  SELECT package_metadata.id, package_metadata.home_page, 
package_metadata.location_id, package_metadata.license_set_id, 
package_metadata.package_description_set_id, 
package_metadata.package_synopsis_set_id FROM package_metadata INNER JOIN 
temp_package_metadata ON (package_metadata.home_page = 
temp_package_metadata.home_page OR (package_metadata.home_page IS NULL AND 
temp_package_metadata.home_page IS NULL)) AND (package_metadata.location_id = 
temp_package_metadata.location_id OR (package_metadata.location_id IS NULL AND 
temp_package_metadata.location_id IS NULL)) AND 
(package_metadata.license_set_id = temp_package_metadata.license_set_id OR 
(package_metadata.license_set_id IS NULL AND 
temp_package_metadata.license_set_id IS NULL)) AND 
(package_metadata.package_description_set_id = 
temp_package_metadata.package_description_set_id OR 
(package_metadata.package_description_set_id IS NULL AND 
temp_package_metadata.package_description_set_id IS NULL)) AND 
(package_metadata.package_synopsis_set_id = 
temp_package_metadata.package_synopsis_set_id OR 
(package_metadata.package_synopsis_set_id IS NULL AND 
temp_package_metadata.package_synopsis_set_id IS NULL))


signature.asc
Description: PGP signature

Re: [Outreachy] - Guix Data Service - Set a more informative page title

Christopher Baines  writes:

> The approach I'd recommend is, make yourself a realistic
> temp_package_metadata table by populating it with all the
> package_metadata entries for a single revision already in your local
> database. Then construct and try the slow query, and see how long it
> takes, and look at the query plan (run the query with EXPLAIN at the
> start).

Following up on your question on IRC about creating the temporary table,
the code that does this is here [1]. I wouldn't suggest running the
code, just the same SQL it uses, and use relevant values for
temp-table-name and table-name that are appropriate for the
package_metadata table.

1: 
https://git.savannah.gnu.org/cgit/guix/data-service.git/tree/guix-data-service/model/utils.scm#n313

signature.asc
Description: PGP signature

Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

Vincent Legoll  writes:

> Hello,
>
> On Sat, Apr 24, 2021 at 8:28 AM Christopher Baines  wrote:
>> With some prompting, there's now a blog post about the Guix Build
>> Coordinator
>
> Nice post that explains a lot, but I'm still not so sure about the
> relations to cuirass. I'd have liked a small paragraph explaining
> what the differences are, how can they complement each other,
> kind of the CI envisionned big picture...

Thanks for the feedback Vincent :)

I did think about trying to include something about Cuirass, but I don't
have a clear picture of it's scope or purpose, so I'm not really the
right person to attempt to write authoritatively about it.

On a technical level, there's no connection, although there was talk
over the last 6 months or so about trying to have or allow Cuirass to
benefit from the Guix Build Coordinator's ability to perform builds in a
methodical manor across multiple machines. That hasn't happened yet
though, and in the mean time, Cuirass has gained it's own mechanism of
running builds on other machines.

I try to avoid using the CI (Continuous Integration) term as I'm not
sure there's a shared understanding of the term (I think it's the
practice of multiple people frequently merging their changes to some
software they're all working on).

In terms of building things for substitutes, that's one of the use cases
I had in mind when I designed the Guix Build Coordinator, and I think
that design has worked out well, so I'm still interested in trying to
get some benefits for Guix users through using the Guix Build
Coordinator to produce substitutes in a faster and more reliable way.

Does that make any sense? Do say if you have more questions.

Thanks,

Chris

signature.asc
Description: PGP signature

Update on using the Guix Build Coordinator for substitutes