Re: Another update on the Guix Data Service
Ludovic Courtès writes: > Christopher Baines skribis: > >> There's now a page to compare two derivations [4], it's not a >> particularly clear comparison yet, but can hopefully be improved in the >> future. This is linked to from the derivation history pages [5]. >> >> 4: >> http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv >> 5: >> http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history > > I didn’t know about this and it’s really cool! It’s often necessary to > compare derivations when you want to understand what’s going on. > > Do you think there’s derivation comparison code out there that could > make it into (guix derivations) or similar, with an eye on perhaps > having a CLI derivation comparison tool using the same code in the > future? I'm not sure any of the Guix Data Service code is worth moving across, but it might be worth trying to make something better that uses the records in Guix, and then using that in the Guix Data Service. >> Provide narinfo and nar files for derivations, and the required source >> files ([12] for example). This means the Guix Data Service can be used >> as a substitute server for derivations, for example [13]. >> >> 12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo >> 13: guix build --substitute-urls="http://data.guix.gnu.org >> https://ci.guix.gnu.org; >> /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv > > Well done. :-) > > I’m interested in making sure we can share the relevant code between > (guix scripts publish) and the Data Service if that’s not already the > case. I don't think I had any big problems duplicating code in (guix scripts publish), however, it might be worth looking at perhaps trying to identify similarities if there are cases where duplication can be avoided. >> In particular, I'm still thinking about; >> >> - Automated code review for Guix patches, which I was working on when I >>started working on the Guix Data Service, and most of the >>functionality can be beneficial there. > > Yeah, I think at the Guix Days you showed how to take advantage of the > Data Service while reviewing patches; perhaps we should write down a > tutorial or something. Yep, I'm definately hoping it's getting close to the point where it can be useful. >> - The Guix Weekly News project I was working on recently uses data from >>the Guix Data Service, and that's what motivated storing the channel >>news entries. > > Yup, that looked nice! What’s missing, actually? I haven't sorted anything out to automatically update the data, and "publish" the news articles. I also need to improve the large lists of packages, as that isn't very useful. signature.asc Description: PGP signature
Re: Another update on the Guix Data Service
Hi! I’m late to the party and learned about most of these things in the meantime at the Guix Days, and this is all very cool! Christopher Baines skribis: > There's now a page to compare two derivations [4], it's not a > particularly clear comparison yet, but can hopefully be improved in the > future. This is linked to from the derivation history pages [5]. > > 4: > http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv > 5: > http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history I didn’t know about this and it’s really cool! It’s often necessary to compare derivations when you want to understand what’s going on. Do you think there’s derivation comparison code out there that could make it into (guix derivations) or similar, with an eye on perhaps having a CLI derivation comparison tool using the same code in the future? > Provide narinfo and nar files for derivations, and the required source > files ([12] for example). This means the Guix Data Service can be used > as a substitute server for derivations, for example [13]. > > 12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo > 13: guix build --substitute-urls="http://data.guix.gnu.org > https://ci.guix.gnu.org; > /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv Well done. :-) I’m interested in making sure we can share the relevant code between (guix scripts publish) and the Data Service if that’s not already the case. > In particular, I'm still thinking about; > > - Automated code review for Guix patches, which I was working on when I >started working on the Guix Data Service, and most of the >functionality can be beneficial there. Yeah, I think at the Guix Days you showed how to take advantage of the Data Service while reviewing patches; perhaps we should write down a tutorial or something. > - The Guix Weekly News project I was working on recently uses data from >the Guix Data Service, and that's what motivated storing the channel >news entries. Yup, that looked nice! What’s missing, actually? > - Trust in builds, once the data about package build reproducibility is >more complete, hopefully that will be informative and show where >build reproducibility issues can be fixed. > > - Improving the speed in which substitutes are made available. > > - Explaining the data that underpins Guix to a technical audience, like >derivations and how they work. Nice! Thanks, Ludo’.
Re: Another update on the Guix Data Service
Pierre Neidhardt writes: > Christopher Baines writes: > >> Is there a specific thing you're interested in/unsure about? > > Well, mostly the patch continuous integration that you mentioned. Cool, I'm hoping to have something more concrete by the upcoming Guix Days! > Regarding the documentation, it'd be nice to explain _what_ can be done > with Guix Data Service, since it is rather large already. Indeed, I'll try to get around to improving this. Thanks, Chris signature.asc Description: PGP signature
Re: Another update on the Guix Data Service
Hi Christopher, Thanks for the details! Christopher Baines writes: > Is there a specific thing you're interested in/unsure about? Well, mostly the patch continuous integration that you mentioned. Regarding the documentation, it'd be nice to explain _what_ can be done with Guix Data Service, since it is rather large already. Good luck! -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Re: Another update on the Guix Data Service
Pierre Neidhardt writes: >> Channel news entries are recorded in the database, you can see all that >> are present in a revision [3], but the comparison page should show new >> news entries as well. >> >> 3: >> http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news > > Is there a link to the news page from > > http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299 > > ? No, but there probably should be one. >> - Automated code review for Guix patches, which I was working on when I >>started working on the Guix Data Service, and most of the >>functionality can be beneficial there. > > Some time ago you did some work on Patchwork. > > https://patchwork.cbaines.net/project/guix-patches/list/ > > I haven't followed very closely, so I wonder what your plans are > regarding patchwork and Guix Data Service. So I'd like to resurrect my slightly clunky but somewhat working Patchwork + Laminar + Git repo + Guix Data Service (+ Cuirass?) setup for testing patches. Here, the Guix Data Service would process the Guix revision with the patches under test having been applied. This would reveal the lint warning changes. It would also reveal the new derivations, which can be built to check what the effect of the patches are. > Last, is there any documentation on how to _use_ it? (I don't mean the > setup.) So the only documentation currently is the README file, which is available at http://data.guix.gnu.org/README . I haven't considered writing standalone documentation yet, but I have been considering incorporating more in to the web interface. Is there a specific thing you're interested in/unsure about? Chris signature.asc Description: PGP signature
Re: Another update on the Guix Data Service
Hi Christopher! Nice update as usual! A few questions: > Channel news entries are recorded in the database, you can see all that > are present in a revision [3], but the comparison page should show new > news entries as well. > > 3: > http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news Is there a link to the news page from http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299 ? > - Automated code review for Guix patches, which I was working on when I >started working on the Guix Data Service, and most of the >functionality can be beneficial there. Some time ago you did some work on Patchwork. https://patchwork.cbaines.net/project/guix-patches/list/ I haven't followed very closely, so I wonder what your plans are regarding patchwork and Guix Data Service. Last, is there any documentation on how to _use_ it? (I don't mean the setup.) Cheers, and thanks again with the great work! -- Pierre Neidhardt https://ambrevar.xyz/ signature.asc Description: PGP signature
Another update on the Guix Data Service
Hey, It's been a while since I send out an update about the Guix Data Service, so here's a somewhat failed attempt at a short update about recent progress. There are formatted [1] and plain views of derivations [2]. The plain view is constructed from the representation in the database, but it should match what Guix produces. 1: http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/formatted 2: http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/plain Channel news entries are recorded in the database, you can see all that are present in a revision [3], but the comparison page should show new news entries as well. 3: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news There's now a page to compare two derivations [4], it's not a particularly clear comparison yet, but can hopefully be improved in the future. This is linked to from the derivation history pages [5]. 4: http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv 5: http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history There's support for receiving build information from build servers, but this hasn't been used yet. I'm slowly looking at getting support for using this in to Cuirass [6] 6: https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00467.html Information about nars (substitutes) can now be stored ([7] for example). This data also shows up on a page for derivation outputs for a revision ([8] for example). 7: http://data.guix.gnu.org/gnu/store/001j0wk7srm7jlnd9lai7p1yzg4bdl3k-dunst-1.4.1 8: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/derivation-outputs There's a page for builds for a revision [9]. There's also a page for individual builds [10]. 9: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/builds 10: http://data.guix.gnu.org/build-server/2/build?derivation_file_name=/gnu/store/yc0pb7c0c198nb68c5wha41brzvrls01-k3b-19.08.3.drv The derivation outputs page for a revision shows information about outputs matching across build servers, but there's also a page about package reproducibility for a revision [11]. 11: http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/package-reproducibility Provide narinfo and nar files for derivations, and the required source files ([12] for example). This means the Guix Data Service can be used as a substitute server for derivations, for example [13]. 12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo 13: guix build --substitute-urls="http://data.guix.gnu.org https://ci.guix.gnu.org; /gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv Fix generating derivations for foreign architectures. There was an issue where only the x86_64-linux and i686-linux derivations were properly loaded, but now that's been fixed. I added support for loading mbox files a while ago, but I've only now got around to testing a few older revisions, and now I've loaded the mbox files for the first 3 months from 2019 so eventually those will be loaded in. A few things above were influenced by thinking about reproducible builds in the run up, during and after the reproducible builds summit at the start of December. I'm excited that there's some support in the Guix Data Service for comparing builds across build farms, but there's still a lot to do in terms of getting every derivation built by at least 2 separate build farms, and then getting that data promptly in to the Guix Data Service for comparison. Also at the reproducible builds summit, Ludovic made me aware that you can tell Guix to build a derivation, and if it doesn't have it locally, it will attempt to download it from substitute servers. I've now got the Guix Data Service setup to act as a substitute server for derivations, which while on it's own isn't particularly exciting (as build farms already can provide substitutes for derivations), it's now possible to get the information about derivations within the Guix Data Service, out of the Guix Data Service in a usable manor. This might have applications in the future, like speeding up guix pull by providing the derivation that corresponds to a revision, or providing derivations for a build farm to build. Like the example of providing substitutes for derivations, quite a lot of this work isn't directly impactful (at least yet), but I'm hoping it's laid the foundations for some more directly impactful stuff. In particular, I'm still thinking about; - Automated code review for Guix patches, which I was working on when I started working on the Guix Data Service, and most of the functionality can be beneficial there. - The Guix Weekly News project I was working on recently uses data from the