Re: A script to check an edit does not break anything
Hi Edouard, Edouard Klein skribis: > Ludovic Courtès writes: [...] >> However, this is definitely something ‘guix lint’ could check with >> something along the lines of the patch below. > > Thank you for pushing profile-collisions, it certainly is helpful for > finding such problems, and it perfectly integrates within guix. > >> >>> - Secondly, it does not limit itself to the dependents (as listed by >>> guix refresh --list-dependents) of the packages one is meddling with, >>> but to the whole reverse bags (as listed by guix graph >>> --type=reverse-bag). >> >> I think it’s equivalent: ‘guix refresh -l’ simply shows the contour of >> the graph whereas ‘guix graph’ lists every node. > > The problem lies when the leafs are OK but the nodes in the middle are > not. See for example in the attached image, the failure of jupyter is > masked by r-irkernel being both buildable and installable. When you write “the failure of jupyter”, you mean failure to _install_ jupyter, right? ‘guix refresh -l’ returns the set of leaves whose closure encompasses all the dependents of the given package. Thus, if you build all the nodes returned by ‘guix refresh -l jupyter’, you’ll definitely notice a build failure in jupyter. > Now, the new tool you added to guix lint solves the discoverability > problem. It is now indeed reported that jupyter has a problem. > > Still, it takes around 10 minutes to run on my (admittedly underpowered) > machine, and one has to rummage to the output (or diff with a previous > run) to see if a specific action caused or solved problems. What takes 10 minutes? ‘guix lint -c profile-collisions’ without arguments? It runs in 2 minutes on my laptop, which is admittedly too much, but note that most of the time you’ll just run: guix lint -c profile-collisions jupyter That should take a few seconds at most. > gpwc.sh has a real time visual output that is specific to current > modifications (it could even be paired with Ricardo's automatic commit > message writer to automatically guess which root packages to start with) > that allows the developer to start investing a problem quicker, without > having to wait for the end of the run. Also, the visual output makes > seeing who depends on whom easier, the same information in text form > makes my head hurt. > > Provided I rewrite it in scheme, do you think gpwc could make it > into guix/etc ? I haven’t looked in detail at gwpc.sh but I guess we’d all like improved tooling. To have it in Guix, it’s better if it’s well integrated with existing tools and written in Scheme. Thanks! Ludo’.
Re: A script to check an edit does not break anything
Hi Edouard, Edouard Klein skribis: > Because of the subtle consequences of editing a package are hard to keep > track of, I wrote the attached script (you need sharness to run it, but > with slight modification it can become a standalone script). > > This script goes beyond the instructions of the manual on two fronts: > - First, it not only tries to build the packages, but also to install > them, I think this is probably the first time we have this problem (that I remember of), probably because the Jupyter dependency graph has so many propagated inputs. However, this is definitely something ‘guix lint’ could check with something along the lines of the patch below. > - Secondly, it does not limit itself to the dependents (as listed by > guix refresh --list-dependents) of the packages one is meddling with, > but to the whole reverse bags (as listed by guix graph > --type=reverse-bag). I think it’s equivalent: ‘guix refresh -l’ simply shows the contour of the graph whereas ‘guix graph’ lists every node. Thanks for your feedback! Ludo’. diff --git a/guix/lint.scm b/guix/lint.scm index 82861b8a27..84ddfd6b73 100644 --- a/guix/lint.scm +++ b/guix/lint.scm @@ -41,6 +41,7 @@ #:use-module (guix upstream) #:use-module (guix utils) #:use-module (guix memoization) + #:use-module (guix profiles) #:use-module (guix scripts) #:use-module ((guix ui) #:select (texi->plain-text fill-paragraph)) #:use-module (guix gnu-maintenance) @@ -84,6 +85,7 @@ check-for-updates check-formatting check-archival +check-profile-collisions lint-warning lint-warning? @@ -970,6 +972,20 @@ descriptions maintained upstream." (with-store store (check-with-store store +(define* (check-profile-collisions package #:key store) + (guard (c ((profile-collision-error? c) + (let ((first (profile-collision-error-entry c)) + (second (profile-collision-error-conflict c))) + (list (make-warning package + (G_ "collision between ~a@~a and ~a@~a") + (list (manifest-entry-name first) + (manifest-entry-version first) + (manifest-entry-name second) + (manifest-entry-version second))) +(check-for-collisions (packages->manifest (list package)) + (%current-system)) +'())) + (define (check-license package) "Warn about type errors of the 'license' field of PACKAGE." (match (package-license package) @@ -1349,6 +1365,11 @@ or a list thereof") (description "Report failure to compile a package to a derivation") (check check-derivation) (requires-store? #t)) + (lint-checker + (name'profile-collisions) + (description "Report collisions that would occur due to propagated inputs") + (check check-profile-collisions) + (requires-store? #t)) (lint-checker (name'patch-file-names) (description "Validate file names and availability of patches") diff --git a/guix/profiles.scm b/guix/profiles.scm index 25ff146bdf..56ef4c0b91 100644 --- a/guix/profiles.scm +++ b/guix/profiles.scm @@ -105,6 +105,7 @@ manifest-installed? manifest-matching-entries manifest-search-paths +check-for-collisions manifest-transaction manifest-transaction? diff --git a/guix/ui.scm b/guix/ui.scm index 98b30445c8..2595e44062 100644
Re: A script to check an edit does not break anything
Hello Edouard,
I think you left a hardcoded package name in your script, line 29, fix
with:
s/python-websockets/$package/
Also you sometimes use $package vs ${package}
This kind of thing (maybe converted into scheme what about
`guix before-submit') should be a valuable addition to guix,
it would help beginners (like me do less mistakes) and comitters
could have more confidence in a submission if it has gone through
this. Which would in turn enable us to integrate patches quicker.
WDYT ?
--
Vincent Legoll
Re: A script to check an edit does not break anything
An now with the attachment.
#!/bin/bash
# Put this script in a subfolder (e.g. gpwc)
# of the guix source
# And install sharness in it
# https://github.com/chriscool/sharness
# Install prove as well
# https://linux.die.net/man/1/prove
# To run the tests, invoke
# prove gpwc.t
# Fill in the names of the packages you've been meddling with
# In my case it was python-prompt-toolkit
PACKAGES=(python-prompt-toolkit python-ipywidgets)
test_description="Checking my meddling with ${PACKAGES[*]} did not break anything"
. ./sharness.sh
# Check that the new packages themselves are OK
for package in "${PACKAGES[@]}"
do
test_expect_success "Build $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env guix build $package
"
test_expect_success "Lint $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env guix lint $package
"
test_expect_success "See if $package is reproducible" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env guix build --rounds=2 python-websockets
"
test_expect_success "Install $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env \
guix install --profile=./${package}-tmp-profile ${package}
"
done
# Check the consequences of having meddled with the new packages
dependents(){
guix environment guix --pure -- "${SHARNESS_BUILD_DIRECTORY}"/pre-inst-env \
guix graph --type=reverse-bag "$1" | \
grep @ | sed 's/.*\"\(.*\)@.*/\1/'
}
# Union of all the dependents
for package in "${PACKAGES[@]}"
do
dependents "$package"
done | sort -u > dependents.txt
for package in $(cat dependents.txt)
do
test_expect_success "Build dependent $package" "
guix environment guix --pure -- \"${SHARNESS_BUILD_DIRECTORY}\"/pre-inst-env guix build $package
"
profile="$SHARNESS_TRASH_DIRECTORY/$(echo $package | sed 's/\(.*\)@/\1'/)-tmp-profile"
test_expect_success "Install dependent $package" "
guix environment guix --pure -- ${SHARNESS_BUILD_DIRECTORY}/pre-inst-env \
guix install --profile=\"${profile}\" ${package}
"
done
test_done
Edouard Klein writes:
> Dear Guixers,
>
> I recently broke jupyter while updating python-prompt-toolkit (a mistake
> I'm still trying to fix...) despite doing my best to follow the
> instructions of the manual:
> https://guix.gnu.org/manual/en/html_node/Packaging-Guidelines.html
> https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches
>
> Because of the subtle consequences of editing a package are hard to keep
> track of, I wrote the attached script (you need sharness to run it, but
> with slight modification it can become a standalone script).
>
> This script goes beyond the instructions of the manual on two fronts:
> - First, it not only tries to build the packages, but also to install
> them,
> - Secondly, it does not limit itself to the dependents (as listed by
> guix refresh --list-dependents) of the packages one is meddling with,
> but to the whole reverse bags (as listed by guix graph
> --type=reverse-bag).
>
> Both these extensions are necessary to discover that my update broke the
> jupyter package:
> - jupyter builds, so if you don't try to install it you won't discover
> it's broken
> - the leafs of the dependency graph that depend on jupyter fail to build
> for reasons unrelated to python-prompt-toolkit, so by just building the
> leafs (which are what is being returned by guix refresh
> --list-dependents) you can't discover jupyter's broken status.
>
> I think an update of the manual is in order (I've added it to my own
> TODO queue, but I have no idea when I'll be able to get to it) but I
> first wanted to share my script in case it may be useful to someone
> else, and gather some feedback as to whether other developers have
> encountered these kind of errors, and the way seasoned developers make
> sure they don't push breaking changes.
>
> Cheers,
>
> Edouard.
A script to check an edit does not break anything
Dear Guixers, I recently broke jupyter while updating python-prompt-toolkit (a mistake I'm still trying to fix...) despite doing my best to follow the instructions of the manual: https://guix.gnu.org/manual/en/html_node/Packaging-Guidelines.html https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches Because of the subtle consequences of editing a package are hard to keep track of, I wrote the attached script (you need sharness to run it, but with slight modification it can become a standalone script). This script goes beyond the instructions of the manual on two fronts: - First, it not only tries to build the packages, but also to install them, - Secondly, it does not limit itself to the dependents (as listed by guix refresh --list-dependents) of the packages one is meddling with, but to the whole reverse bags (as listed by guix graph --type=reverse-bag). Both these extensions are necessary to discover that my update broke the jupyter package: - jupyter builds, so if you don't try to install it you won't discover it's broken - the leafs of the dependency graph that depend on jupyter fail to build for reasons unrelated to python-prompt-toolkit, so by just building the leafs (which are what is being returned by guix refresh --list-dependents) you can't discover jupyter's broken status. I think an update of the manual is in order (I've added it to my own TODO queue, but I have no idea when I'll be able to get to it) but I first wanted to share my script in case it may be useful to someone else, and gather some feedback as to whether other developers have encountered these kind of errors, and the way seasoned developers make sure they don't push breaking changes. Cheers, Edouard.
