Re: 04/09: gnu: mesa: Update to 23.0.3.
John Kehayias writes: > Hi Christopher and Maxim, > > On Mon, May 08, 2023 at 02:41 AM, Christopher Baines wrote: > >> guix-comm...@gnu.org writes: >> >>> apteryx pushed a commit to branch master >>> in repository guix. >>> >>> commit 0be7838105806819f4586ec9130382a66a22880e >>> Author: Kaelyn Takata >>> AuthorDate: Thu May 4 20:12:46 2023 + >>> >>> gnu: mesa: Update to 23.0.3. >>> >>> * gnu/packages/gl.scm (mesa): Update to 23.0.3. >>> [source]: Remove obsolete patch and update HTTPS url. >>> [arguments]: Enable the crocus gallium driver. >>> * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete >>> file. >>> * gnu/local.mk (dist_patch_DATA): Remove it. >>> --- >>> gnu/local.mk | 1 - >>> gnu/packages/gl.scm| 14 --- >>> .../patches/mesa-fix-sporadic-test-failures.patch | 27 >>> -- >>> 3 files changed, 5 insertions(+), 37 deletions(-) >> >> → guix refresh -l mesa >> Building the following 1954 packages would ensure 4257 dependent >> packages are rebuilt ... >> >> >> I know there's been some discussion about changing processes regarding >> changes like this that impact lots of packages, but as far as I'm aware, >> the documented process hasn't changed yet. So should this have gone to >> core-updates, and not been directly pushed to master? >> > > I should take some responsibility over what happened here as I had > volunteered as a sort of "branch manager" (to use the terminology > later in this thread) but I was a bit slower than I wanted. My plan > was roughly in line with what we are discussing, reviewing the > patches, pushing to a new "mesa-updates" branch, and then asking for > someone to start a build job. We could then have people more easily > test or just to check build for build failures and ensure good > substitute coverage upon merging. Don't feel responsible for not doing things John, I'm advocating here for less haste and more caution. signature.asc Description: PGP signature
Re: 04/09: gnu: mesa: Update to 23.0.3.
Hi Christopher and Maxim, On Mon, May 08, 2023 at 02:41 AM, Christopher Baines wrote: > guix-comm...@gnu.org writes: > >> apteryx pushed a commit to branch master >> in repository guix. >> >> commit 0be7838105806819f4586ec9130382a66a22880e >> Author: Kaelyn Takata >> AuthorDate: Thu May 4 20:12:46 2023 + >> >> gnu: mesa: Update to 23.0.3. >> >> * gnu/packages/gl.scm (mesa): Update to 23.0.3. >> [source]: Remove obsolete patch and update HTTPS url. >> [arguments]: Enable the crocus gallium driver. >> * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete >> file. >> * gnu/local.mk (dist_patch_DATA): Remove it. >> --- >> gnu/local.mk | 1 - >> gnu/packages/gl.scm| 14 --- >> .../patches/mesa-fix-sporadic-test-failures.patch | 27 >> -- >> 3 files changed, 5 insertions(+), 37 deletions(-) > > → guix refresh -l mesa > Building the following 1954 packages would ensure 4257 dependent > packages are rebuilt ... > > > I know there's been some discussion about changing processes regarding > changes like this that impact lots of packages, but as far as I'm aware, > the documented process hasn't changed yet. So should this have gone to > core-updates, and not been directly pushed to master? > I should take some responsibility over what happened here as I had volunteered as a sort of "branch manager" (to use the terminology later in this thread) but I was a bit slower than I wanted. My plan was roughly in line with what we are discussing, reviewing the patches, pushing to a new "mesa-updates" branch, and then asking for someone to start a build job. We could then have people more easily test or just to check build for build failures and ensure good substitute coverage upon merging. Anyway, I appreciate this coming up for discussion here, so thanks Christopher. And thanks Maxim for pushing through these changes anyway, as there were some important fixes for people (some couldn't use wayland I think due to missing Mesa drivers, video decoding acceleration didn't work, etc.) and selfishly I'm happy to have the latest Mesa sooner. My apologies for not helping lead more on this front after my public volunteering to do so! John
Re: 04/09: gnu: mesa: Update to 23.0.3.
Maxim Cournoyer writes: >>> Seeing the build machines were idling in the European night, I figured I >>> could get away with it for this time. >> >> Some build machines may have been idle, but I'm not sure what you mean >> by "get away with it"? > > I meant that I believed there was enough capacity to process the 4K > rebuilds (per architecture) in a way that wouldn't negatively affect > users too much. That may well be the case, but I see problems using this as justification for similar actions in the future. Even if it's unlikely that people use mesa or it's dependencies on systems other than x86_64-linux and i686-linux, this still adds to the backlog of builds for other architectures. Also, while the berlin build farm may be able to build things very quickly for these systems, I think it's good to try and reduce the churn where possible by batching together changes (aka staging/core-updates). Not doing so and just pushing when the build farm can cope will generate more data to store, more for users to download and more to build for people who don't use substitutes. >> While the berlin bulid farm has managed to catch back up for >> x86_64-linux and i686-linux within 24 hours, I think these changes >> impact other systems as well. >> >> Also, the bordeaux build farm has a lot fewer machines to do these >> builds, so while the substitute availability had caught up (and >> surpassed ci.guix.gnu.org for x86_64-linux) prior to these changes, it's >> going to be several days at least I think before substitute availability >> is looking good again. >> >> I was watching the substitute availability recover after the >> core-updates merge as I'd like to re-enable testing patches on the >> qa-frontpage, but now that'll have to wait some more for all these new >> builds to complete. > > Hm, sorry about that. Cuirass seems to have mostly caught up already > (was 64% before, 62% now for the master specification). I think this is a problematic metric to use. Cuirass should be building for aarch64-linux, but substitute availability sits below 20% for that system. Even though the bordeaux build farm has less machines, it has 70%+ substitute availability. So for aarch64-linux for the berlin build farm, I think these builds have just been added to the long backlog. This metric doesn't articulate how the situation has got worse in this way. (also, I don't know what this number means, but if it's something like substitute availability, ~60% seems pretty bad) >>> But the situation will repeat; I'd like to push some xorg updates that >>> fix a CVE; we'll nead a 'xorg-team' branch or similar. Should we create >>> these branches from the maintenance repository (permanent branches) ? >> >> I don't really understand the question, surely the branches would be in >> the guix Git repository? > > Yes, the branch would be in the Guix repository, but I meant regarding > the Cuirass specifications affecting which branches it builds; sorry for > being unclear. No problem, this is something that needs looking at. On the berlin build farm side, yes, configuring Cuirass to look at the branches is one thing, but there's also bigger issues, e.g. the lack of substitutes for aarch64-linux and armhf-linux (and thus delays in testing/merging branches due to this). On the bordeaux build farm side, there's also a "how to start and stop building a branch issue". Currently it's a code change in the qa-frontpage (plus reconfiguring bayfront and restarting the service), but it would be good to make this easier too. Plus, like the berlin build farm, there's also issues getting things built fast enough. >> Anyway, package replacements+grafts can be used for security issues so >> that shouldn't need to be on a branch as it won't involve lots of >> rebuilds. > > For this case I think so yes, since it's a patch-level update that > should be safe. > >> When it comes to handling changes involving lots of rebuilds though, I >> think that this has been and continues to be difficult, but in my mind >> that's a reason to slow down and try and work on tooling and processes >> to help. > > One of the things that has been bothered me has been the lack of > documentation/tooling to recreate TLS user certificates for Cuirass so > that I can configure branches via its web interface again, or retry > failed builds. I'm currently working on documenting (in Cuirass's > manual) a script Ricardo's made for that task. > > But building lots of packages will still require a lot of processing > power, probably more so when it happens in focused team branches > compared to grouped together as it used to be the case for > e.g. core-updates. I agree. I'm still not really sold on the idea of team specific branches for this reason. Anyway, I think there's still tooling to build (e.g. analyzing the overlap for builds between two changes) and more thinking to do about processes for this. signature.asc Description: PGP signature
Feature branches (was: 04/09: gnu: mesa: Update to 23.0.3)
Hello, indeed someone™ should update the documentation to describe the new process. Probably we should agree on one before doing that as well... In principle all big updates should go through a feature branch now. However, this does not solve the problem of limited build power in our two build farms. Having feature branches that regroup several related changes should help in not rebuilding too often. In principle it could also be okay to regroup unrelated changes (mesa and gcc, for instance), as long as responsibilities are clear. It should be known who is going to act on what when breakages occur in the branch. And I think there should be some kind of "branch manager" and a time frame for the merge so that the branches are short lived. The goal is to avoid the core-updates experience of random commits being dropped in the same place, while hoping that someone at some later point will sort it all out. So how about this suggestion: A feature branch is created upon request by a team, with a branch manager designated by the team. It is accompanied by a short description of what the branch is supposed to achieve, and in which timeframe. The branch manager has the responsibility to communicate regularly with guix-devel on the state of the branch and on what remains to be done, and requests to merge the branch to master once it is ready, and to subsequently delete the branch. This does not yet explain how the branches interact with continuous integration. The branch manager may or may not have commit rights and may or may not be able to create specifications for cuirass, so the full process should take this into account. As written in a different thread, right now I would also suggest to first build the branch only on x86 and powerpc to not overload our few arm machines, but this is a technical detail. Andreas
Re: 04/09: gnu: mesa: Update to 23.0.3.
Hi Christopher, Christopher Baines writes: > Maxim Cournoyer writes: > >> Christopher Baines writes: >> >>> guix-comm...@gnu.org writes: >>> apteryx pushed a commit to branch master in repository guix. commit 0be7838105806819f4586ec9130382a66a22880e Author: Kaelyn Takata AuthorDate: Thu May 4 20:12:46 2023 + gnu: mesa: Update to 23.0.3. * gnu/packages/gl.scm (mesa): Update to 23.0.3. [source]: Remove obsolete patch and update HTTPS url. [arguments]: Enable the crocus gallium driver. * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file. * gnu/local.mk (dist_patch_DATA): Remove it. --- gnu/local.mk | 1 - gnu/packages/gl.scm| 14 --- .../patches/mesa-fix-sporadic-test-failures.patch | 27 -- 3 files changed, 5 insertions(+), 37 deletions(-) >>> >>> → guix refresh -l mesa >>> Building the following 1954 packages would ensure 4257 dependent >>> packages are rebuilt ... >>> >>> >>> I know there's been some discussion about changing processes regarding >>> changes like this that impact lots of packages, but as far as I'm aware, >>> the documented process hasn't changed yet. So should this have gone to >>> core-updates, and not been directly pushed to master? >> >> There isn't currently a core-updates branch, and I need to spend some >> time documenting the authorization process for how to create short lived >> Cuirass branches. I think ideally we would have created a >> 'graphics-team' or similar branch (even the team has yet to be formed) >> and let it build. >> >> Seeing the build machines were idling in the European night, I figured I >> could get away with it for this time. > > Some build machines may have been idle, but I'm not sure what you mean > by "get away with it"? I meant that I believed there was enough capacity to process the 4K rebuilds (per architecture) in a way that wouldn't negatively affect users too much. > While the berlin bulid farm has managed to catch back up for > x86_64-linux and i686-linux within 24 hours, I think these changes > impact other systems as well. > > Also, the bordeaux build farm has a lot fewer machines to do these > builds, so while the substitute availability had caught up (and > surpassed ci.guix.gnu.org for x86_64-linux) prior to these changes, it's > going to be several days at least I think before substitute availability > is looking good again. > > I was watching the substitute availability recover after the > core-updates merge as I'd like to re-enable testing patches on the > qa-frontpage, but now that'll have to wait some more for all these new > builds to complete. Hm, sorry about that. Cuirass seems to have mostly caught up already (was 64% before, 62% now for the master specification). >> But the situation will repeat; I'd like to push some xorg updates that >> fix a CVE; we'll nead a 'xorg-team' branch or similar. Should we create >> these branches from the maintenance repository (permanent branches) ? > > I don't really understand the question, surely the branches would be in > the guix Git repository? Yes, the branch would be in the Guix repository, but I meant regarding the Cuirass specifications affecting which branches it builds; sorry for being unclear. > Anyway, package replacements+grafts can be used for security issues so > that shouldn't need to be on a branch as it won't involve lots of > rebuilds. For this case I think so yes, since it's a patch-level update that should be safe. > When it comes to handling changes involving lots of rebuilds though, I > think that this has been and continues to be difficult, but in my mind > that's a reason to slow down and try and work on tooling and processes > to help. One of the things that has been bothered me has been the lack of documentation/tooling to recreate TLS user certificates for Cuirass so that I can configure branches via its web interface again, or retry failed builds. I'm currently working on documenting (in Cuirass's manual) a script Ricardo's made for that task. But building lots of packages will still require a lot of processing power, probably more so when it happens in focused team branches compared to grouped together as it used to be the case for e.g. core-updates. -- Thanks, Maxim
Re: 04/09: gnu: mesa: Update to 23.0.3.
Maxim Cournoyer writes: > Christopher Baines writes: > >> guix-comm...@gnu.org writes: >> >>> apteryx pushed a commit to branch master >>> in repository guix. >>> >>> commit 0be7838105806819f4586ec9130382a66a22880e >>> Author: Kaelyn Takata >>> AuthorDate: Thu May 4 20:12:46 2023 + >>> >>> gnu: mesa: Update to 23.0.3. >>> >>> * gnu/packages/gl.scm (mesa): Update to 23.0.3. >>> [source]: Remove obsolete patch and update HTTPS url. >>> [arguments]: Enable the crocus gallium driver. >>> * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete >>> file. >>> * gnu/local.mk (dist_patch_DATA): Remove it. >>> --- >>> gnu/local.mk | 1 - >>> gnu/packages/gl.scm| 14 --- >>> .../patches/mesa-fix-sporadic-test-failures.patch | 27 >>> -- >>> 3 files changed, 5 insertions(+), 37 deletions(-) >> >> → guix refresh -l mesa >> Building the following 1954 packages would ensure 4257 dependent >> packages are rebuilt ... >> >> >> I know there's been some discussion about changing processes regarding >> changes like this that impact lots of packages, but as far as I'm aware, >> the documented process hasn't changed yet. So should this have gone to >> core-updates, and not been directly pushed to master? > > There isn't currently a core-updates branch, and I need to spend some > time documenting the authorization process for how to create short lived > Cuirass branches. I think ideally we would have created a > 'graphics-team' or similar branch (even the team has yet to be formed) > and let it build. > > Seeing the build machines were idling in the European night, I figured I > could get away with it for this time. Some build machines may have been idle, but I'm not sure what you mean by "get away with it"? While the berlin bulid farm has managed to catch back up for x86_64-linux and i686-linux within 24 hours, I think these changes impact other systems as well. Also, the bordeaux build farm has a lot fewer machines to do these builds, so while the substitute availability had caught up (and surpassed ci.guix.gnu.org for x86_64-linux) prior to these changes, it's going to be several days at least I think before substitute availability is looking good again. I was watching the substitute availability recover after the core-updates merge as I'd like to re-enable testing patches on the qa-frontpage, but now that'll have to wait some more for all these new builds to complete. > But the situation will repeat; I'd like to push some xorg updates that > fix a CVE; we'll nead a 'xorg-team' branch or similar. Should we create > these branches from the maintenance repository (permanent branches) ? I don't really understand the question, surely the branches would be in the guix Git repository? Anyway, package replacements+grafts can be used for security issues so that shouldn't need to be on a branch as it won't involve lots of rebuilds. When it comes to handling changes involving lots of rebuilds though, I think that this has been and continues to be difficult, but in my mind that's a reason to slow down and try and work on tooling and processes to help. signature.asc Description: PGP signature
Re: 04/09: gnu: mesa: Update to 23.0.3.
Hi Christopher, Christopher Baines writes: > guix-comm...@gnu.org writes: > >> apteryx pushed a commit to branch master >> in repository guix. >> >> commit 0be7838105806819f4586ec9130382a66a22880e >> Author: Kaelyn Takata >> AuthorDate: Thu May 4 20:12:46 2023 + >> >> gnu: mesa: Update to 23.0.3. >> >> * gnu/packages/gl.scm (mesa): Update to 23.0.3. >> [source]: Remove obsolete patch and update HTTPS url. >> [arguments]: Enable the crocus gallium driver. >> * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete >> file. >> * gnu/local.mk (dist_patch_DATA): Remove it. >> --- >> gnu/local.mk | 1 - >> gnu/packages/gl.scm| 14 --- >> .../patches/mesa-fix-sporadic-test-failures.patch | 27 >> -- >> 3 files changed, 5 insertions(+), 37 deletions(-) > > → guix refresh -l mesa > Building the following 1954 packages would ensure 4257 dependent > packages are rebuilt ... > > > I know there's been some discussion about changing processes regarding > changes like this that impact lots of packages, but as far as I'm aware, > the documented process hasn't changed yet. So should this have gone to > core-updates, and not been directly pushed to master? There isn't currently a core-updates branch, and I need to spend some time documenting the authorization process for how to create short lived Cuirass branches. I think ideally we would have created a 'graphics-team' or similar branch (even the team has yet to be formed) and let it build. Seeing the build machines were idling in the European night, I figured I could get away with it for this time. But the situation will repeat; I'd like to push some xorg updates that fix a CVE; we'll nead a 'xorg-team' branch or similar. Should we create these branches from the maintenance repository (permanent branches) ? -- Thanks, Maxim
Re: 04/09: gnu: mesa: Update to 23.0.3.
guix-comm...@gnu.org writes: > apteryx pushed a commit to branch master > in repository guix. > > commit 0be7838105806819f4586ec9130382a66a22880e > Author: Kaelyn Takata > AuthorDate: Thu May 4 20:12:46 2023 + > > gnu: mesa: Update to 23.0.3. > > * gnu/packages/gl.scm (mesa): Update to 23.0.3. > [source]: Remove obsolete patch and update HTTPS url. > [arguments]: Enable the crocus gallium driver. > * gnu/packages/patches/mesa-fix-sporadic-test-failures.patch: Delete file. > * gnu/local.mk (dist_patch_DATA): Remove it. > --- > gnu/local.mk | 1 - > gnu/packages/gl.scm| 14 --- > .../patches/mesa-fix-sporadic-test-failures.patch | 27 > -- > 3 files changed, 5 insertions(+), 37 deletions(-) → guix refresh -l mesa Building the following 1954 packages would ensure 4257 dependent packages are rebuilt ... I know there's been some discussion about changing processes regarding changes like this that impact lots of packages, but as far as I'm aware, the documented process hasn't changed yet. So should this have gone to core-updates, and not been directly pushed to master? signature.asc Description: PGP signature
