Re: Re: Need a buildd build after trip through NEW -- best practice?
On Thu, 2022-09-01 at 08:45 -0500, Steven Robbins wrote: > OK, so let's call it "bin nmu", then. And add the "+bN" version > suffix. As others said, binNMUs exist, but not for arch:all packages. Debian doesn't yet have sourceful no-change rebuilds, so you have to manually do a sourceful no-change upload (or a regular upload). Ubuntu does tho. dch supports the --rebuild option for this, but only on Ubuntu systems. > This would clearly be optimal. I'm just asking about an additional / > alternative mechanism. Yeah, we need sourceful no-change rebuilds in addition to NEW discarding. The first step would be updating dch to support this, after that the infrastructure can be updated, although I guess the infra could reimplement what `dch --rebuild` does instead. This still needs a volunteer to implement it, so far we have none to do that. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Re: Need a buildd build after trip through NEW -- best practice?
On Thu, Sep 01, 2022 at 08:45:06AM -0500, Steven Robbins wrote: > > > Specficially: in the case of a NEW binary upload, could a manual request > > > be > > > implemented (pick a different name if "give back" is not suitable) such > > > that it is thrown away and replaced by a buildd build? > > > > If you are suggesting the ability for dak to replace binaries already > > in the archive with different content without a new source version, > > then I don't think that should be implemented for Debian at least, > > since our archive is meant to only contain immutable package files. > > OK, so let's call it "bin nmu", then. And add the "+bN" version suffix. Have you seen and ? (and ) -- WBR, wRAR signature.asc Description: PGP signature
Re: Re: Need a buildd build after trip through NEW -- best practice?
Paul Said: > On Sun, 2022-08-28 at 17:54 -0500, Steven Robbins wrote: > > Specficially: in the case of a NEW binary upload, could a manual request > > be > > implemented (pick a different name if "give back" is not suitable) such > > that it is thrown away and replaced by a buildd build? > > If you are suggesting the ability for dak to replace binaries already > in the archive with different content without a new source version, > then I don't think that should be implemented for Debian at least, > since our archive is meant to only contain immutable package files. OK, so let's call it "bin nmu", then. And add the "+bN" version suffix. > The dak software already has an option to enable throwing away all the > binary packages from NEW before they reach the archive, but this option > is not yet enabled on the Debian ftp-master server unfortunately. This would clearly be optimal. I'm just asking about an additional / alternative mechanism. -Steve signature.asc Description: This is a digitally signed message part.
Re: Need a buildd build after trip through NEW -- best practice?
On Sun, Aug 28, 2022 at 05:54:36PM -0500, Steven Robbins wrote: > I understand that the current state is that one can only "give back" a failed > build. I'm asking whether this must necessarily be the case. Yes, by definition. > Specficially: in the case of a NEW binary upload, could a manual request be > implemented (pick a different name if "give back" is not suitable) such that > it > is thrown away and replaced by a buildd build? That's called a binNMU and it was already explained in this thread that it already happens and that it cannot work for packages building arch:all. -- WBR, wRAR signature.asc Description: PGP signature
Re: Need a buildd build after trip through NEW -- best practice?
On Sun, 2022-08-28 at 17:54 -0500, Steven Robbins wrote: > Specficially: in the case of a NEW binary upload, could a manual request be > implemented (pick a different name if "give back" is not suitable) such that > it > is thrown away and replaced by a buildd build? The dak software already has an option to enable throwing away all the binary packages from NEW before they reach the archive, but this option is not yet enabled on the Debian ftp-master server unfortunately. If you are suggesting the ability for dak to replace binaries already in the archive with different content without a new source version, then I don't think that should be implemented for Debian at least, since our archive is meant to only contain immutable package files. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Need a buildd build after trip through NEW -- best practice?
On Tuesday, August 23, 2022 6:56:32 P.M. CDT Nilesh Patra wrote: > On 24 August 2022 3:29:10 am IST, Steven Robbins wrote: > >The binary upload cannot transition to testing -- a buildd binary build is > >required. So far as I know -- assuming [1] is still up-to-date, this means > >a nuisance upload just bumping the debian revision from -1 to -2. Is this > >still the recommended practice? > >I've also been wondering about the "Give Back" action button on the buildd > >log page. It doesn't work in this case because "Package in state > >Installed, cannot give back. ✗". > >Wondering if the logic can be modified to also check > >whether the build was done on a buildd -- e.g. check whether the logs > >column contains "no log". If it wasn't a buildd build, can the giveback > >be allowed? > It was probably intentional. In any case, you might want to CC the > wanna-build team ML as well I understand that the current state is that one can only "give back" a failed build. I'm asking whether this must necessarily be the case. Specficially: in the case of a NEW binary upload, could a manual request be implemented (pick a different name if "give back" is not suitable) such that it is thrown away and replaced by a buildd build? Thanks, -Steve signature.asc Description: This is a digitally signed message part.
Re: Need a buildd build after trip through NEW -- best practice?
Hello, On Wed 24 Aug 2022 at 12:09AM GMT, Holger Levsen wrote: > > On Tue, Aug 23, 2022 at 04:59:10PM -0500, Steven Robbins wrote: >> Commonly, I update a package that provides a shared library. Due to the >> package naming convention, a new SOVERSION necessitates a trip through NEW, >> which in turn means a binary upload. >> >> The binary upload cannot transition to testing -- a buildd binary build is >> required. So far as I know -- assuming [1] is still up-to-date, this means a >> nuisance upload just bumping the debian revision from -1 to -2. Is this >> still >> the recommended practice? > > yes. > > it's rather easy to do too, though maybe there should be something in > src:devscripts > implementing something along these lines: > > dch -i -m "Source only upload for testing migration." > dch -r > debuild -S > cd .. ; dput $changes_file > # git commit & git tag When the Emacs team needed to rebuild all our arch:all packages David did it with something like for foo in ...; do dgit clone foo dch "Rebuild for ..." dch -r git commit debian/changelog -m"..." dgit push-source done The advantage being that it's git workflow-agnostic, so perhaps more more useful to have that in devscripts. -- Sean Whitton signature.asc Description: PGP signature
Re: Need a buildd build after trip through NEW -- best practice?
On Wed, Aug 24, 2022 at 10:06:55PM +0200, Paul Gevers wrote: > > The patch for dropping NEW binaries has been in dak for two years but > > its configuration options were never enabled by ftp-master so far. > > Dinstall::ThrowAwayNewBinarySuites > > Dinstall::ThrowAwayNewBinaryComponents > I would be a great fan of this happening. YES, please. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ If you liked Corona, you will also enjoy the upcoming global climate disaster. signature.asc Description: PGP signature
Re: Need a buildd build after trip through NEW -- best practice?
On Thu, Aug 25, 2022 at 07:13:52PM +0800, Paul Wise wrote: > On Thu, 2022-08-25 at 11:04 +0200, Paul Gevers wrote: > > In testing and on release architectures, I'm only aware [1] of one that > > can't build arch:all+arch:any binaries on amd64 (cmucl), but even that > > one builds its arch:all binaries on amd64. I'm wondering if there are > > packages where this is a known issue (and with the missing header, is > > there a way the outside world can track this)? > I guess finding out the list of such packages would require someone to > do a rebuild run of the arch:all packages on arm64 or similar. https://tests.reproducible-builds.org/debian/ does rebuild of all source packages for arm64, armhf, i386 and amd64. https://tests.reproducible-builds.org/debian/unstable/arm64/index_blacklisted.html shows 3 packages we have blacklisted on unstable/arm64. https://tests.reproducible-builds.org/debian/bookworm/arm64/index_blacklisted.html shows 1 package blacklisted on bookworm/arm64. https://tests.reproducible-builds.org/debian/bookworm/arm64/index_FTBFS.html shows 1105 packages failing to build on bookworm/arm64, while only 829 packages fail to build on bookworm/amd64 as shown on https://tests.reproducible-builds.org/debian/bookworm/amd64/index_FTBFS.html This data is also available via json as described in https://jenkins.debian.net/userContent/about.html#_reproducible_builds_jobs There are two JSON files which can be downloaded from https://tests.reproducible-builds.org/reproducible.json and https://tests.reproducible-builds.org/reproducible-tracker.json The 1st one has all the data (except history) and the 2nd has all the data we consider relevant to bother maintainers with, that is, some ftbfs isses are excluded. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ Another end of the world is possible. signature.asc Description: PGP signature
Re: Need a buildd build after trip through NEW -- best practice?
On Thu, 2022-08-25 at 11:04 +0200, Paul Gevers wrote: > In testing and on release architectures, I'm only aware [1] of one that > can't build arch:all+arch:any binaries on amd64 (cmucl), but even that > one builds its arch:all binaries on amd64. I'm wondering if there are > packages where this is a known issue (and with the missing header, is > there a way the outside world can track this)? I guess finding out the list of such packages would require someone to do a rebuild run of the arch:all packages on arm64 or similar. > I recall some ports have a not-for-us list, is that exposed for amd64? The Auto-Not-For-Us state for amd64 is filled with packages that do not have amd64 in their host architecture list. I think it contains things for other ports and things that aren't 64-bit yet. https://buildd.debian.org/status/architecture.php?a=amd64=sid There is also the Not-For-Us state, I think that is set manually by porters or buildd admins, but this seems rarely done, one example: https://buildd.debian.org/status/architecture.php?a=mipsel=sid -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: Need a buildd build after trip through NEW -- best practice?
Hi all On 25-08-2022 02:43, Paul Wise wrote: I don't think Build-Architecture header is done yet? Neither do I. Although since we build all arch:all packages on amd64 machines now I don't think this is needed for throwing away NEW binaries? In testing and on release architectures, I'm only aware [1] of one that can't build arch:all+arch:any binaries on amd64 (cmucl), but even that one builds its arch:all binaries on amd64. I'm wondering if there are packages where this is a known issue (and with the missing header, is there a way the outside world can track this)? I recall some ports have a not-for-us list, is that exposed for amd64? So probably the feature is ready to be enabled, although maybe after the bookworm release is the best time to enable it in case there is any unforeseen autocruft/(re)bootstrap/other fallout. I think there's still time to fix stuff if we enable it soon. Paul [1] https://qa.debian.org/dose/debcheck/src_testing_main/index.html (difference between amd64 and each) OpenPGP_signature Description: OpenPGP digital signature
Re: Need a buildd build after trip through NEW -- best practice?
On 2022-08-25 08:43:58 +0800, Paul Wise wrote: > On Wed, 2022-08-24 at 23:19 +0200, Sebastian Ramacher wrote: > > > I run > > > > $ drt-tools process-excuses > > > > once a day (except during VAC or when I am not in front of a box with my > > Debian keys). That schedules binNMUs for all packages that only require > > a rebuild and have no other issues preventing migration. > > Perhaps those binNMUs should be done from release.d.o, so > that the responsibility for them is the full release team? In theory I agree … but the code requires a rustc version that is not in stable and a bunch of rust crates that are not packaged. Since I don't have the time to backport rustc and I don't want to burden DSA with maintining a rustup/cargo setup, that's currently not possible. Cheers -- Sebastian Ramacher
Re: Need a buildd build after trip through NEW -- best practice?
On Wed, 2022-08-24 at 23:19 +0200, Sebastian Ramacher wrote:
> I run
>
> $ drt-tools process-excuses
>
> once a day (except during VAC or when I am not in front of a box with my
> Debian keys). That schedules binNMUs for all packages that only require
> a rebuild and have no other issues preventing migration.
Perhaps those binNMUs should be done from release.d.o, so
that the responsibility for them is the full release team?
The binNMUs will still be needed when all NEW binaries are discarded,
because maintainers will still occasionally accidentally or on purpose
(for eg rebootstraps) do uploads with both source and binaries and the
dak patches only discard NEW binaries, not all maintainer binaries.
> > > Dinstall::ThrowAwayNewBinarySuites
> > > Dinstall::ThrowAwayNewBinaryComponents
> >
> > I would be a great fan of this happening.
>
> Indeed.
The dak docs/TODO file still has this in it.
* Throw away all DD uploaded .debs. (Depend on "Lintian based automated
rejects")
- Need a way to define a build-architecture for arch_all debs. Some of
them can only be built on certain architectures.
A control file header build-architecture: YXY should do it.
- It's a suite option, not active for all at once.
- Should have all buildd machines under dsa control
Lintian based rejects is done long ago.
I don't think Build-Architecture header is done yet? Although since we
build all arch:all packages on amd64 machines now I don't think this is
needed for throwing away NEW binaries?
AFAICS from `git grep -iW throw.*away`, the code works by to saving all
binaries from NEW uploads to the morgue instead of the archive, for
combinations of suite and component listed in the config options.
https://salsa.debian.org/ftp-team/dak/
The options aren't set, except in the test suite:
Dinstall {
ThrowAwayNewBinarySuites {
"unstable";
};
ThrowAwayNewBinaryComponents {
"main";
};
};
All buildds for official architectures are run by DSA these days.
The tests for this feature assume "that uploads by buildds use a suffix
(like pkgnew_0.1-1_amd64-buildd.changes), to avoid filename conflicts
with the orignal upload", looks like that is true for Debian now, based
on a quick look through the morgue and the proposed-updates dir:
https://deb.debian.org/debian/dists/bullseye-proposed-updates/
I don't know if the cruft report code will detect these sourceful
uploads without the discarded binaries as cruft and remove them,
but I guess that scenario was tested before the feature was merged.
The only other issue I can think of is in a bootstrap situation,
you want to keep maintainer-built binaries rather than discarding them,
but I guess a maintainer binary-only upload can work around that issue,
followed of course by binNMUs and the corresponding buildd uploads.
So probably the feature is ready to be enabled, although maybe after
the bookworm release is the best time to enable it in case there is any
unforeseen autocruft/(re)bootstrap/other fallout.
--
bye,
pabs
https://wiki.debian.org/PaulWise
signature.asc
Description: This is a digitally signed message part
Re: Need a buildd build after trip through NEW -- best practice?
On 2022-08-24 22:06:55 +0200, Paul Gevers wrote: > Hi, > > On 24-08-2022 02:05, Paul Wise wrote: > > The release team automatically do binNMUs for packages that need a > > rebuild to transition to testing and are able to be binNMUed > > Maybe my fellow Release Team members have automated this locally, but I'm > not aware of shared tools (or even cron jobs) that do this. If we spot them, > we *may* (and often do ) binNMU. I run $ drt-tools process-excuses once a day (except during VAC or when I am not in front of a box with my Debian keys). That schedules binNMUs for all packages that only require a rebuild and have no other issues preventing migration. The code is at https://github.com/sebastinas/drt-tools. > > The patch for dropping NEW binaries has been in dak for two years but > > its configuration options were never enabled by ftp-master so far. > > > > Dinstall::ThrowAwayNewBinarySuites > > Dinstall::ThrowAwayNewBinaryComponents > > I would be a great fan of this happening. Indeed. Cheers -- Sebastian Ramacher
Re: Need a buildd build after trip through NEW -- best practice?
Hi, On 24-08-2022 02:05, Paul Wise wrote: The release team automatically do binNMUs for packages that need a rebuild to transition to testing and are able to be binNMUed Maybe my fellow Release Team members have automated this locally, but I'm not aware of shared tools (or even cron jobs) that do this. If we spot them, we *may* (and often do ) binNMU. The patch for dropping NEW binaries has been in dak for two years but its configuration options were never enabled by ftp-master so far. Dinstall::ThrowAwayNewBinarySuites Dinstall::ThrowAwayNewBinaryComponents I would be a great fan of this happening. Paul OpenPGP_signature Description: OpenPGP digital signature
Re: Need a buildd build after trip through NEW -- best practice?
Am Wed, Aug 24, 2022 at 12:09:20AM + schrieb Holger Levsen: > > it's rather easy to do too, though maybe there should be something in > src:devscripts > implementing something along these lines: Sure its easy and may be everybody (including me) has written some local scripts. The fact that it is easy is no good reason to force a lot of developers to work on the symptoms, that binary name changes have to pass NEW. IMHO Debian would be an easier place if this would not be the case. To give some example: bamtools has an RC bug #1015861 which is now pending for five months due to passing NEW. And yes, I have pinged on IRC about this - no idea what the proper pinging frequency might be. In nearly all cases it worked nicely for me by fast processing by ftpmaster (and I again use this chance to thank the ftpmaster team for this.) On the other hand I'd love to pull some work from their shoulders and I keep on thinking that binary name changes force passing NEW is a burden for them that can be removed. In a previous thread about this Scott Kitterman gave some explanation[1] which I summarize here (please read full posting of Scott[1] to get the whole arguments - may be the summary is to short): 1. Second pair of eyeballs verifying that SONAME bump has not broken anything. 2. New binary package "steals" binary from another source. 3. Overall sense of the rename. It's not just let's do extra copyright/license checks. (which was the only argument I have heard before - AT) In his mail Scott explicitly said: Speaking only for myself, not the FTP Team. I admit I like the technical arguments given by Scott. However, to my perception the issues named above might be uncovered by automated tools we are just using and would raise according bug reports. In my (possibly naive) eyes the issue is caused by a "feature" in the ftpmaster scripts and could be solved by enhancing those scripts - provided that we as a community decide that the migration via NEW is not really needed in case of binary name changes. So should we vote about this (and if yes is there any volunteer to implement this change.) Kind regards Andreas. [1] https://lists.debian.org/debian-devel/2022/01/msg00231.html -- http://fam-tille.de
Re: Need a buildd build after trip through NEW -- best practice?
[My earlier mail went blank, not sure what's up with K-9] On 24 August 2022 3:29:10 am IST, Steven Robbins wrote: >The binary upload cannot transition to testing -- a buildd binary build is >required. So far as I know -- assuming [1] is still up-to-date, this means a >nuisance upload just bumping the debian revision from -1 to -2. Is this still >the recommended practice? Yes. >I've also been wondering about the "Give Back" action button on the buildd log >page. It doesn't work in this case because "Package in state Installed, >cannot give back. ✗". Yes, because it already succeeded. You can only `give back' for builds that failed. > Wondering if the logic can be modified to also check >whether the build was done on a buildd -- e.g. check whether the logs column >contains "no log". If it wasn't a buildd build, can the giveback be allowed? It seems intentional, so that source-only uploads are really done. But you might want to ask wanna-build team (I am not a member/admin) here[2] >[1] https://wiki.debian.org/SourceOnlyUpload [2]: https://lists.debian.org/debian-wb-team/ -- Best, Nilesh
Re: Need a buildd build after trip through NEW -- best practice?
On 24 August 2022 3:29:10 am IST, Steven Robbins wrote: >The binary upload cannot transition to testing -- a buildd binary build is >required. So far as I know -- assuming [1] is still up-to-date, this means a >nuisance upload just bumping the debian revision from -1 to -2. Is this still >the recommended practice? Yes. >I've also been wondering about the "Give Back" action button on the buildd log >page. It doesn't work in this case because "Package in state Installed, >cannot give back. ✗". This was probably done to ensure only source-only uploads make it through. >Wondering if the logic can be modified to also check >whether the build was done on a buildd -- e.g. check whether the logs column >contains "no log". If it wasn't a buildd build, can the giveback be allowed? It was probably intentional. In any case, you might want to CC the wanna-build team ML as well[2] >[1] https://wiki.debian.org/SourceOnlyUpload [2]: https://lists.debian.org/debian-wb-team/ -- Regards, Nilesh
Re: Need a buildd build after trip through NEW -- best practice?
On Tue, Aug 23, 2022 at 04:59:10PM -0500, Steven Robbins wrote: > Commonly, I update a package that provides a shared library. Due to the > package naming convention, a new SOVERSION necessitates a trip through NEW, > which in turn means a binary upload. > > The binary upload cannot transition to testing -- a buildd binary build is > required. So far as I know -- assuming [1] is still up-to-date, this means a > nuisance upload just bumping the debian revision from -1 to -2. Is this > still > the recommended practice? yes. it's rather easy to do too, though maybe there should be something in src:devscripts implementing something along these lines: dch -i -m "Source only upload for testing migration." dch -r debuild -S cd .. ; dput $changes_file # git commit & git tag 4 out of these 5 steps I've automated for myself with small scripts written catering how "my" packages are maintained (which is the part where putting this in src:devscripts is not that easy...). "debuild -S" I do manually, because sometimes that's all I do and sometimes I feed the resulting source package into yet another small script called 'spb', because it's running _s_udo _p_builder _b_uilt on the latest source package in $PWD. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ⠈⠳⣄ When this virus is over, I still want some of you all to stay away from me. signature.asc Description: PGP signature
Re: Need a buildd build after trip through NEW -- best practice?
On Tue, 2022-08-23 at 16:59 -0500, Steven Robbins wrote: > The binary upload cannot transition to testing -- a buildd binary build is > required. So far as I know -- assuming [1] is still up-to-date, this means a > nuisance upload just bumping the debian revision from -1 to -2. Is this > still > the recommended practice? The release team automatically do binNMUs for packages that need a rebuild to transition to testing and are able to be binNMUed, so for many packages you don't need to worry about this. If for some reason they don't do this or for arch:all packages, you need to reupload. Usually I use this as an opportunity for some extra package polish. The patch for dropping NEW binaries has been in dak for two years but its configuration options were never enabled by ftp-master so far. Dinstall::ThrowAwayNewBinarySuites Dinstall::ThrowAwayNewBinaryComponents > I've also been wondering about the "Give Back" action button on the buildd log The give-back action is only for failed builds, not successful ones. For successful builds you need a binNMU or new source version. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Need a buildd build after trip through NEW -- best practice?
Commonly, I update a package that provides a shared library. Due to the package naming convention, a new SOVERSION necessitates a trip through NEW, which in turn means a binary upload. The binary upload cannot transition to testing -- a buildd binary build is required. So far as I know -- assuming [1] is still up-to-date, this means a nuisance upload just bumping the debian revision from -1 to -2. Is this still the recommended practice? I've also been wondering about the "Give Back" action button on the buildd log page. It doesn't work in this case because "Package in state Installed, cannot give back. ✗". Wondering if the logic can be modified to also check whether the build was done on a buildd -- e.g. check whether the logs column contains "no log". If it wasn't a buildd build, can the giveback be allowed? Thanks, -Steve [1] https://wiki.debian.org/SourceOnlyUpload signature.asc Description: This is a digitally signed message part.
