Re: Architecture: all binNMUs (was: deduplicating jquery/)
On 12/6/20 12:27 PM, Philipp Kern wrote: > On 06.12.20 01:08, Paul Wise wrote: >> On Sat, Dec 5, 2020 at 12:21 PM Matthias Klose wrote: >> >>> Maybe there is more. But there's no progress, or intent to fix every tool >>> to be >>> aware of binNMUs. Maybe it's better to rethink how sourceful no-change >>> no-maintainer uploads could be done without introducing the above issues? >> >> `dch --rebuild` already exists, so this would just need support in >> wanna-build/sbuild for generating such uploads and support in dak for >> accepting sourceful uploads from wanna-build/buildds rather than >> maintainers. > > Given the whole source code trust story it'd be better if dak were to do > it by itself rather than relying on an external service to do it. > > (Or we make it culturally allowed to do it using client-side tooling, as > long as it is a no-change-but-debian/changelog upload.) a useful "change" is an extra build dependency on the version you are running a transition for. Apparently working this way for binNMUs. Matthias
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On 06.12.20 01:08, Paul Wise wrote: > On Sat, Dec 5, 2020 at 12:21 PM Matthias Klose wrote: > >> Maybe there is more. But there's no progress, or intent to fix every tool to >> be >> aware of binNMUs. Maybe it's better to rethink how sourceful no-change >> no-maintainer uploads could be done without introducing the above issues? > > `dch --rebuild` already exists, so this would just need support in > wanna-build/sbuild for generating such uploads and support in dak for > accepting sourceful uploads from wanna-build/buildds rather than > maintainers. Given the whole source code trust story it'd be better if dak were to do it by itself rather than relying on an external service to do it. (Or we make it culturally allowed to do it using client-side tooling, as long as it is a no-change-but-debian/changelog upload.) Kind regards Philipp Kern signature.asc Description: OpenPGP digital signature
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Sat, Dec 5, 2020 at 12:21 PM Matthias Klose wrote: > Maybe there is more. But there's no progress, or intent to fix every tool to > be > aware of binNMUs. Maybe it's better to rethink how sourceful no-change > no-maintainer uploads could be done without introducing the above issues? `dch --rebuild` already exists, so this would just need support in wanna-build/sbuild for generating such uploads and support in dak for accepting sourceful uploads from wanna-build/buildds rather than maintainers. -- bye, pabs https://wiki.debian.org/PaulWise
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On 12/1/20 11:56 AM, Simon McVittie wrote: > On Tue, 01 Dec 2020 at 11:05:54 +0100, Jonas Smedegaard wrote: >> Can someone remind me: Why is it that we cannot do arch-independent >> auto-building? > > We can and do autobuild arch-independent packages (since 2015: on the > timescale of multi-release transitions, this is relatively recent). before we add another kind of binNMUs, please can we consider not doing binNMUs at all? I see that it's convenient for transitions, however - it creates problems with mismatched versions for foreign architectures. Yes, binNMUs are currently done for M-A: same packages for all the release architectures, but usually it's a pain to construct a cross-build environment for a port out of packages in the archive. - No QA done for binNMUs. Not a single autopkg test is run for a binNMU, and it's known that these introduce regressions in testing. - lintian is not run for binNMUs. - tracker.debian.org doesn't track binNMUs - maintainers are not notified of binNMU uploads and migrations. Maybe there is more. But there's no progress, or intent to fix every tool to be aware of binNMUs. Maybe it's better to rethink how sourceful no-change no-maintainer uploads could be done without introducing the above issues? Matthias
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Wed, 02 Dec 2020 at 11:15:44 +0100, Raphael Hertzog wrote:
> But this has the obvious downside that "${source:Version}" is unchanged
> and that you might have issues with dependencies against the arch: all
> package.
Yes, I already said that this is why OBS can't realistically do its builds
like binNMUs (or more precisely, with binary-only=yes changelog entries).
Breaking the ${source:Version} assumption would break the packages listed
in https://lintian.debian.org/tags/maybe-not-arch-all-binnmuable.html and
possibly others, which is undesirable if your goal is to rebuild Debian or
Ubuntu packages with minimal modification (which is the case for my
employer's uses of OBS).
smcv
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Wed, 02 Dec 2020, Raphael Hertzog wrote: > > potentially different content, breaking the important design principle > > that things that are different should have different names. [...] > And as an aside, the archive has big holes when enforcing this "design > principle": > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876643 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620356 > It would be nice to have this fixed. I knem I missed one, there's this one too which caught me: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=949962 Cheers, -- ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog ⣾⠁⢠⠒⠀⣿⡁ ⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/ ⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: Architecture: all binNMUs (was: deduplicating jquery/)
Hi,
On Tue, 01 Dec 2020, Simon McVittie wrote:
> If I understand correctly, one of the ftp team's objections to discarding
> and rebuilding maintainer-uploaded binaries is that if I upload foo_1.2-3,
> and they discard my binary upload and rebuild it on the buildds, this
> would result in having two binary builds of foo-bin_1.2-3_amd64.deb and two
> binary builds of foo-data_1.2-3_all.deb with the same version number but
> potentially different content, breaking the important design principle
> that things that are different should have different names.
That objection is debatable for packages that have not yet reached the
archive. Dropping the binary after NEW review ought to be sufficient.
And as an aside, the archive has big holes when enforcing this "design
principle":
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876643
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620356
It would be nice to have this fixed.
> At the moment, builds in OBS cannot make use of Debian's binNMU machinery
> for this, because if they did, they would break the same ${source:Version}
> assumption I described in my previous message. The result is that they
> have to behave as though +bsos1 was a new sourceful upload, and
> foo-bin_1.2-3+bsos1_amd64.deb and foo-data_1.2-3+bsos1_all.deb both have
> Source: foo (= 1.2-3+bsos1). Consumers have to "just know" that to
> get the source code for foo-bin_1.2-3+bsos1_amd64.deb, you strip the
> /\+bsos[0-9]+$/ suffix from the version number before looking for the
> foo=1.2-3 source package. Obviously, this scales poorly if you are looking
> at multiple derivatives each with its own pseudo-binNMU suffix.
>
> If we had a way to do Architecture: all binNMUs, then OBS would be able to
> use it for all builds: for instance, foo-bin_1.2-3+bsos1_amd64.deb and
> foo-data_1.2-3+bsos1_all.deb could both have Source: foo (= 1.2-3).
In fact, dpkg has support for such binary-only rebuild. Have your
changelog entry like this:
| foo (1.2-3+bsos1) unstable; urgency=low,binary-only=yes
|
| * Binary rebuild.
|
| -- Raphaël Hertzog Wed, 02 Dec 2020 10:48:12 +0100
And then "arch: all" and "arch: any" packages will have the proper
"Source" entry with a value of "foo (1.2-3)".
I just tested it and it works (at least with raw dpkg-buildpackage).
But this has the obvious downside that "${source:Version}" is unchanged
and that you might have issues with dependencies against the arch: all
package.
In the end, the solution to make it easy to bump the version with a no-change
source upload is more likely to happen quickly.
Cheers,
--
⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋The Debian Handbook: https://debian-handbook.info/get/
⠈⠳⣄ Debian Long Term Support: https://deb.li/LTS
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Tue, Dec 01, 2020 at 12:28:38PM +0100, Julien Cristau wrote:
> On Tue, Dec 01, 2020 at 10:56:28AM +, Simon McVittie wrote:
> > Possible solutions:
> >
> > - Change at least 622 packages so they have something more like
> > Depends: foo-data (>= ${source:Version}), foo-data (<<
> > ${source:Version}+c)
> > - Change at least 622 packages so that when foo-data is binNMU'd, it
> > automatically gets Provides: foo-data (= 1.2-3)
> > - Change some more central component so that the dependencies are edited
> > or the Provides is added globally
> Make no-change-other-than-version-bump source uploads easier?
+1000.
For anyone who uses rebuilt packages outside of official Tier-1
infrastructure, binNMUs are nasty.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Imagine there are bandits in your house, your kid is bleeding out,
⢿⡄⠘⠷⠚⠋⠀ the house is on fire, and seven giant trumpets are playing in the
⠈⠳⣄ sky. Your cat demands food. The priority should be obvious...
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Tue, Dec 1, 2020 at 11:36 AM Julien Cristau wrote: > Make no-change-other-than-version-bump source uploads easier? `dch --rebuild` already exists, so this would just need support in wanna-build/sbuild for generating such uploads and support in dak for accepting sourceful uploads from wanna-build/buildds rather than maintainers. -- bye, pabs https://wiki.debian.org/PaulWise
Re: Architecture: all binNMUs (was: deduplicating jquery/)
Le 01/12/2020 à 12:19, Jonas Smedegaard a écrit :
> Quoting Simon McVittie (2020-12-01 11:56:28)
>> On Tue, 01 Dec 2020 at 11:05:54 +0100, Jonas Smedegaard wrote:
>>> Can someone remind me: Why is it that we cannot do arch-independent
>>> auto-building?
>>
>> We can and do autobuild arch-independent packages (since 2015: on the
>> timescale of multi-release transitions, this is relatively recent).
>
> Yes - thanks for catching and tightening my far too sloppy framing of
> the issue at hand.
>
>
>> However, we cannot currently do arch-independent *binNMUs*, because a
>> large number of packages have patterns like this:
>>
>> Source: foo
>>
>> Package: foo-bin
>> Architecture: any
>> Depends: foo-data (= ${source:Version})
>>
>> Package: foo-data
>> Architecture: all
>>
>> which is based on the assumption that foo-data cannot be binNMU'd.
>>
>> For example, consider foo_1.2-3, and suppose it has been binNMU'd 5
>> times on amd64, so we have foo-bin_1.2-3+b5_amd64.deb and
>> foo-data_1.2-3_all.deb. foo-bin Depends: foo-data (= 1.2-3) and the
>> dependency is satisfied.
>>
>> Now suppose we also binNMU foo-data. We get foo-data_1.2-3+b1_all.deb,
>> which no longer satisfies foo-data (= 1.2-3), so foo-bin is uninstallable.
>>
>> https://lintian.debian.org/tags/maybe-not-arch-all-binnmuable.html lists
>> 622 affected packages. I don't know whether there are other problematic
>> patterns that Lintian does not detect.
>
> Thanks for the excellent explanation.
>
> (and sorry for my failing memory: I am sure someone explained it to me
> at least once before, related to debian-installer image building)
>
>
>> Possible solutions:
>>
>> - Change at least 622 packages so they have something more like
>> Depends: foo-data (>= ${source:Version}), foo-data (<< ${source:Version}+c)
>> (also hope that all of their maintainers can get those runes right, taking
>> into account what the binNMU suffix is, and hope that it won't break
>> derivatives like Ubuntu that use a source-upload suffix like ubuntu1 that
>> compares less than +b1)
>>
>> - Change at least 622 packages so that when foo-data is binNMU'd, it
>> automatically gets Provides: foo-data (= 1.2-3)
>>
>> - Change some more central component so that the dependencies are edited
>> or the Provides is added globally
>>
>> - Something clever that I haven't thought of
>
> - Introduce a control field "Build-Dependencies-Require-Rebuild: no"
> (similar to "Rules-Requires-Root: no")
+1 !
Thanks!
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Tue, Dec 01, 2020 at 10:56:28AM +, Simon McVittie wrote:
> Possible solutions:
>
> - Change at least 622 packages so they have something more like
> Depends: foo-data (>= ${source:Version}), foo-data (<< ${source:Version}+c)
> (also hope that all of their maintainers can get those runes right, taking
> into account what the binNMU suffix is, and hope that it won't break
> derivatives like Ubuntu that use a source-upload suffix like ubuntu1 that
> compares less than +b1)
>
> - Change at least 622 packages so that when foo-data is binNMU'd, it
> automatically gets Provides: foo-data (= 1.2-3)
>
> - Change some more central component so that the dependencies are edited
> or the Provides is added globally
>
> - Something clever that I haven't thought of
>
Make no-change-other-than-version-bump source uploads easier?
Cheers,
Julien
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Tue, 01 Dec 2020 at 10:56:28 +, Simon McVittie wrote:
> We can and do autobuild arch-independent packages (since 2015: on the
> timescale of multi-release transitions, this is relatively recent).
>
> However, we cannot currently do arch-independent *binNMUs* [because...]
I've thought in the past that if the inability to do arch-independent
binNMUs can be fixed, it would be a good way to deal with some problems
that we currently face.
If I understand correctly, one of the ftp team's objections to discarding
and rebuilding maintainer-uploaded binaries is that if I upload foo_1.2-3,
and they discard my binary upload and rebuild it on the buildds, this
would result in having two binary builds of foo-bin_1.2-3_amd64.deb and two
binary builds of foo-data_1.2-3_all.deb with the same version number but
potentially different content, breaking the important design principle
that things that are different should have different names.
In the Open Build Service (OBS), which is analogous to wanna-build + sbuild,
the normal configuration is that binaries built by OBS *always* get a
build suffix. For example, if I upload foo_1.2-3 to the OBS instance used
by SteamOS, the result will usually be foo-bin_1.2-3+bsos1_amd64.deb and
foo-data_1.2-3+bsos1_all.deb, because SteamOS' OBS was configured to use
suffix +bsos (where bsos is mnemonic for "built for SteamOS"
and is an automatically incremented integer).
At the moment, builds in OBS cannot make use of Debian's binNMU machinery
for this, because if they did, they would break the same ${source:Version}
assumption I described in my previous message. The result is that they
have to behave as though +bsos1 was a new sourceful upload, and
foo-bin_1.2-3+bsos1_amd64.deb and foo-data_1.2-3+bsos1_all.deb both have
Source: foo (= 1.2-3+bsos1). Consumers have to "just know" that to
get the source code for foo-bin_1.2-3+bsos1_amd64.deb, you strip the
/\+bsos[0-9]+$/ suffix from the version number before looking for the
foo=1.2-3 source package. Obviously, this scales poorly if you are looking
at multiple derivatives each with its own pseudo-binNMU suffix.
If we had a way to do Architecture: all binNMUs, then OBS would be able to
use it for all builds: for instance, foo-bin_1.2-3+bsos1_amd64.deb and
foo-data_1.2-3+bsos1_all.deb could both have Source: foo (= 1.2-3).
Similarly, we could consider using it in Debian to address the ftp
team's valid concern about having buildd-built binaries whose version
matches different maintainer-built binaries. Instead of starting from
the absence of a binNMU suffix, the buildds could start from +b1, so
that we'd have this workflow:
- maintainer builds, tests and uploads foo_1.2-3
- maintainer's packages are foo_1.2-3.dsc, foo-bin_1.2-3_amd64.deb and
foo-data_1.2-3_all.deb
- or perhaps they would be foo_1.2-3.dsc, foo-bin_1.2-3+b0_amd64.deb and
foo-data_1.2-3+b0_all.deb
- (if NEW) ftp team inspects maintainer-built source and binaries
- ftp machinery discards maintainer-built binaries
- buildd receives foo_1.2-3.dsc
- buildds produce foo-bin_1.2-3+b1_ARCH.deb for each architecture
- Architecture: all buildd produces foo-data_1.2-3+b1_all.deb
and if, later, the release team triggers some binNMUs for a transition:
- buildds produce foo-bin_1.2-3+bN_ARCH.deb, N >= 2, for each architecture
and/or
- Architecture: all buildd produces foo-data_1.2-3+bN_all.deb, N >= 2
smcv
Re: Architecture: all binNMUs (was: deduplicating jquery/)
Quoting Simon McVittie (2020-12-01 11:56:28)
> On Tue, 01 Dec 2020 at 11:05:54 +0100, Jonas Smedegaard wrote:
> > Can someone remind me: Why is it that we cannot do arch-independent
> > auto-building?
>
> We can and do autobuild arch-independent packages (since 2015: on the
> timescale of multi-release transitions, this is relatively recent).
Yes - thanks for catching and tightening my far too sloppy framing of
the issue at hand.
> However, we cannot currently do arch-independent *binNMUs*, because a
> large number of packages have patterns like this:
>
> Source: foo
>
> Package: foo-bin
> Architecture: any
> Depends: foo-data (= ${source:Version})
>
> Package: foo-data
> Architecture: all
>
> which is based on the assumption that foo-data cannot be binNMU'd.
>
> For example, consider foo_1.2-3, and suppose it has been binNMU'd 5
> times on amd64, so we have foo-bin_1.2-3+b5_amd64.deb and
> foo-data_1.2-3_all.deb. foo-bin Depends: foo-data (= 1.2-3) and the
> dependency is satisfied.
>
> Now suppose we also binNMU foo-data. We get foo-data_1.2-3+b1_all.deb,
> which no longer satisfies foo-data (= 1.2-3), so foo-bin is uninstallable.
>
> https://lintian.debian.org/tags/maybe-not-arch-all-binnmuable.html lists
> 622 affected packages. I don't know whether there are other problematic
> patterns that Lintian does not detect.
Thanks for the excellent explanation.
(and sorry for my failing memory: I am sure someone explained it to me
at least once before, related to debian-installer image building)
> Possible solutions:
>
> - Change at least 622 packages so they have something more like
> Depends: foo-data (>= ${source:Version}), foo-data (<< ${source:Version}+c)
> (also hope that all of their maintainers can get those runes right, taking
> into account what the binNMU suffix is, and hope that it won't break
> derivatives like Ubuntu that use a source-upload suffix like ubuntu1 that
> compares less than +b1)
>
> - Change at least 622 packages so that when foo-data is binNMU'd, it
> automatically gets Provides: foo-data (= 1.2-3)
>
> - Change some more central component so that the dependencies are edited
> or the Provides is added globally
>
> - Something clever that I haven't thought of
- Introduce a control field "Build-Dependencies-Require-Rebuild: no"
(similar to "Rules-Requires-Root: no")
I mean, we could limit BinNMUing to packages that pre-declares that they
have taken into account the possibility of version skew not only for
arch-specific but also arch-independent package relations.
Yes, It feels wrong to introduce an explicit control field for that, but
there is a real benefit here for packages using doxygen, and there is a
real benefit for JavaScript packages in general, and I am sure other
areas as well (including possibly the debian-installer images?).
- Jonas
--
* Jonas Smedegaard - idealist & Internet-arkitekt
* Tlf.: +45 40843136 Website: http://dr.jones.dk/
[x] quote me freely [ ] ask before reusing [ ] keep private
signature.asc
Description: signature
Re: Architecture: all binNMUs (was: deduplicating jquery/)
On Tue, 01 Dec 2020 at 11:05:54 +0100, Jonas Smedegaard wrote:
> Can someone remind me: Why is it that we cannot do arch-independent
> auto-building?
We can and do autobuild arch-independent packages (since 2015: on the
timescale of multi-release transitions, this is relatively recent).
However, we cannot currently do arch-independent *binNMUs*, because a
large number of packages have patterns like this:
Source: foo
Package: foo-bin
Architecture: any
Depends: foo-data (= ${source:Version})
Package: foo-data
Architecture: all
which is based on the assumption that foo-data cannot be binNMU'd.
For example, consider foo_1.2-3, and suppose it has been binNMU'd 5
times on amd64, so we have foo-bin_1.2-3+b5_amd64.deb and
foo-data_1.2-3_all.deb. foo-bin Depends: foo-data (= 1.2-3) and the
dependency is satisfied.
Now suppose we also binNMU foo-data. We get foo-data_1.2-3+b1_all.deb,
which no longer satisfies foo-data (= 1.2-3), so foo-bin is uninstallable.
https://lintian.debian.org/tags/maybe-not-arch-all-binnmuable.html lists
622 affected packages. I don't know whether there are other problematic
patterns that Lintian does not detect.
Possible solutions:
- Change at least 622 packages so they have something more like
Depends: foo-data (>= ${source:Version}), foo-data (<< ${source:Version}+c)
(also hope that all of their maintainers can get those runes right, taking
into account what the binNMU suffix is, and hope that it won't break
derivatives like Ubuntu that use a source-upload suffix like ubuntu1 that
compares less than +b1)
- Change at least 622 packages so that when foo-data is binNMU'd, it
automatically gets Provides: foo-data (= 1.2-3)
- Change some more central component so that the dependencies are edited
or the Provides is added globally
- Something clever that I haven't thought of
smcv
