Re: packages expected to fail on some archs
On Tue, Sep 27, 2022 at 12:23:57AM +0300, Adrian Bunk wrote: > If we limit the problem to avoiding build failures in cases that > upstream does not support, there would be the trivial solution of > having a package ship Provides like: > - architecture-is-64bit > - architecture-is-32bit > - architecture-is-little-endian > - architecture-is-big-endian > - architecture-has-64bit-timet > Build-Depends: architecture-is-64bit, architecture-is-little-endian,... > would be a package that only supports 64bit little endian architectures, > and that would never be attempted to build on 32bit or big endian > architectures. > > The buildd page would then show for i386: > mypackage build-depends on missing: > - architecture-is-64bit Alas, this scheme won't work for cross-building or multi-arch :( Meow! -- ⢀⣴⠾⠻⢶⣦⠀ Genetics say, virgin birth may not possibly produce male offspring. ⣾⠁⢠⠒⠀⣿⡁ Thus, either Jesus was fathered by (Abdes?) Pantera, or she was trans. ⢿⡄⠘⠷⠚⠋⠀ ⠈⠳⣄ In neither case Joseph is involved, making Jesus a bastard.
Re: packages expected to fail on some archs
Hi Le lun. 26 sept. 2022 à 23:42, Adrian Bunk a écrit : > > If we limit the problem to avoiding build failures in cases that > upstream does not support, there would be the trivial solution of > having a package ship Provides like: > - architecture-is-64bit > - architecture-is-32bit > - architecture-is-little-endian > - architecture-is-big-endian > - architecture-has-64bit-timet > -... > > Build-Depends: architecture-is-64bit, architecture-is-little-endian,... > would be a package that only supports 64bit little endian architectures, > and that would never be attempted to build on 32bit or big endian > architectures. > > The buildd page would then show for i386: > mypackage build-depends on missing: > - architecture-is-64bit > > Not building a source package on one specific architecture could already > today be achieved with: > Build-Depends: package-is-broken-on-ppc64el [ppc64el],... > > This might not be the most elegant solution, but it should be sufficient > to solve the problem in this thread and it does not require any tool > changes. > I find it both simple and elegant -- and it's probably pretty efficient too. Perhaps there should be a conventional naming scheme for such virtual packages ; say deb-missing-feature, deb-unsupported-architecture or some such? J.Puydt >
Re: packages expected to fail on some archs
On Wed, Sep 14, 2022 at 01:38:01PM +0200, Guillem Jover wrote: >... > [ Mostly to summarize the status re dpkg. ] >... > * Lack of bits/endianness arch "aliases" (#962848). The main problem > with this one is that we cannot simply add such aliases, as then > those would silently be considered as regular arches, and they do > not map into the current naming convention at all. These would need > to be added with a breaking syntax (say with some non-accepted > char, such as % or whatever) so that they do not introduce silent > breakage. This would then need to be supported by anything > handling arch restrictions (field and dependencies) which can be a > rather large surface. Then there is the problem that architectures > are evaluated as ORed lists, and the bits/endianness might require > to be treated as ANDed lists some times (of course the latter > could be handled by combining them into single aliases, but meh). If we limit the problem to avoiding build failures in cases that upstream does not support, there would be the trivial solution of having a package ship Provides like: - architecture-is-64bit - architecture-is-32bit - architecture-is-little-endian - architecture-is-big-endian - architecture-has-64bit-timet -... Build-Depends: architecture-is-64bit, architecture-is-little-endian,... would be a package that only supports 64bit little endian architectures, and that would never be attempted to build on 32bit or big endian architectures. The buildd page would then show for i386: mypackage build-depends on missing: - architecture-is-64bit Not building a source package on one specific architecture could already today be achieved with: Build-Depends: package-is-broken-on-ppc64el [ppc64el],... This might not be the most elegant solution, but it should be sufficient to solve the problem in this thread and it does not require any tool changes. > Thanks, > Guillem cu Adrian
Re: packages expected to fail on some archs
Guillem Jover, le mer. 14 sept. 2022 13:38:01 +0200, a ecrit: > Something else to consider is that, for packages that make sense > porting, deny-listing them from building means we do not have build > failure logs, so deciding what to port or trying to check for patterns > becomes more costly for humans, Yes! > of course at the cost of potentially throwing at it buildd resources. Ports buildd maintainers can blacklist packages easily if they are seen to hog buildd resources. That's easier for them to maintain than having to get a package uploaded. > we already had something similar with the Packages-arch-specific file, > but my understanding is that we are moving away from it? Yes we are because it's tedious to update. Samuel
Re: packages expected to fail on some archs
Hi! [ Mostly to summarize the status re dpkg. ] On Sun, 2022-09-11 at 17:08:57 +0200, Samuel Thibault wrote: > The issue we see is that some DDs end up setting a hardcoded list in > the "Architecture" field, rather than just letting builds keep failing > on these archs (and then possibly succeeding after some time whenever > somebody contributes a fix upstream that gets propagated to Debian). > That said, I guess DDs would still set a manual Arch list so as to get > the red flags away from their sight on the buildd page status, e.g. from > https://buildd.debian.org/status/package.php?p=sthibault%40debian.org=yes=compact > so that they can easily notice which build failures are actually not > expected, to be able to efficiently work on those. > > So perhaps what is missing is a way to express that an FTBFS is > expected, so that the buildd page status represents it differently, e.g. > in orange rather than red? The idea would be that it should be really > easy to set (as easy as setting an Architecture list) so that we can > easily recommend it. Yes, I agree that this seems like a problem with representing state, where people feel the need to move from the red to green. I think the proposals that have been mentioned elsethread are nice, where a regression would be red but whenever a package was not previously built (because it never has or it got removed) for an arch, that is shown in a way consistent with how testing transitions would take it (orange or another non-red color seems good). Something else to consider is that, for packages that make sense porting, deny-listing them from building means we do not have build failure logs, so deciding what to port or trying to check for patterns becomes more costly for humans, of course at the cost of potentially throwing at it buildd resources. > We could for instance: > - Add an Architecture-FTBFS field to debian/control > - Add an environment variable to debian/rules so that on these archs dh > fails with a different exit code that buildds would notice. > - Add a Architecture-FTBFS field in the wb database that DDs can set > > The tricky part I see is that AIUI the buildd status page doesn't > have direct access to debian/control, only to the wb database, so the > first solution is not immediate to implement. The third option would > be the most obvious from the buildd point of view, but that's not very > convenient for DDs. Possibly such field could be automatically set > according to the Packages entry when a newer upload is done? While these might be worth it on their own, it feels (as has been covered elsethread) to be a bit of a workaround for lack of better expressiveness in our Architecture declarations, and build state representation. But see my reply to #807263 about how I see the different cases. Also for the third we already had something similar with the Packages-arch-specific file, but my understanding is that we are moving away from it? The common mentioned omissions are: * Lack of negated architectures in the field. As mentioned on #807264 this is kind of supported already by dpkg, but it would need to be accepted by the project, and the actual syntax and semantics documented. (Personally I think this should not generally be used, and there are very few cases where this seems legitimate?) * Lack of bits/endianness arch "aliases" (#962848). The main problem with this one is that we cannot simply add such aliases, as then those would silently be considered as regular arches, and they do not map into the current naming convention at all. These would need to be added with a breaking syntax (say with some non-accepted char, such as % or whatever) so that they do not introduce silent breakage. This would then need to be supported by anything handling arch restrictions (field and dependencies) which can be a rather large surface. Then there is the problem that architectures are evaluated as ORed lists, and the bits/endianness might require to be treated as ANDed lists some times (of course the latter could be handled by combining them into single aliases, but meh). Thanks, Guillem
Re: packages expected to fail on some archs
On Mon, Sep 12, 2022 at 04:08:08PM +0200, Tobias Frost wrote: >... > The problem is that if you want to exclude an arch explicitly, you have to > list all archs you want to build it on. IOW, I'm missing an easy way to say > "not on THIS architecture", somthing like "[!armel]" >... > I don't actually believe people are using them on that arch, because if they > would, I would get bug reports about them… Upstream agrees with that: Although > off-topic, I would be eager to know if there is anybody using $PACKAGE on an > s390/System Z?" A relevant quesion would be whether this is [!s390x] or [littleendian]. There are 5 additional big endian architectures in ports, plus people elsewhere apparently working on ports like arm64eb. And while it is unclear whether s390x has any users at all on Debian, ports architectures like hppa or powerpc do have users. cu Adrian
Re: packages expected to fail on some archs
Tobias Frost, le lun. 12 sept. 2022 18:36:09 +0200, a ecrit: > On Mon, Sep 12, 2022 at 05:11:46PM +0200, Samuel Thibault wrote: > > Tobias Frost, le lun. 12 sept. 2022 16:08:08 +0200, a ecrit: > > > The problem is that if you want to exclude an arch explicitly, you have to > > > list all archs you want to build it on. IOW, I'm missing an easy way to > > > say > > > "not on THIS architecture", somthing like "[!armel]" > > > > Yes, but see below. > > > > > There are a few packages I take care of which make trouble on some archs > > > or > > > simply do not make much sense to run on those low-end archs. > > > > If they make trouble, I would say just let the package FTBFS there. > > Well, it compiles there… Of course I could fail it artifically, but that > isn't something I would say it would be appropiate. That may still be more informative than hardcoding an Architecture list? I happen to be doing that in sphinxbase actually: https://salsa.debian.org/a11y-team/sphinxbase/-/blob/master/debian/rules#L30 > > > I was spending siginifant time in the past weeks on such a package, to fix > > > autopkgtests issues specific to that arch -- unsuccessfully, I disabled > > > the > > > tests in the end --, > > > > Is it possible to get the same test be performed during package build > > time? That way, it will be just not built, not shipped, and the state > > will be clear on the buildd status page, and you can move on to more > > useful work. For instance in my pocketsphinx package case: > > Would do that if it would be possible; The tests won't run properly without > the > data installed properly. Ok, so it's a corner case we were discussing on debian-ports indeed. Samuel
Re: packages expected to fail on some archs
On Mon, Sep 12, 2022 at 05:11:46PM +0200, Samuel Thibault wrote: > Hello, > > Tobias Frost, le lun. 12 sept. 2022 16:08:08 +0200, a ecrit: > > The problem is that if you want to exclude an arch explicitly, you have to > > list all archs you want to build it on. IOW, I'm missing an easy way to say > > "not on THIS architecture", somthing like "[!armel]" > > Yes, but see below. > > > There are a few packages I take care of which make trouble on some archs or > > simply do not make much sense to run on those low-end archs. > > If they make trouble, I would say just let the package FTBFS there. Well, it compiles there… Of course I could fail it artifically, but that isn't something I would say it would be appropiate. > > I was spending siginifant time in the past weeks on such a package, to fix > > autopkgtests issues specific to that arch -- unsuccessfully, I disabled the > > tests in the end --, > > Is it possible to get the same test be performed during package build > time? That way, it will be just not built, not shipped, and the state > will be clear on the buildd status page, and you can move on to more > useful work. For instance in my pocketsphinx package case: Would do that if it would be possible; The tests won't run properly without the data installed properly. -- tobi
Re: packages expected to fail on some archs
Hello, Tobias Frost, le lun. 12 sept. 2022 16:08:08 +0200, a ecrit: > The problem is that if you want to exclude an arch explicitly, you have to > list all archs you want to build it on. IOW, I'm missing an easy way to say > "not on THIS architecture", somthing like "[!armel]" Yes, but see below. > There are a few packages I take care of which make trouble on some archs or > simply do not make much sense to run on those low-end archs. If they make trouble, I would say just let the package FTBFS there. > I was spending siginifant time in the past weeks on such a package, to fix > autopkgtests issues specific to that arch -- unsuccessfully, I disabled the > tests in the end --, Is it possible to get the same test be performed during package build time? That way, it will be just not built, not shipped, and the state will be clear on the buildd status page, and you can move on to more useful work. For instance in my pocketsphinx package case: https://buildd.debian.org/status/package.php?p=pocketsphinx the mips tests fail, I just let it fail. If anybody feels interested enough to take the time to fix the bug, then great. In the meanwhile it will just not be available since it is broken. The only work I have done for that problem is reporting the issue upstream: https://github.com/cmusphinx/pocketsphinx/issues/252 What was proposed in the thread was to make the buildd page show the failure in orange, so that people know that it's a known failure, and not a new bug. Samuel
Re: packages expected to fail on some archs
On Sun, Sep 11, 2022 at 11:07:13PM +0300, Adrian Bunk wrote: > On Sun, Sep 11, 2022 at 05:08:57PM +0200, Samuel Thibault wrote: > >... > > The issue we see is that some DDs end up setting a hardcoded list in > > the "Architecture" field, rather than just letting builds keep failing > > on these archs (and then possibly succeeding after some time whenever > > somebody contributes a fix upstream that gets propagated to Debian). > >... > > I'd say it is the best solution when a package needs non-trivial > architecture-specific porting for every architecture it supports. > > With "non-trivial" I mean not just adding a new architecture to a few > #ifdefs, but serious upstream porting efforts. E.g. valgrind does not > support riscv64, and if it would ever gain the support in a new upstream > version I'd expect the maintainer to add that to the Architecture field > when upstream announces support for a new architecture. > > But Architecture lists for expressing e.g. "64bit" or "little endian" > are a real pain for everyone working on bringup of a new port. > > Which happens far more often than most people realize. > > There is not only riscv64 (64bit, little endian). > > Ports is about to start building for arc (32bit, little endian). > > There are people working on ports like arm64be (64bit, big endian), > loongarch64 (64bit, little endian) and many other ports that might > never end up being built in the Debian infrastructure (but some of > them might get built by derivatives). > > Architecture lists containing all 64bit ports or all little endian > ports create much extra work for anyone adding support for a new 64bit > little endian architecture. The problem is that if you want to exclude an arch explicitly, you have to list all archs you want to build it on. IOW, I'm missing an easy way to say "not on THIS architecture", somthing like "[!armel]" There are a few packages I take care of which make trouble on some archs or simply do not make much sense to run on those low-end archs. They compile there beautifully, or at least the usually do, (which then creates a lot of maintainer overhead then, because out-of-date…) I was spending siginifant time in the past weeks on such a package, to fix autopkgtests issues specific to that arch -- unsuccessfully, I disabled the tests in the end --, where noone will ever use that package. I was very close just to do what Sam asked us not to do. (And I agree with him that this is not good.) I don't actually believe people are using them on that arch, because if they would, I would get bug reports about them… Upstream agrees with that: Although off-topic, I would be eager to know if there is anybody using $PACKAGE on an s390/System Z?" > > Samuel > > cu > Adrian >
Re: packages expected to fail on some archs
On Sun, 2022-09-11 at 23:07 +0300, Adrian Bunk wrote: > Architecture lists containing all 64bit ports or all little endian > ports create much extra work for anyone adding support for a new > 64bit little endian architecture. Since dpkg already knows about the bits and endianness of ports, perhaps it could add some sort of architecture wildcard system, which would get resolved to real architectures at .dsc build time? Alternatively the wildcards could be added to the .dsc too but that would require sbuild, wb, dak and other tools to understand them too. Until one of those happens, an unofficial Architecture-Tags field that the Debian Janitor could use to update Architecture fields might work. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: packages expected to fail on some archs
Hi, Am 11.09.22 um 22:07 schrieb Adrian Bunk: On Sun, Sep 11, 2022 at 05:08:57PM +0200, Samuel Thibault wrote: ... The issue we see is that some DDs end up setting a hardcoded list in the "Architecture" field, rather than just letting builds keep failing on these archs (and then possibly succeeding after some time whenever somebody contributes a fix upstream that gets propagated to Debian). ... I'd say it is the best solution when a package needs non-trivial architecture-specific porting for every architecture it supports. With "non-trivial" I mean not just adding a new architecture to a few #ifdefs, but serious upstream porting efforts. E.g. valgrind does not support riscv64, and if it would ever gain the support in a new upstream version I'd expect the maintainer to add that to the Architecture field when upstream announces support for a new architecture. As a maintainer of such package I agree. I would not like it to fail when it is known that it won't work. Ever. Unless someone actually writes the matching asm part, knowing the ABI and calling convetions, etc. Regards, Reme
Re: packages expected to fail on some archs
On Sun, Sep 11, 2022 at 05:08:57PM +0200, Samuel Thibault wrote: >... > The issue we see is that some DDs end up setting a hardcoded list in > the "Architecture" field, rather than just letting builds keep failing > on these archs (and then possibly succeeding after some time whenever > somebody contributes a fix upstream that gets propagated to Debian). >... I'd say it is the best solution when a package needs non-trivial architecture-specific porting for every architecture it supports. With "non-trivial" I mean not just adding a new architecture to a few #ifdefs, but serious upstream porting efforts. E.g. valgrind does not support riscv64, and if it would ever gain the support in a new upstream version I'd expect the maintainer to add that to the Architecture field when upstream announces support for a new architecture. But Architecture lists for expressing e.g. "64bit" or "little endian" are a real pain for everyone working on bringup of a new port. Which happens far more often than most people realize. There is not only riscv64 (64bit, little endian). Ports is about to start building for arc (32bit, little endian). There are people working on ports like arm64be (64bit, big endian), loongarch64 (64bit, little endian) and many other ports that might never end up being built in the Debian infrastructure (but some of them might get built by derivatives). Architecture lists containing all 64bit ports or all little endian ports create much extra work for anyone adding support for a new 64bit little endian architecture. > Samuel cu Adrian
Re: packages expected to fail on some archs
On Sun, Sep 11, 2022 at 09:25:40PM +0200, Samuel Thibault wrote: > Paul Gevers, le dim. 11 sept. 2022 21:16:08 +0200, a ecrit: > > > > - color packages that "never" had a successful built on an architecture > > different. That information is already available because that's what marks > > the package as "uncompiled" vs "out-of-date". >... > That doesn't cover the case when a package stopped building on an arch, > though. In practive it does, when Paul wrote "never" that actually means "no older version is in the archive on that architecture". On release architectures people are usually fast with getting stale versions of no longer buildable packages removed since it prevents testing migration. > Samuel cu Adrian
Re: packages expected to fail on some archs
Paul Gevers, le dim. 11 sept. 2022 21:16:08 +0200, a ecrit: > On 11-09-2022 17:08, Samuel Thibault wrote: > > We could for instance: > > - Add an Architecture-FTBFS field to debian/control > > - Add an environment variable to debian/rules so that on these archs dh > >fails with a different exit code that buildds would notice. > > - Add a Architecture-FTBFS field in the wb database that DDs can set > > - color packages that "never" had a successful built on an architecture > different. That information is already available because that's what marks > the package as "uncompiled" vs "out-of-date". Oh, right, that information is already available in wb. Mehdi, do you think you can implement that easily for a start? That doesn't cover the case when a package stopped building on an arch, though. Samuel
Re: packages expected to fail on some archs
Hi Samuel, On 11-09-2022 17:08, Samuel Thibault wrote: We could for instance: - Add an Architecture-FTBFS field to debian/control - Add an environment variable to debian/rules so that on these archs dh fails with a different exit code that buildds would notice. - Add a Architecture-FTBFS field in the wb database that DDs can set - color packages that "never" had a successful built on an architecture different. That information is already available because that's what marks the package as "uncompiled" vs "out-of-date". I think it has added value if both cases (marked by the maintainer vs detected automatically) have value and could have different colors. Comparing it to autopkgtest, we currently color no-regression orange, while "neutral" (because of all kind of reasons, but the result of an actively set property) is blue. Paul OpenPGP_signature Description: OpenPGP digital signature
