Re: deduplicating noarch subpackages
Josh Stone wrote: > On 2/13/20 7:26 AM, Kevin Kofler wrote: >> built only on one of them.) As far as I know, this was implemented that >> way to make QEMU firmwares work (which are built on and for a specific >> architecture, and then shipped as noarch packages for all of them so that >> the architecture can be emulated). > > Ah, I suppose you are referring to ipxe? I was mainly referring to all those BIOS equivalents that are used for qemu-system-* full system emulation. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
Le jeudi 13 février 2020 à 12:00 -0800, Josh Stone a écrit : > On 2/13/20 11:26 AM, Nicolas Mailhot wrote: > > Le jeudi 13 février 2020 à 09:59 -0800, Josh Stone a écrit : > > > > > It is so black and white. If you can not produce bit-perfect > > identical > > builds, don’t try to make the result noarch. > > AIUI, there's no requirement that noarch has to be perfectly > reproducible -- Koji explicitly ignores file size and checksum. Right, out infra has some loopholes right now, they allow the kind of breakage that the reproducible build people try to erradicate Anyway, sorry about the tone, I should not answer while battling upstream breakage. -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
On 2/13/20 11:26 AM, Nicolas Mailhot wrote: > Le jeudi 13 février 2020 à 09:59 -0800, Josh Stone a écrit : >> >> It's not so black and white. In theory, the only thing that should >> matter is the target triple, but the metadata also hashes the >> metadata >> of all its build dependencies. That in turn may include procedural >> macros (essentially compiler plugins) which are host binaries. The >> end >> result for the target should be the same, but it's a real concern for >> upstream that ignoring the host would break caching that relies on >> filename differences. > > It is so black and white. If you can not produce bit-perfect identical > builds, don’t try to make the result noarch. AIUI, there's no requirement that noarch has to be perfectly reproducible -- Koji explicitly ignores file size and checksum. > Neither koji, nor auditors, nor other third parties, can be expected to > check binary differences for you, and identify what differences matter > and what differences do not matter. > > Neither koji, nor auditors, nor other third parties, can be expected to > trust that you did this checking for every single build that differs. > > The *only* sane check strategy is a black and white equal/inequal test. > > And you can not handwave those concerns aways because you feel your > builds should be noarch (feeling is not implementable in software). I'm not waving it away. Believe it or not, I actually do understand the impact here. > When you write > >> it's a real concern for upstream that ignoring the host would break >> caching > > you are writing that upstream thinks the result is archful (because if > it actually were noarch, it would be perfectly fine to substitute one > build output for another, and there would be no caching concerns). If you read that upstream PR, you might understand my comment about caching better, and see that I'm also trying to find a better solution for those concerns. > The proof is in the pudding. noarch → binary differences do not matter, > they can be removed during the build process, and koji will be happy. > archful → removing differences will break something. > > And, removing differences does not necessarily mean removing things. > The reproducible build project didn’t remove timestamps. It just made > sure the same timestamp would be used for all builds. > > Regards, > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
Le jeudi 13 février 2020 à 09:59 -0800, Josh Stone a écrit : > > It's not so black and white. In theory, the only thing that should > matter is the target triple, but the metadata also hashes the > metadata > of all its build dependencies. That in turn may include procedural > macros (essentially compiler plugins) which are host binaries. The > end > result for the target should be the same, but it's a real concern for > upstream that ignoring the host would break caching that relies on > filename differences. It is so black and white. If you can not produce bit-perfect identical builds, don’t try to make the result noarch. Neither koji, nor auditors, nor other third parties, can be expected to check binary differences for you, and identify what differences matter and what differences do not matter. Neither koji, nor auditors, nor other third parties, can be expected to trust that you did this checking for every single build that differs. The *only* sane check strategy is a black and white equal/inequal test. And you can not handwave those concerns aways because you feel your builds should be noarch (feeling is not implementable in software). When you write > it's a real concern for upstream that ignoring the host would break > caching you are writing that upstream thinks the result is archful (because if it actually were noarch, it would be perfectly fine to substitute one build output for another, and there would be no caching concerns). The proof is in the pudding. noarch → binary differences do not matter, they can be removed during the build process, and koji will be happy. archful → removing differences will break something. And, removing differences does not necessarily mean removing things. The reproducible build project didn’t remove timestamps. It just made sure the same timestamp would be used for all builds. Regards, -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
On 2/12/20 8:29 AM, Adam Jackson wrote: > On Tue, 2020-02-11 at 16:21 -0800, Josh Stone wrote: > >> Another alternative is to try to remove the host information from the >> metadata hash, which I've already started upstream[3], but I'm not sure >> alleviate their concerns about caching and such. >> >> [3] https://github.com/rust-lang/cargo/pull/7873 >> >> Worst case, I could just build them as arch'ed subpackages, specific to >> the host which compiled them. > > I'd lean in favor of removing at least the architecture part of the > host triple from the hash. koji is still going to build those noarch > packages on every architecture and compare them, so if you do remove > the arch from the hash and they all compare equal, it can't have > mattered which arch you built it from. FWIW, I don't think Koji actually cares if the files are *identical*, as long as the file list is the same. The rpmdiff call says: # ignore differences in file size, md5sum, and mtime # (files may have been generated at build time and contain # embedded dates or other insignificant differences) d = koji.rpmdiff.Rpmdiff(joinpath(basepath, first_rpm), joinpath(basepath, other_rpm), ignore='S5TN') ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
On 2/12/20 12:45 AM, Nicolas Mailhot wrote: > Le 2020-02-12 01:21, Josh Stone a écrit : > >> The problem is that those cross-target libraries built by two different >> host arches will have different metadata hashes in the filenames, >> because the hash includes the full "rustc -Vv" version output, >> including >> the host triple. > > koji is doing the right thing. For audits and QA checks a noarch package > must be produced exactly the same ways regardless of the arch of the > builder. > > Either recording the triplet is meaningless, and you should inhibit > it(the same way reproducible builds inhibit date recording), or it > serves a purpose, and your builds are archfull by construction. It's not so black and white. In theory, the only thing that should matter is the target triple, but the metadata also hashes the metadata of all its build dependencies. That in turn may include procedural macros (essentially compiler plugins) which are host binaries. The end result for the target should be the same, but it's a real concern for upstream that ignoring the host would break caching that relies on filename differences. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
On 2/13/20 7:26 AM, Kevin Kofler wrote: > Neal Gompa wrote: >> My instinct is that this wouldn't work, but I'm not certain. Have you >> tried this change with a scratch build? Scratch builds run the same >> checks that normal builds do, and would be a good way to verify if >> your theory is true. OK, I ran a scratch build where only x86_64 built the noarch wasm target. I can't tell if koji actually ran the checks though. https://koji.fedoraproject.org/koji/taskinfo?taskID=41472922 > %ifarch-ing noarch subpackages (note: noarch SUBpackages of arch-dependent > packages) actually works and does the right thing in Koji. (Koji will still > copy them to all the architectures, even if they were built only on one of > them.) As far as I know, this was implemented that way to make QEMU > firmwares work (which are built on and for a specific architecture, and then > shipped as noarch packages for all of them so that the architecture can be > emulated). Ah, I suppose you are referring to ipxe? It looks like that package has always been built this way, as was gpxe before it. Even further back was etherboot which did this since etherboot-5.4.4-13.fc11, and its changelog for release 10 said, "Koji now supports noarch subpackages." So it seems this has worked nearly as long as we've had noarch at all. Thanks, it's useful to have history/precedent for this kind of design. > Back when we had secondary Koji instances, the secondary architecture people > used to complain about that practice because those noarch subpackages would > then be missing on their Koji instances. But now that we build alternative > architectures on the primary Koji, I do not see a good reason to not %ifarch > the noarch subpackages, at least in the cases where it works around a known > bogus comparison failure. (In the other cases, you may still want Koji to > actually do that comparison as a form of automated QA, even if it is > technically a waste of resources.) > > Building, e.g., noarch documentation subpackages only on a fast architecture > such as x86_64 also helps speeding up builds on slower architectures such as > armv7hl, without actually affecting their users (as per the first > paragraph). > > Kevin Kofler > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
Neal Gompa wrote: > My instinct is that this wouldn't work, but I'm not certain. Have you > tried this change with a scratch build? Scratch builds run the same > checks that normal builds do, and would be a good way to verify if > your theory is true. %ifarch-ing noarch subpackages (note: noarch SUBpackages of arch-dependent packages) actually works and does the right thing in Koji. (Koji will still copy them to all the architectures, even if they were built only on one of them.) As far as I know, this was implemented that way to make QEMU firmwares work (which are built on and for a specific architecture, and then shipped as noarch packages for all of them so that the architecture can be emulated). Back when we had secondary Koji instances, the secondary architecture people used to complain about that practice because those noarch subpackages would then be missing on their Koji instances. But now that we build alternative architectures on the primary Koji, I do not see a good reason to not %ifarch the noarch subpackages, at least in the cases where it works around a known bogus comparison failure. (In the other cases, you may still want Koji to actually do that comparison as a form of automated QA, even if it is technically a waste of resources.) Building, e.g., noarch documentation subpackages only on a fast architecture such as x86_64 also helps speeding up builds on slower architectures such as armv7hl, without actually affecting their users (as per the first paragraph). Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
On Tue, 2020-02-11 at 16:21 -0800, Josh Stone wrote: > Another alternative is to try to remove the host information from the > metadata hash, which I've already started upstream[3], but I'm not sure > alleviate their concerns about caching and such. > > [3] https://github.com/rust-lang/cargo/pull/7873 > > Worst case, I could just build them as arch'ed subpackages, specific to > the host which compiled them. I'd lean in favor of removing at least the architecture part of the host triple from the hash. koji is still going to build those noarch packages on every architecture and compare them, so if you do remove the arch from the hash and they all compare equal, it can't have mattered which arch you built it from. - ajax ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
Le 2020-02-12 01:21, Josh Stone a écrit : The problem is that those cross-target libraries built by two different host arches will have different metadata hashes in the filenames, because the hash includes the full "rustc -Vv" version output, including the host triple. koji is doing the right thing. For audits and QA checks a noarch package must be produced exactly the same ways regardless of the arch of the builder. Either recording the triplet is meaningless, and you should inhibit it(the same way reproducible builds inhibit date recording), or it serves a purpose, and your builds are archfull by construction. Regards, -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: deduplicating noarch subpackages
On Tue, Feb 11, 2020 at 7:22 PM Josh Stone wrote: > > When building a normal arch package with noarch subpackages, would it be > acceptable in Fedora if only *one* of the arches actually took > responsibility to build those noarch subpackages? > > Context: the "rust" package normally builds arch-specific packages, like > "rust" for rustc, rustdoc, and their libraries, and "rust-std-static" > for the standard library static archives (*.rlib). Since Rust has no > stable ABI, these are closely tied together, so they always need to be > built from the same srpm. > > I've had requests to include cross-target std builds for wasm[1] and > mingw[2] too. These would most naturally be noarch subpackages, > something like rust-std-static-$TRIPLE.noarch.rpm, and for the same ABI > reasons it should be built as part of the same rust srpm. > > [1] https://src.fedoraproject.org/rpms/rust/pull-request/6 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1762856 > > The problem is that those cross-target libraries built by two different > host arches will have different metadata hashes in the filenames, > because the hash includes the full "rustc -Vv" version output, including > the host triple. Koji will reject this when the overlapping noarch > packages are compared, since the file list doesn't match. The libraries > are still perfectly usable by rustc from a different host arch though. > > So my question/proposal is, what if I just build those cross-target > noarch subpackages from a single host (probably x86_64)? My reading of > the koji's "check_noarch_rpms" seems like it would be OK with this, as > there just wouldn't be any duplicates to rpmdiff. I worry that it would > be against policy somehow though... > > Another alternative is to try to remove the host information from the > metadata hash, which I've already started upstream[3], but I'm not sure > alleviate their concerns about caching and such. > > [3] https://github.com/rust-lang/cargo/pull/7873 > > Worst case, I could just build them as arch'ed subpackages, specific to > the host which compiled them. > > Thoughts? My instinct is that this wouldn't work, but I'm not certain. Have you tried this change with a scratch build? Scratch builds run the same checks that normal builds do, and would be a good way to verify if your theory is true. That said, this is really weird and I'd really be much more comfortable if this was fixed properly upstream first before you do this, as you've already started working on. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
