Re: Why we use specific compiler versions - was: Re: JDK-8284772 - was RE: [jdk17] RFR: 8269148: Update minor GCC version in GitHub Actions pipeline
On 2022-04-14 19:42, Andrew Hughes wrote: On 12:57 Wed 13 Apr , Magnus Ihse Bursie wrote: I disagree completely. We had it this way in mainline originally, but it was fixed in https://bugs.openjdk.java.net/browse/JDK-8256393. Prior to this patch, it seems there were no GCC version requirements. That's not what I'm suggesting. What I'm suggesting is that we replace gcc-10=10.2.0-5ubuntu1~20.04 (or whatever it is now) with just gcc-10. Is it possible to set the requirement to 10.2? That'd be okay for me, since it matches how we check for gcc versions in configure (currently it is at least "6.0"), and how we have traditionally described our requirements. Or are the only two options "just major version" or "major, minor, patch and ubuntu release"? :( /Magnus That still selects a specific major version of GCC. It is between major versions that we see new optimisations introduced, deprecations, etc. I would certainly not suggest that we allow it to be switched from e.g. GCC 10 to G11 behind our backs. I have dealt with such transitions in Fedora and know the changes they bring. What I don't see the advantage of is requiring a very specific package version. If this was OpenJDK, it would be like asking to build 8u with specifically 8u332-b05 rather than just 8u. I can't see that we would do that without a very good reason. I don't see such a good reason for requiring the same of GCC. These two are handled differently on the GCC development side too. Breakage is expected with the move to a new major version. This is why three stable versions are currently being maintained [0]. If breakage were to happen between minor releases of a stable version, however, that would be a bug. I've yet to see a case of it happening, though of course it's possible. As you might know, I'm not too fond of the GHA solution, since we can't debug issues with Github's hosts. Nevertheless, many users look at the GHA results as a way to sanity check their code. Any and all spurious build failures is a problem then, since it will present a red marker -- even if the new code in the PR is okay. This specific versioning is producing precisely these spurious failures. The reason I started digging into this was because my PR failed on Linux x86_64. There were no code changes in the PR (I was backporting the GHA setup to our Shenandoah fork of 8u). Having only just added support to 8u mainline, I found this very odd. It turns out it failed because it could no longer download the specific version of GCC. [1] I agree GHA can be painful to debug - it took me weeks to get 8u working in full - but it is useful for testing on architectures and operating systems one doesn't have easy access to. The less control we have over the build platform, the greater the chance for odd and non-reproducible build failures. Selecting a specific version of the compiler is a way to guarantee reproducible build results. If the build succeeds in mainline, and I submit correct code, chances are higher that the build also succeeds in my PR. In contrast, if the gcc version suddenly were changed behind my back, the mainline build might succeed, and my PR build fail, not due to anything I've done wrong, but just due to the fact that the compiler was switched by the Ubuntu team in the meantime. Yes, it means a bit more annoyance when upgrading the compiler, but that also means it is a conscious (and hopefully well tested) choice. I'll take that any day over re-introducing more uncertainty into an already-unstable testing procedure. As I say, I'm not suggesting we don't select a specific version, just that we are not *too* specific to the point we are constantly changing the version specification every time the Ubuntu maintainer fixes a typo. /Magnus [0] https://gcc.gnu.org/ [1] https://github.com/openjdk-bots/shenandoah-jdk8u/runs/5889665932?check_suite_focus=true Thanks
Re: Why we use specific compiler versions - was: Re: JDK-8284772 - was RE: [jdk17] RFR: 8269148: Update minor GCC version in GitHub Actions pipeline
On 12:57 Wed 13 Apr , Magnus Ihse Bursie wrote: > I disagree completely. We had it this way in mainline originally, but it > was fixed in https://bugs.openjdk.java.net/browse/JDK-8256393. > Prior to this patch, it seems there were no GCC version requirements. That's not what I'm suggesting. What I'm suggesting is that we replace gcc-10=10.2.0-5ubuntu1~20.04 (or whatever it is now) with just gcc-10. That still selects a specific major version of GCC. It is between major versions that we see new optimisations introduced, deprecations, etc. I would certainly not suggest that we allow it to be switched from e.g. GCC 10 to G11 behind our backs. I have dealt with such transitions in Fedora and know the changes they bring. What I don't see the advantage of is requiring a very specific package version. If this was OpenJDK, it would be like asking to build 8u with specifically 8u332-b05 rather than just 8u. I can't see that we would do that without a very good reason. I don't see such a good reason for requiring the same of GCC. These two are handled differently on the GCC development side too. Breakage is expected with the move to a new major version. This is why three stable versions are currently being maintained [0]. If breakage were to happen between minor releases of a stable version, however, that would be a bug. I've yet to see a case of it happening, though of course it's possible. > As you might know, I'm not too fond of the GHA solution, since we can't > debug issues with Github's hosts. Nevertheless, many users look at the > GHA results as a way to sanity check their code. Any and all spurious > build failures is a problem then, since it will present a red marker -- > even if the new code in the PR is okay. This specific versioning is producing precisely these spurious failures. The reason I started digging into this was because my PR failed on Linux x86_64. There were no code changes in the PR (I was backporting the GHA setup to our Shenandoah fork of 8u). Having only just added support to 8u mainline, I found this very odd. It turns out it failed because it could no longer download the specific version of GCC. [1] I agree GHA can be painful to debug - it took me weeks to get 8u working in full - but it is useful for testing on architectures and operating systems one doesn't have easy access to. > The less control we have over the build platform, the greater the > chance for odd and non-reproducible build failures. Selecting a > specific version of the compiler is a way to guarantee reproducible > build results. If the build succeeds in mainline, and I submit > correct code, chances are higher that the build also succeeds in my > PR. In contrast, if the gcc version suddenly were changed behind my > back, the mainline build might succeed, and my PR build fail, not > due to anything I've done wrong, but just due to the fact that the > compiler was switched by the Ubuntu team in the meantime. Yes, it > means a bit more annoyance when upgrading the compiler, but that > also means it is a conscious (and hopefully well tested) > choice. I'll take that any day over re-introducing more uncertainty > into an already-unstable testing procedure. As I say, I'm not suggesting we don't select a specific version, just that we are not *too* specific to the point we are constantly changing the version specification every time the Ubuntu maintainer fixes a typo. /Magnus [0] https://gcc.gnu.org/ [1] https://github.com/openjdk-bots/shenandoah-jdk8u/runs/5889665932?check_suite_focus=true Thanks -- Andrew :) Pronouns: he / him or they / them Senior Free Java Software Engineer OpenJDK Package Owner Red Hat, Inc. (http://www.redhat.com) PGP Key: ed25519/0xCFDA0F9B35964222 (hkp://keys.gnupg.net) Fingerprint = 5132 579D D154 0ED2 3E04 C5A0 CFDA 0F9B 3596 4222
RE: Why we use specific compiler versions - was: Re: JDK-8284772 - was RE: [jdk17] RFR: 8269148: Update minor GCC version in GitHub Actions pipeline
Hi Magnus, thanks for pointing to the original change that introduced this. Generally I agree that the build environment should be stable and only consciously be updated to newer versions of tools. E.g. the Ubuntu Release and the gcc major version should remain stable. However, this case it is just about changing minor steppings due to vendor packaging in a scenario where pinning to a certain version could fail out of a sudden because the old vendor package is removed. My personal feeling here is that the tradeoff to unpin the package name is the better choice. As for stability of the environment I think that also the ubuntu container that is used is updated from time to time for certain patches and we usually don't recognize it. Best regards Christoph > -Original Message- > From: Magnus Ihse Bursie > Sent: Mittwoch, 13. April 2022 12:58 > To: Langer, Christoph ; Andrew Hughes > ; David Holmes ; > Aleksey Shipilev > Cc: build-dev@openjdk.java.net > Subject: Why we use specific compiler versions - was: Re: JDK-8284772 - was > RE: [jdk17] RFR: 8269148: Update minor GCC version in GitHub Actions > pipeline > > I disagree completely. We had it this way in mainline originally, but it was > fixed > in https://bugs.openjdk.java.net/browse/JDK-8256393. > > As you might know, I'm not too fond of the GHA solution, since we can't debug > issues with Github's hosts. Nevertheless, many users look at the GHA results > as > a way to sanity check their code. Any and all spurious build failures is a > problem > then, since it will present a red marker -- even if the new code in the PR is > okay. > > The less control we have over the build platform, the greater the chance for > odd and non-reproducible build failures. > > Selecting a specific version of the compiler is a way to guarantee > reproducible > build results. If the build succeeds in mainline, and I submit correct code, > chances are higher that the build also succeeds in my PR. In contrast, if the > gcc > version suddenly were changed behind my back, the mainline build might > succeed, and my PR build fail, not due to anything I've done wrong, but just > due > to the fact that the compiler was switched by the Ubuntu team in the > meantime. > > Yes, it means a bit more annoyance when upgrading the compiler, but that also > means it is a conscious (and hopefully well tested) choice. I'll take that > any day > over re-introducing more uncertainty into an already-unstable testing > procedure. > > /Magnus > > On 2022-04-13 08:25, Langer, Christoph wrote: > > Hi Andrew, > > > >>>> One dummy question: > >>>> Why do we need to specify the real package name here? > >>>> If we install gcc-10, I think apt system will pick up the latest gcc-10 > >>>> for us. > >>> IIRC the intent is to keep control over the gcc version and not > >>> randomly update whenever the distro updates. Upgrading compiler > >>> versions for the OpenJDK is actually a very involved process when > >>> done properly and we often find code changes need to be made, or > >>> warnings adjusted, when a new version of the compiler is to be used. > >>> This approach forces us to check the new version is okay before > >>> switching over to it. At least that is the theory. :) > >>> > >>> Cheers, > >>> David > >>> > >>> > >> I'm in complete agreement with you as regards major versions of GCC. > >> Fedora's eager adoption of them means we often encounter these early. > >> JDK-8282231 is just the latest example from the introduction of GCC 12. > >> > >> However, the GHA workflow in OpenJDK doesn't just depend on a major > >> version of GCC - which is actually contained in the Ubuntu package > >> name of > >> gcc-9 or gcc-10 itself - but the full revision number, even down to > >> local packaging changes. > >> > >> I believe this is overkill and leads to valuable time being wasted on > >> issues like > >> JDK-8283778 where the GCC version itself didn't even change at all, > >> just the Ubuntu version suffix. > >> > >> Having just encountered this with 8u, I've filed JDK-8284772 there to > >> just use the package name, which includes the major GCC version. > >> That's already how it is depending on the x86_32 GCC, which hasn't > >> suffered broken dependencies in the same way as x86_64. > >> > >> I have yet to see an issue be specific to a minor GCC version bump, > >> whereas the current setup is pretty much guaranteed to mean furthe
