PR with complete fix for "main" and "2.8.0": https://github.com/apache/airflow/pull/36243
Once it passes all the tests and we will know the solution there works, I will also start removing the repos for old released images. J On Fri, Dec 15, 2023 at 3:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Small update. Oracle/ MySQL **just** resigned their repo with a new key, > with a new one. The new workaround is: > > > *RUN apt-key adv --keyserver keyserver.ubuntu.com > <http://keyserver.ubuntu.com> --recv-keys B7B3B788A8D3785C && gpg --export > "B7B3B788A8D3785C" > "/etc/apt/trusted.gpg.d/mysql.gpg"* > > However this key has 2 years expiry so in ~ 2 years we will have the same > problem again and all our released images will have a similar problem if we > do not switch to mariadb/remove the repository for MySQL (which might have > side effects). > > > > > > > *root@14002eba8b66:/opt/airflow# gpg --list-keys B7B3B788A8D3785Cpub > rsa4096 2023-10-23 [SC] [expires: 2025-10-22] > BCA43417C3B485DD128EC6D4B7B3B788A8D3785Cuid [ unknown] MySQL > Release Engineering <mysql-bu...@oss.oracle.com > <mysql-bu...@oss.oracle.com>>sub rsa4096 2023-10-23 [E] [expires: > 2025-10-22]* > > So my take is - let's switch to MariaDB client. The Client is generally > very thin and it follows well-defined and rather simple protocol, so I do > not expect some "real" issues with it. I will send a LAZY CONSENSUS soon > and PR with proposal of permanent fix > > J. > > > > On Fri, Dec 15, 2023 at 1:46 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > Isn't there the other option and that is to include the key for >> verification. Thus pinning the certificate / key? This way it is more safe >> to ignore the expiry date? >> >> Very good question. >> >> We already HAD the pinned certificate and that was part of the problem >> (the expired key ID is hard-coded in all our past images). Pinning the key >> is one thing and recommended, but `apt` will verify such a key (correctly) >> anyway and when it is expired, apt will refuse it, unless you allow the >> repo to be - basically - unverified. >> >> With Andrey we looked at other options but it seems there is no `easy` >> way to **just** ignore expiry date for that certificate. But if someone >> knows other ways to only ignore expiration, that would be best. We could >> not find such a selective option. We **could** potentially temporary change >> the container's system date during this installation step (and hard code it >> to the day before yesterday for example), but it would be extremely lame >> and there could be many side-effects in the future, so I personally rule it >> out as an option. >> >> There are two ways to ignore an expired key. You can apply it per >> repository (but it does not make them more secure this way): >> >> •Trusted (trusted) is a tri-state value which defaults to APT deciding if >> a source is considered trusted or if warnings should be raised before e.g. >> packages are installed from this source. This option can be used to >> override that decision. The value yes tells APT always to consider this >> source as trusted, even if it doesn't pass authentication checks. It >> disables parts of apt-secure(8), and should therefore only be used in a >> local and trusted context (if at all) as otherwise security is breached. >> The value no does the opposite, causing the source to be handled as >> untrusted even if the authentication checks passed successfully. The >> default value can't be set explicitly. >> >> •Allow-Insecure (allow-insecure), Allow-Weak (allow-weak) and >> Allow-Downgrade-To-Insecure (allow-downgrade-to-insecure) are boolean >> values which all default to no. If set to yes they circumvent parts of >> apt-secure(8) and should therefore not be used lightly! >> >> Both of them are security risks. >> >> And to add, how bad (and pretty unexpected and wrong) Oracle's policy is. >> EVERYONE ELSE uses signing keys without expiry date, precisely to avoid >> this kind of problem. MariaDB, Postgres, and even MSSQL keys have no expiry >> date - I think because unlike Oracle, they care about their users so they >> do not have to worry about rebuilding all their past images every 2 years. >> It's actually a bad practice for apt packages to add expiry dates for >> signing keys because it basically means you are putting expiry on your >> packages to be installed. >> >> Also a digression here: this is also one of the weaknesses of the GPG >> security scheme - and that's why modern replacements that are coming >> (sigstore is the main contender) are way better because there, you keep the >> public ledger of who signed what and the keys used to sign the packages are >> generated temporarily to sign the packages and private keys are deleted >> afterwards. This is a way better schema and everyone is moving there >> (including us working with Apache Software Foundation and planning to >> implement it next year), but at least for now we are stuck with what we >> have, and signing keys for apt packages should have no expiry date. This is >> a big flaw in Oracle/MySQL approach and to be honest, I kind of lost my >> faith in them doing a good job on maintaining MySQL repos. >> >> J. >> >> >> >> >> On Fri, Dec 15, 2023 at 1:17 PM Bolke de Bruin <bdbr...@gmail.com> wrote: >> >>> Isn't there the other option and that is to include the key for >>> verification. Thus pinning the certificate / key? This way it is more >>> safe >>> to ignore the expiry date? >>> >>> I'm okay with all the other options just verifying if we are not >>> overlooking anything. The reason for this is that we do not support >>> MariaDB >>> as a backend (afaik) but now we suddenly use the client as a dependency >>> to >>> talk to MySQL servers? >>> >>> My 2 cents, >>> Bolke >>> >>> On Fri, 15 Dec 2023 at 12:36, Andrey Anshin <andrey.ans...@taragol.is> >>> wrote: >>> >>> > I would rather stick to the Option 2, with small modifications: >>> > By default use MariaDB libraries which compatible with MySQL, so change >>> > everywhere INSTALL_MYSQL_CLIENT_TYPE=mariadb >>> > - For x86_64 keep option to switch to Oracle MySQL libraries >>> > - For ARM it would always forced (or until Oracle provide arm >>> compatible >>> > binaries for Debian) to mariad >>> > >>> > I think it is pretty safe to use it, because for a long time popular >>> linux >>> > distributions such as Debian, Ubuntu, RHEL, Centos provide MariaDB >>> > libraries as compatible with MySQL. For example Debian Bookworm: >>> > https://packages.debian.org/bookworm/default-mysql-client >>> > >>> > And this also make consistent ARM and x86_64 by default, and keep >>> > possibility to build custom image with Oracle libraries >>> > >>> > --- >>> > >>> > Maybe someone who uses custom images for their Managed Airflow Services >>> > could give info about what kind of libraries they use for MySQL? I >>> assume >>> > in AWS MWAA also use MariaDB (at least what i could see for >>> local-runner) >>> > but it my assumption which required confirmation before taking it into >>> the >>> > account. >>> > >>> > On Fri, 15 Dec 2023 at 14:54, Jarek Potiuk <ja...@potiuk.com> wrote: >>> > >>> > > Hello everyone, >>> > > >>> > > (sorry for capital letters but we need to act quickly). >>> > > >>> > > *TL;DR; Due to Oracle breaking MySQL repo signing, Together with >>> Andrey >>> > we >>> > > have a proposal to - immediately - switch to MariaDB clients in >>> Airflow >>> > > images for main and the upcoming Airflow 2.8.0 - giving the user an >>> > option >>> > > to build their own custom images with MySQL clients if they need >>> > > compatibility. * >>> > > >>> > > Last night it turned out that Oracle failed miserably with managing >>> > signing >>> > > their repositories and their repositories are now badly signed with >>> > expired >>> > > keys since yesterday. They acknowledged the issue and they apparently >>> > > started to fix it but we have no idea how long it will take. >>> > > >>> > > The problem is that the policy for signing the MySQL repos Oracle is >>> > > essentially flawed - the expiry date is very short - and we will >>> have to >>> > > re-release all (!) our historical images - with removed Mysql >>> repository. >>> > > >>> > > Currently none of our users can extend our images without applying a >>> > > workaround where they remove the mysql repository manually or allow >>> > > unsecure repositories. >>> > > >>> > > *Context*: >>> > > >>> > > Just about we should have our Airflow 2.8.0 release, we have a high >>> > > severity breaking change caused by Oracle failing miserably in >>> resigning >>> > > their MySQL repositories. The key they used to sign the repositories >>> > > expired yesterday. >>> > > >>> > > They have a new key released in October - that has expiry date in >>> 2025 - >>> > > but they forgot to re-sign their repositories with it and currently >>> it is >>> > > not possible to securely install MySQL packages via their APT >>> > repositories >>> > > at all. There is simply no way to do it securely (!). >>> > > >>> > > If they did it in October, we would have much more time to react and >>> fix >>> > > it, but they have not and now we are in a difficult situation (and it >>> > will >>> > > likely delay the 2.8.0 release). >>> > > >>> > > We track the issue in https://github.com/apache/airflow/issues/36231 >>> > > >>> > > There are two severe issues in MySQL bug system (I commented on both >>> of >>> > > them): >>> > > >>> > > https://bugs.mysql.com/bug.php?id=113427 >>> > > https://bugs.mysql.com/bug.php?id=113428 >>> > > >>> > > Basically anyone installing MySQL from the official APT repos has >>> this >>> > > problem now. >>> > > >>> > > *Bigger Problem:* >>> > > >>> > > The problem is that the way Oracle policy is set (even if they would >>> not >>> > > forget to resign the repos), is flawed. It means that all our images >>> that >>> > > we release have fixed expiry date (2023-12-14 as it turned out) and >>> we >>> > will >>> > > have to re-release the images again when they change the key and >>> > basically >>> > > it requires manually applying the new key when they re-sign the repos >>> > > anyway. >>> > > >>> > > The current workaround is to add this by to their Dockerfile when >>> they >>> > are >>> > > extending any of our past images: >>> > > >>> > > >>> > > *RUN rm /etc/apt/sources.list.d/mysql.list* >>> > > >>> > > But this has side effects - for example if someone will run `apt >>> upgrade` >>> > > as part of their process, mysql clients will get installed from the >>> > > "distro" packages - and they will be outdated. Another workaround >>> is to >>> > > add >>> > > >>> > > *RUN apt update --allow-insecure-repositories* >>> > > >>> > > But this has obvious issues with security and is not at all >>> recommended. >>> > > >>> > > We will basically have to rebuild and re-release all our images in >>> order >>> > to >>> > > fix the problem - regardless of what Oracle does. This is something >>> we >>> > > discuss with Andrey how to address it in the best way. >>> > > >>> > > *Options we have for now:* >>> > > >>> > > We discussed it with Andrey and we are discussing how to prevent it >>> > > happening in the future but we have immediate issue to address now - >>> for >>> > > the upcoming 2.8.0 >>> > > >>> > > We have the following options: >>> > > >>> > > 1) We wait until Oracle fixes it and signs the repos with new keys. >>> The >>> > new >>> > > keys are valid till 2025. However we can modify our dockerfile to >>> remove >>> > > the mysql repositories after installing mysql to avoid similar >>> problems >>> > in >>> > > the future. They responded that they are uploading new keys but we >>> are >>> > not >>> > > sure if we have >>> > > >>> > > 2) Thanks to what Andrey implemented a few months ago we have an >>> easy way >>> > > to switch to MariaDB clients. Those are binary compatible with MySQL >>> and >>> > > they are stable and production ready, and they pass all our tests in >>> CI - >>> > > https://github.com/apache/airflow/pull/36235 >>> > > >>> > > 3) Drop MySQL support. Joking here of course - though I suggested it >>> in >>> > the >>> > > past a few times because of all the problems we have with MySQL and >>> this >>> > > one is by far the worst problem we and our users could have >>> experienced). >>> > > >>> > > Option 1) is a bit "safer" in the "corporate" world - generally >>> sticking >>> > to >>> > > the "official" clients of "official" DB that we support is the >>> default. >>> > > Though my trust with Oracle as MySQL steward had been greatly >>> diminished >>> > by >>> > > that issue and the flawed approach for their policies. But we do not >>> know >>> > > how long it will take. >>> > > >>> > > Option 2) MariaDB is also very popular and solid and exhibits no such >>> > > issues - their signing key is set to "no expiry" so if we switch to >>> > > MariaDB, the problem will be gone. We run it for months for all our >>> ARM >>> > > images - and anyone using ARM Macbooks and running MySQL with Breeze >>> > would >>> > > use those. >>> > > >>> > > *Recommendation:* >>> > > >>> > > Both myself and Andrey's recommendation is to go to MariaDB for main >>> and >>> > > 2.8.0. We are pretty fed-up with a number of past issues we had to >>> fight >>> > > with - and even if Oracle fixes it now, we have no faith Oracle can >>> be >>> > > trusted as MySQL steward for the repos (Andrey did not have faith for >>> > quite >>> > > some time now and I lost it today). >>> > > >>> > > In order to respond to potential edge cases, we can also keep the >>> option >>> > > that users who really want it, might be able to rebuild the image >>> with >>> > > MySQL drivers - similarly as we give the users the option to use >>> bullseye >>> > > in 2.8.0. And we can even run a CI build with it to see that it >>> continues >>> > > to work. >>> > > >>> > > I believe this is the best option and we can get it implemented >>> quickly >>> > > today and get RC4 of Airflow with mariadb clients. If we see a >>> > > support/consensus for the community we will **just** proceed with it >>> and >>> > > start a formal LAZY_CONSENSUS thread. >>> > > >>> > > WDYT? >>> > > >>> > > J. >>> > > >>> > >>> >>> >>> -- >>> >>> -- >>> Bolke de Bruin >>> bdbr...@gmail.com >>> >>
