Small update. Oracle/ MySQL **just** resigned their repo with a new key, with a new one. The new workaround is:
*RUN apt-key adv --keyserver keyserver.ubuntu.com <http://keyserver.ubuntu.com> --recv-keys B7B3B788A8D3785C && gpg --export "B7B3B788A8D3785C" > "/etc/apt/trusted.gpg.d/mysql.gpg"* However this key has 2 years expiry so in ~ 2 years we will have the same problem again and all our released images will have a similar problem if we do not switch to mariadb/remove the repository for MySQL (which might have side effects). *root@14002eba8b66:/opt/airflow# gpg --list-keys B7B3B788A8D3785Cpub rsa4096 2023-10-23 [SC] [expires: 2025-10-22] BCA43417C3B485DD128EC6D4B7B3B788A8D3785Cuid [ unknown] MySQL Release Engineering <[email protected] <[email protected]>>sub rsa4096 2023-10-23 [E] [expires: 2025-10-22]* So my take is - let's switch to MariaDB client. The Client is generally very thin and it follows well-defined and rather simple protocol, so I do not expect some "real" issues with it. I will send a LAZY CONSENSUS soon and PR with proposal of permanent fix J. On Fri, Dec 15, 2023 at 1:46 PM Jarek Potiuk <[email protected]> wrote: > > Isn't there the other option and that is to include the key for > verification. Thus pinning the certificate / key? This way it is more safe > to ignore the expiry date? > > Very good question. > > We already HAD the pinned certificate and that was part of the problem > (the expired key ID is hard-coded in all our past images). Pinning the key > is one thing and recommended, but `apt` will verify such a key (correctly) > anyway and when it is expired, apt will refuse it, unless you allow the > repo to be - basically - unverified. > > With Andrey we looked at other options but it seems there is no `easy` > way to **just** ignore expiry date for that certificate. But if someone > knows other ways to only ignore expiration, that would be best. We could > not find such a selective option. We **could** potentially temporary change > the container's system date during this installation step (and hard code it > to the day before yesterday for example), but it would be extremely lame > and there could be many side-effects in the future, so I personally rule it > out as an option. > > There are two ways to ignore an expired key. You can apply it per > repository (but it does not make them more secure this way): > > •Trusted (trusted) is a tri-state value which defaults to APT deciding if > a source is considered trusted or if warnings should be raised before e.g. > packages are installed from this source. This option can be used to > override that decision. The value yes tells APT always to consider this > source as trusted, even if it doesn't pass authentication checks. It > disables parts of apt-secure(8), and should therefore only be used in a > local and trusted context (if at all) as otherwise security is breached. > The value no does the opposite, causing the source to be handled as > untrusted even if the authentication checks passed successfully. The > default value can't be set explicitly. > > •Allow-Insecure (allow-insecure), Allow-Weak (allow-weak) and > Allow-Downgrade-To-Insecure (allow-downgrade-to-insecure) are boolean > values which all default to no. If set to yes they circumvent parts of > apt-secure(8) and should therefore not be used lightly! > > Both of them are security risks. > > And to add, how bad (and pretty unexpected and wrong) Oracle's policy is. > EVERYONE ELSE uses signing keys without expiry date, precisely to avoid > this kind of problem. MariaDB, Postgres, and even MSSQL keys have no expiry > date - I think because unlike Oracle, they care about their users so they > do not have to worry about rebuilding all their past images every 2 years. > It's actually a bad practice for apt packages to add expiry dates for > signing keys because it basically means you are putting expiry on your > packages to be installed. > > Also a digression here: this is also one of the weaknesses of the GPG > security scheme - and that's why modern replacements that are coming > (sigstore is the main contender) are way better because there, you keep the > public ledger of who signed what and the keys used to sign the packages are > generated temporarily to sign the packages and private keys are deleted > afterwards. This is a way better schema and everyone is moving there > (including us working with Apache Software Foundation and planning to > implement it next year), but at least for now we are stuck with what we > have, and signing keys for apt packages should have no expiry date. This is > a big flaw in Oracle/MySQL approach and to be honest, I kind of lost my > faith in them doing a good job on maintaining MySQL repos. > > J. > > > > > On Fri, Dec 15, 2023 at 1:17 PM Bolke de Bruin <[email protected]> wrote: > >> Isn't there the other option and that is to include the key for >> verification. Thus pinning the certificate / key? This way it is more safe >> to ignore the expiry date? >> >> I'm okay with all the other options just verifying if we are not >> overlooking anything. The reason for this is that we do not support >> MariaDB >> as a backend (afaik) but now we suddenly use the client as a dependency to >> talk to MySQL servers? >> >> My 2 cents, >> Bolke >> >> On Fri, 15 Dec 2023 at 12:36, Andrey Anshin <[email protected]> >> wrote: >> >> > I would rather stick to the Option 2, with small modifications: >> > By default use MariaDB libraries which compatible with MySQL, so change >> > everywhere INSTALL_MYSQL_CLIENT_TYPE=mariadb >> > - For x86_64 keep option to switch to Oracle MySQL libraries >> > - For ARM it would always forced (or until Oracle provide arm compatible >> > binaries for Debian) to mariad >> > >> > I think it is pretty safe to use it, because for a long time popular >> linux >> > distributions such as Debian, Ubuntu, RHEL, Centos provide MariaDB >> > libraries as compatible with MySQL. For example Debian Bookworm: >> > https://packages.debian.org/bookworm/default-mysql-client >> > >> > And this also make consistent ARM and x86_64 by default, and keep >> > possibility to build custom image with Oracle libraries >> > >> > --- >> > >> > Maybe someone who uses custom images for their Managed Airflow Services >> > could give info about what kind of libraries they use for MySQL? I >> assume >> > in AWS MWAA also use MariaDB (at least what i could see for >> local-runner) >> > but it my assumption which required confirmation before taking it into >> the >> > account. >> > >> > On Fri, 15 Dec 2023 at 14:54, Jarek Potiuk <[email protected]> wrote: >> > >> > > Hello everyone, >> > > >> > > (sorry for capital letters but we need to act quickly). >> > > >> > > *TL;DR; Due to Oracle breaking MySQL repo signing, Together with >> Andrey >> > we >> > > have a proposal to - immediately - switch to MariaDB clients in >> Airflow >> > > images for main and the upcoming Airflow 2.8.0 - giving the user an >> > option >> > > to build their own custom images with MySQL clients if they need >> > > compatibility. * >> > > >> > > Last night it turned out that Oracle failed miserably with managing >> > signing >> > > their repositories and their repositories are now badly signed with >> > expired >> > > keys since yesterday. They acknowledged the issue and they apparently >> > > started to fix it but we have no idea how long it will take. >> > > >> > > The problem is that the policy for signing the MySQL repos Oracle is >> > > essentially flawed - the expiry date is very short - and we will have >> to >> > > re-release all (!) our historical images - with removed Mysql >> repository. >> > > >> > > Currently none of our users can extend our images without applying a >> > > workaround where they remove the mysql repository manually or allow >> > > unsecure repositories. >> > > >> > > *Context*: >> > > >> > > Just about we should have our Airflow 2.8.0 release, we have a high >> > > severity breaking change caused by Oracle failing miserably in >> resigning >> > > their MySQL repositories. The key they used to sign the repositories >> > > expired yesterday. >> > > >> > > They have a new key released in October - that has expiry date in >> 2025 - >> > > but they forgot to re-sign their repositories with it and currently >> it is >> > > not possible to securely install MySQL packages via their APT >> > repositories >> > > at all. There is simply no way to do it securely (!). >> > > >> > > If they did it in October, we would have much more time to react and >> fix >> > > it, but they have not and now we are in a difficult situation (and it >> > will >> > > likely delay the 2.8.0 release). >> > > >> > > We track the issue in https://github.com/apache/airflow/issues/36231 >> > > >> > > There are two severe issues in MySQL bug system (I commented on both >> of >> > > them): >> > > >> > > https://bugs.mysql.com/bug.php?id=113427 >> > > https://bugs.mysql.com/bug.php?id=113428 >> > > >> > > Basically anyone installing MySQL from the official APT repos has this >> > > problem now. >> > > >> > > *Bigger Problem:* >> > > >> > > The problem is that the way Oracle policy is set (even if they would >> not >> > > forget to resign the repos), is flawed. It means that all our images >> that >> > > we release have fixed expiry date (2023-12-14 as it turned out) and we >> > will >> > > have to re-release the images again when they change the key and >> > basically >> > > it requires manually applying the new key when they re-sign the repos >> > > anyway. >> > > >> > > The current workaround is to add this by to their Dockerfile when they >> > are >> > > extending any of our past images: >> > > >> > > >> > > *RUN rm /etc/apt/sources.list.d/mysql.list* >> > > >> > > But this has side effects - for example if someone will run `apt >> upgrade` >> > > as part of their process, mysql clients will get installed from the >> > > "distro" packages - and they will be outdated. Another workaround is >> to >> > > add >> > > >> > > *RUN apt update --allow-insecure-repositories* >> > > >> > > But this has obvious issues with security and is not at all >> recommended. >> > > >> > > We will basically have to rebuild and re-release all our images in >> order >> > to >> > > fix the problem - regardless of what Oracle does. This is something we >> > > discuss with Andrey how to address it in the best way. >> > > >> > > *Options we have for now:* >> > > >> > > We discussed it with Andrey and we are discussing how to prevent it >> > > happening in the future but we have immediate issue to address now - >> for >> > > the upcoming 2.8.0 >> > > >> > > We have the following options: >> > > >> > > 1) We wait until Oracle fixes it and signs the repos with new keys. >> The >> > new >> > > keys are valid till 2025. However we can modify our dockerfile to >> remove >> > > the mysql repositories after installing mysql to avoid similar >> problems >> > in >> > > the future. They responded that they are uploading new keys but we are >> > not >> > > sure if we have >> > > >> > > 2) Thanks to what Andrey implemented a few months ago we have an easy >> way >> > > to switch to MariaDB clients. Those are binary compatible with MySQL >> and >> > > they are stable and production ready, and they pass all our tests in >> CI - >> > > https://github.com/apache/airflow/pull/36235 >> > > >> > > 3) Drop MySQL support. Joking here of course - though I suggested it >> in >> > the >> > > past a few times because of all the problems we have with MySQL and >> this >> > > one is by far the worst problem we and our users could have >> experienced). >> > > >> > > Option 1) is a bit "safer" in the "corporate" world - generally >> sticking >> > to >> > > the "official" clients of "official" DB that we support is the >> default. >> > > Though my trust with Oracle as MySQL steward had been greatly >> diminished >> > by >> > > that issue and the flawed approach for their policies. But we do not >> know >> > > how long it will take. >> > > >> > > Option 2) MariaDB is also very popular and solid and exhibits no such >> > > issues - their signing key is set to "no expiry" so if we switch to >> > > MariaDB, the problem will be gone. We run it for months for all our >> ARM >> > > images - and anyone using ARM Macbooks and running MySQL with Breeze >> > would >> > > use those. >> > > >> > > *Recommendation:* >> > > >> > > Both myself and Andrey's recommendation is to go to MariaDB for main >> and >> > > 2.8.0. We are pretty fed-up with a number of past issues we had to >> fight >> > > with - and even if Oracle fixes it now, we have no faith Oracle can be >> > > trusted as MySQL steward for the repos (Andrey did not have faith for >> > quite >> > > some time now and I lost it today). >> > > >> > > In order to respond to potential edge cases, we can also keep the >> option >> > > that users who really want it, might be able to rebuild the image with >> > > MySQL drivers - similarly as we give the users the option to use >> bullseye >> > > in 2.8.0. And we can even run a CI build with it to see that it >> continues >> > > to work. >> > > >> > > I believe this is the best option and we can get it implemented >> quickly >> > > today and get RC4 of Airflow with mariadb clients. If we see a >> > > support/consensus for the community we will **just** proceed with it >> and >> > > start a formal LAZY_CONSENSUS thread. >> > > >> > > WDYT? >> > > >> > > J. >> > > >> > >> >> >> -- >> >> -- >> Bolke de Bruin >> [email protected] >> >
