> Isn't there the other option and that is to include the key for verification. Thus pinning the certificate / key? This way it is more safe to ignore the expiry date?
Very good question. We already HAD the pinned certificate and that was part of the problem (the expired key ID is hard-coded in all our past images). Pinning the key is one thing and recommended, but `apt` will verify such a key (correctly) anyway and when it is expired, apt will refuse it, unless you allow the repo to be - basically - unverified. With Andrey we looked at other options but it seems there is no `easy` way to **just** ignore expiry date for that certificate. But if someone knows other ways to only ignore expiration, that would be best. We could not find such a selective option. We **could** potentially temporary change the container's system date during this installation step (and hard code it to the day before yesterday for example), but it would be extremely lame and there could be many side-effects in the future, so I personally rule it out as an option. There are two ways to ignore an expired key. You can apply it per repository (but it does not make them more secure this way): •Trusted (trusted) is a tri-state value which defaults to APT deciding if a source is considered trusted or if warnings should be raised before e.g. packages are installed from this source. This option can be used to override that decision. The value yes tells APT always to consider this source as trusted, even if it doesn't pass authentication checks. It disables parts of apt-secure(8), and should therefore only be used in a local and trusted context (if at all) as otherwise security is breached. The value no does the opposite, causing the source to be handled as untrusted even if the authentication checks passed successfully. The default value can't be set explicitly. •Allow-Insecure (allow-insecure), Allow-Weak (allow-weak) and Allow-Downgrade-To-Insecure (allow-downgrade-to-insecure) are boolean values which all default to no. If set to yes they circumvent parts of apt-secure(8) and should therefore not be used lightly! Both of them are security risks. And to add, how bad (and pretty unexpected and wrong) Oracle's policy is. EVERYONE ELSE uses signing keys without expiry date, precisely to avoid this kind of problem. MariaDB, Postgres, and even MSSQL keys have no expiry date - I think because unlike Oracle, they care about their users so they do not have to worry about rebuilding all their past images every 2 years. It's actually a bad practice for apt packages to add expiry dates for signing keys because it basically means you are putting expiry on your packages to be installed. Also a digression here: this is also one of the weaknesses of the GPG security scheme - and that's why modern replacements that are coming (sigstore is the main contender) are way better because there, you keep the public ledger of who signed what and the keys used to sign the packages are generated temporarily to sign the packages and private keys are deleted afterwards. This is a way better schema and everyone is moving there (including us working with Apache Software Foundation and planning to implement it next year), but at least for now we are stuck with what we have, and signing keys for apt packages should have no expiry date. This is a big flaw in Oracle/MySQL approach and to be honest, I kind of lost my faith in them doing a good job on maintaining MySQL repos. J. On Fri, Dec 15, 2023 at 1:17 PM Bolke de Bruin <bdbr...@gmail.com> wrote: > Isn't there the other option and that is to include the key for > verification. Thus pinning the certificate / key? This way it is more safe > to ignore the expiry date? > > I'm okay with all the other options just verifying if we are not > overlooking anything. The reason for this is that we do not support MariaDB > as a backend (afaik) but now we suddenly use the client as a dependency to > talk to MySQL servers? > > My 2 cents, > Bolke > > On Fri, 15 Dec 2023 at 12:36, Andrey Anshin <andrey.ans...@taragol.is> > wrote: > > > I would rather stick to the Option 2, with small modifications: > > By default use MariaDB libraries which compatible with MySQL, so change > > everywhere INSTALL_MYSQL_CLIENT_TYPE=mariadb > > - For x86_64 keep option to switch to Oracle MySQL libraries > > - For ARM it would always forced (or until Oracle provide arm compatible > > binaries for Debian) to mariad > > > > I think it is pretty safe to use it, because for a long time popular > linux > > distributions such as Debian, Ubuntu, RHEL, Centos provide MariaDB > > libraries as compatible with MySQL. For example Debian Bookworm: > > https://packages.debian.org/bookworm/default-mysql-client > > > > And this also make consistent ARM and x86_64 by default, and keep > > possibility to build custom image with Oracle libraries > > > > --- > > > > Maybe someone who uses custom images for their Managed Airflow Services > > could give info about what kind of libraries they use for MySQL? I assume > > in AWS MWAA also use MariaDB (at least what i could see for local-runner) > > but it my assumption which required confirmation before taking it into > the > > account. > > > > On Fri, 15 Dec 2023 at 14:54, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Hello everyone, > > > > > > (sorry for capital letters but we need to act quickly). > > > > > > *TL;DR; Due to Oracle breaking MySQL repo signing, Together with Andrey > > we > > > have a proposal to - immediately - switch to MariaDB clients in Airflow > > > images for main and the upcoming Airflow 2.8.0 - giving the user an > > option > > > to build their own custom images with MySQL clients if they need > > > compatibility. * > > > > > > Last night it turned out that Oracle failed miserably with managing > > signing > > > their repositories and their repositories are now badly signed with > > expired > > > keys since yesterday. They acknowledged the issue and they apparently > > > started to fix it but we have no idea how long it will take. > > > > > > The problem is that the policy for signing the MySQL repos Oracle is > > > essentially flawed - the expiry date is very short - and we will have > to > > > re-release all (!) our historical images - with removed Mysql > repository. > > > > > > Currently none of our users can extend our images without applying a > > > workaround where they remove the mysql repository manually or allow > > > unsecure repositories. > > > > > > *Context*: > > > > > > Just about we should have our Airflow 2.8.0 release, we have a high > > > severity breaking change caused by Oracle failing miserably in > resigning > > > their MySQL repositories. The key they used to sign the repositories > > > expired yesterday. > > > > > > They have a new key released in October - that has expiry date in 2025 > - > > > but they forgot to re-sign their repositories with it and currently it > is > > > not possible to securely install MySQL packages via their APT > > repositories > > > at all. There is simply no way to do it securely (!). > > > > > > If they did it in October, we would have much more time to react and > fix > > > it, but they have not and now we are in a difficult situation (and it > > will > > > likely delay the 2.8.0 release). > > > > > > We track the issue in https://github.com/apache/airflow/issues/36231 > > > > > > There are two severe issues in MySQL bug system (I commented on both of > > > them): > > > > > > https://bugs.mysql.com/bug.php?id=113427 > > > https://bugs.mysql.com/bug.php?id=113428 > > > > > > Basically anyone installing MySQL from the official APT repos has this > > > problem now. > > > > > > *Bigger Problem:* > > > > > > The problem is that the way Oracle policy is set (even if they would > not > > > forget to resign the repos), is flawed. It means that all our images > that > > > we release have fixed expiry date (2023-12-14 as it turned out) and we > > will > > > have to re-release the images again when they change the key and > > basically > > > it requires manually applying the new key when they re-sign the repos > > > anyway. > > > > > > The current workaround is to add this by to their Dockerfile when they > > are > > > extending any of our past images: > > > > > > > > > *RUN rm /etc/apt/sources.list.d/mysql.list* > > > > > > But this has side effects - for example if someone will run `apt > upgrade` > > > as part of their process, mysql clients will get installed from the > > > "distro" packages - and they will be outdated. Another workaround is > to > > > add > > > > > > *RUN apt update --allow-insecure-repositories* > > > > > > But this has obvious issues with security and is not at all > recommended. > > > > > > We will basically have to rebuild and re-release all our images in > order > > to > > > fix the problem - regardless of what Oracle does. This is something we > > > discuss with Andrey how to address it in the best way. > > > > > > *Options we have for now:* > > > > > > We discussed it with Andrey and we are discussing how to prevent it > > > happening in the future but we have immediate issue to address now - > for > > > the upcoming 2.8.0 > > > > > > We have the following options: > > > > > > 1) We wait until Oracle fixes it and signs the repos with new keys. The > > new > > > keys are valid till 2025. However we can modify our dockerfile to > remove > > > the mysql repositories after installing mysql to avoid similar problems > > in > > > the future. They responded that they are uploading new keys but we are > > not > > > sure if we have > > > > > > 2) Thanks to what Andrey implemented a few months ago we have an easy > way > > > to switch to MariaDB clients. Those are binary compatible with MySQL > and > > > they are stable and production ready, and they pass all our tests in > CI - > > > https://github.com/apache/airflow/pull/36235 > > > > > > 3) Drop MySQL support. Joking here of course - though I suggested it in > > the > > > past a few times because of all the problems we have with MySQL and > this > > > one is by far the worst problem we and our users could have > experienced). > > > > > > Option 1) is a bit "safer" in the "corporate" world - generally > sticking > > to > > > the "official" clients of "official" DB that we support is the default. > > > Though my trust with Oracle as MySQL steward had been greatly > diminished > > by > > > that issue and the flawed approach for their policies. But we do not > know > > > how long it will take. > > > > > > Option 2) MariaDB is also very popular and solid and exhibits no such > > > issues - their signing key is set to "no expiry" so if we switch to > > > MariaDB, the problem will be gone. We run it for months for all our ARM > > > images - and anyone using ARM Macbooks and running MySQL with Breeze > > would > > > use those. > > > > > > *Recommendation:* > > > > > > Both myself and Andrey's recommendation is to go to MariaDB for main > and > > > 2.8.0. We are pretty fed-up with a number of past issues we had to > fight > > > with - and even if Oracle fixes it now, we have no faith Oracle can be > > > trusted as MySQL steward for the repos (Andrey did not have faith for > > quite > > > some time now and I lost it today). > > > > > > In order to respond to potential edge cases, we can also keep the > option > > > that users who really want it, might be able to rebuild the image with > > > MySQL drivers - similarly as we give the users the option to use > bullseye > > > in 2.8.0. And we can even run a CI build with it to see that it > continues > > > to work. > > > > > > I believe this is the best option and we can get it implemented quickly > > > today and get RC4 of Airflow with mariadb clients. If we see a > > > support/consensus for the community we will **just** proceed with it > and > > > start a formal LAZY_CONSENSUS thread. > > > > > > WDYT? > > > > > > J. > > > > > > > > -- > > -- > Bolke de Bruin > bdbr...@gmail.com >
