Isn't there the other option and that is to include the key for verification. Thus pinning the certificate / key? This way it is more safe to ignore the expiry date?
I'm okay with all the other options just verifying if we are not overlooking anything. The reason for this is that we do not support MariaDB as a backend (afaik) but now we suddenly use the client as a dependency to talk to MySQL servers? My 2 cents, Bolke On Fri, 15 Dec 2023 at 12:36, Andrey Anshin <andrey.ans...@taragol.is> wrote: > I would rather stick to the Option 2, with small modifications: > By default use MariaDB libraries which compatible with MySQL, so change > everywhere INSTALL_MYSQL_CLIENT_TYPE=mariadb > - For x86_64 keep option to switch to Oracle MySQL libraries > - For ARM it would always forced (or until Oracle provide arm compatible > binaries for Debian) to mariad > > I think it is pretty safe to use it, because for a long time popular linux > distributions such as Debian, Ubuntu, RHEL, Centos provide MariaDB > libraries as compatible with MySQL. For example Debian Bookworm: > https://packages.debian.org/bookworm/default-mysql-client > > And this also make consistent ARM and x86_64 by default, and keep > possibility to build custom image with Oracle libraries > > --- > > Maybe someone who uses custom images for their Managed Airflow Services > could give info about what kind of libraries they use for MySQL? I assume > in AWS MWAA also use MariaDB (at least what i could see for local-runner) > but it my assumption which required confirmation before taking it into the > account. > > On Fri, 15 Dec 2023 at 14:54, Jarek Potiuk <ja...@potiuk.com> wrote: > > > Hello everyone, > > > > (sorry for capital letters but we need to act quickly). > > > > *TL;DR; Due to Oracle breaking MySQL repo signing, Together with Andrey > we > > have a proposal to - immediately - switch to MariaDB clients in Airflow > > images for main and the upcoming Airflow 2.8.0 - giving the user an > option > > to build their own custom images with MySQL clients if they need > > compatibility. * > > > > Last night it turned out that Oracle failed miserably with managing > signing > > their repositories and their repositories are now badly signed with > expired > > keys since yesterday. They acknowledged the issue and they apparently > > started to fix it but we have no idea how long it will take. > > > > The problem is that the policy for signing the MySQL repos Oracle is > > essentially flawed - the expiry date is very short - and we will have to > > re-release all (!) our historical images - with removed Mysql repository. > > > > Currently none of our users can extend our images without applying a > > workaround where they remove the mysql repository manually or allow > > unsecure repositories. > > > > *Context*: > > > > Just about we should have our Airflow 2.8.0 release, we have a high > > severity breaking change caused by Oracle failing miserably in resigning > > their MySQL repositories. The key they used to sign the repositories > > expired yesterday. > > > > They have a new key released in October - that has expiry date in 2025 - > > but they forgot to re-sign their repositories with it and currently it is > > not possible to securely install MySQL packages via their APT > repositories > > at all. There is simply no way to do it securely (!). > > > > If they did it in October, we would have much more time to react and fix > > it, but they have not and now we are in a difficult situation (and it > will > > likely delay the 2.8.0 release). > > > > We track the issue in https://github.com/apache/airflow/issues/36231 > > > > There are two severe issues in MySQL bug system (I commented on both of > > them): > > > > https://bugs.mysql.com/bug.php?id=113427 > > https://bugs.mysql.com/bug.php?id=113428 > > > > Basically anyone installing MySQL from the official APT repos has this > > problem now. > > > > *Bigger Problem:* > > > > The problem is that the way Oracle policy is set (even if they would not > > forget to resign the repos), is flawed. It means that all our images that > > we release have fixed expiry date (2023-12-14 as it turned out) and we > will > > have to re-release the images again when they change the key and > basically > > it requires manually applying the new key when they re-sign the repos > > anyway. > > > > The current workaround is to add this by to their Dockerfile when they > are > > extending any of our past images: > > > > > > *RUN rm /etc/apt/sources.list.d/mysql.list* > > > > But this has side effects - for example if someone will run `apt upgrade` > > as part of their process, mysql clients will get installed from the > > "distro" packages - and they will be outdated. Another workaround is to > > add > > > > *RUN apt update --allow-insecure-repositories* > > > > But this has obvious issues with security and is not at all recommended. > > > > We will basically have to rebuild and re-release all our images in order > to > > fix the problem - regardless of what Oracle does. This is something we > > discuss with Andrey how to address it in the best way. > > > > *Options we have for now:* > > > > We discussed it with Andrey and we are discussing how to prevent it > > happening in the future but we have immediate issue to address now - for > > the upcoming 2.8.0 > > > > We have the following options: > > > > 1) We wait until Oracle fixes it and signs the repos with new keys. The > new > > keys are valid till 2025. However we can modify our dockerfile to remove > > the mysql repositories after installing mysql to avoid similar problems > in > > the future. They responded that they are uploading new keys but we are > not > > sure if we have > > > > 2) Thanks to what Andrey implemented a few months ago we have an easy way > > to switch to MariaDB clients. Those are binary compatible with MySQL and > > they are stable and production ready, and they pass all our tests in CI - > > https://github.com/apache/airflow/pull/36235 > > > > 3) Drop MySQL support. Joking here of course - though I suggested it in > the > > past a few times because of all the problems we have with MySQL and this > > one is by far the worst problem we and our users could have experienced). > > > > Option 1) is a bit "safer" in the "corporate" world - generally sticking > to > > the "official" clients of "official" DB that we support is the default. > > Though my trust with Oracle as MySQL steward had been greatly diminished > by > > that issue and the flawed approach for their policies. But we do not know > > how long it will take. > > > > Option 2) MariaDB is also very popular and solid and exhibits no such > > issues - their signing key is set to "no expiry" so if we switch to > > MariaDB, the problem will be gone. We run it for months for all our ARM > > images - and anyone using ARM Macbooks and running MySQL with Breeze > would > > use those. > > > > *Recommendation:* > > > > Both myself and Andrey's recommendation is to go to MariaDB for main and > > 2.8.0. We are pretty fed-up with a number of past issues we had to fight > > with - and even if Oracle fixes it now, we have no faith Oracle can be > > trusted as MySQL steward for the repos (Andrey did not have faith for > quite > > some time now and I lost it today). > > > > In order to respond to potential edge cases, we can also keep the option > > that users who really want it, might be able to rebuild the image with > > MySQL drivers - similarly as we give the users the option to use bullseye > > in 2.8.0. And we can even run a CI build with it to see that it continues > > to work. > > > > I believe this is the best option and we can get it implemented quickly > > today and get RC4 of Airflow with mariadb clients. If we see a > > support/consensus for the community we will **just** proceed with it and > > start a formal LAZY_CONSENSUS thread. > > > > WDYT? > > > > J. > > > -- -- Bolke de Bruin bdbr...@gmail.com
