I would rather stick to the Option 2, with small modifications: By default use MariaDB libraries which compatible with MySQL, so change everywhere INSTALL_MYSQL_CLIENT_TYPE=mariadb - For x86_64 keep option to switch to Oracle MySQL libraries - For ARM it would always forced (or until Oracle provide arm compatible binaries for Debian) to mariad
I think it is pretty safe to use it, because for a long time popular linux distributions such as Debian, Ubuntu, RHEL, Centos provide MariaDB libraries as compatible with MySQL. For example Debian Bookworm: https://packages.debian.org/bookworm/default-mysql-client And this also make consistent ARM and x86_64 by default, and keep possibility to build custom image with Oracle libraries --- Maybe someone who uses custom images for their Managed Airflow Services could give info about what kind of libraries they use for MySQL? I assume in AWS MWAA also use MariaDB (at least what i could see for local-runner) but it my assumption which required confirmation before taking it into the account. On Fri, 15 Dec 2023 at 14:54, Jarek Potiuk <ja...@potiuk.com> wrote: > Hello everyone, > > (sorry for capital letters but we need to act quickly). > > *TL;DR; Due to Oracle breaking MySQL repo signing, Together with Andrey we > have a proposal to - immediately - switch to MariaDB clients in Airflow > images for main and the upcoming Airflow 2.8.0 - giving the user an option > to build their own custom images with MySQL clients if they need > compatibility. * > > Last night it turned out that Oracle failed miserably with managing signing > their repositories and their repositories are now badly signed with expired > keys since yesterday. They acknowledged the issue and they apparently > started to fix it but we have no idea how long it will take. > > The problem is that the policy for signing the MySQL repos Oracle is > essentially flawed - the expiry date is very short - and we will have to > re-release all (!) our historical images - with removed Mysql repository. > > Currently none of our users can extend our images without applying a > workaround where they remove the mysql repository manually or allow > unsecure repositories. > > *Context*: > > Just about we should have our Airflow 2.8.0 release, we have a high > severity breaking change caused by Oracle failing miserably in resigning > their MySQL repositories. The key they used to sign the repositories > expired yesterday. > > They have a new key released in October - that has expiry date in 2025 - > but they forgot to re-sign their repositories with it and currently it is > not possible to securely install MySQL packages via their APT repositories > at all. There is simply no way to do it securely (!). > > If they did it in October, we would have much more time to react and fix > it, but they have not and now we are in a difficult situation (and it will > likely delay the 2.8.0 release). > > We track the issue in https://github.com/apache/airflow/issues/36231 > > There are two severe issues in MySQL bug system (I commented on both of > them): > > https://bugs.mysql.com/bug.php?id=113427 > https://bugs.mysql.com/bug.php?id=113428 > > Basically anyone installing MySQL from the official APT repos has this > problem now. > > *Bigger Problem:* > > The problem is that the way Oracle policy is set (even if they would not > forget to resign the repos), is flawed. It means that all our images that > we release have fixed expiry date (2023-12-14 as it turned out) and we will > have to re-release the images again when they change the key and basically > it requires manually applying the new key when they re-sign the repos > anyway. > > The current workaround is to add this by to their Dockerfile when they are > extending any of our past images: > > > *RUN rm /etc/apt/sources.list.d/mysql.list* > > But this has side effects - for example if someone will run `apt upgrade` > as part of their process, mysql clients will get installed from the > "distro" packages - and they will be outdated. Another workaround is to > add > > *RUN apt update --allow-insecure-repositories* > > But this has obvious issues with security and is not at all recommended. > > We will basically have to rebuild and re-release all our images in order to > fix the problem - regardless of what Oracle does. This is something we > discuss with Andrey how to address it in the best way. > > *Options we have for now:* > > We discussed it with Andrey and we are discussing how to prevent it > happening in the future but we have immediate issue to address now - for > the upcoming 2.8.0 > > We have the following options: > > 1) We wait until Oracle fixes it and signs the repos with new keys. The new > keys are valid till 2025. However we can modify our dockerfile to remove > the mysql repositories after installing mysql to avoid similar problems in > the future. They responded that they are uploading new keys but we are not > sure if we have > > 2) Thanks to what Andrey implemented a few months ago we have an easy way > to switch to MariaDB clients. Those are binary compatible with MySQL and > they are stable and production ready, and they pass all our tests in CI - > https://github.com/apache/airflow/pull/36235 > > 3) Drop MySQL support. Joking here of course - though I suggested it in the > past a few times because of all the problems we have with MySQL and this > one is by far the worst problem we and our users could have experienced). > > Option 1) is a bit "safer" in the "corporate" world - generally sticking to > the "official" clients of "official" DB that we support is the default. > Though my trust with Oracle as MySQL steward had been greatly diminished by > that issue and the flawed approach for their policies. But we do not know > how long it will take. > > Option 2) MariaDB is also very popular and solid and exhibits no such > issues - their signing key is set to "no expiry" so if we switch to > MariaDB, the problem will be gone. We run it for months for all our ARM > images - and anyone using ARM Macbooks and running MySQL with Breeze would > use those. > > *Recommendation:* > > Both myself and Andrey's recommendation is to go to MariaDB for main and > 2.8.0. We are pretty fed-up with a number of past issues we had to fight > with - and even if Oracle fixes it now, we have no faith Oracle can be > trusted as MySQL steward for the repos (Andrey did not have faith for quite > some time now and I lost it today). > > In order to respond to potential edge cases, we can also keep the option > that users who really want it, might be able to rebuild the image with > MySQL drivers - similarly as we give the users the option to use bullseye > in 2.8.0. And we can even run a CI build with it to see that it continues > to work. > > I believe this is the best option and we can get it implemented quickly > today and get RC4 of Airflow with mariadb clients. If we see a > support/consensus for the community we will **just** proceed with it and > start a formal LAZY_CONSENSUS thread. > > WDYT? > > J. >
