Hello everyone, (sorry for capital letters but we need to act quickly).
*TL;DR; Due to Oracle breaking MySQL repo signing, Together with Andrey we have a proposal to - immediately - switch to MariaDB clients in Airflow images for main and the upcoming Airflow 2.8.0 - giving the user an option to build their own custom images with MySQL clients if they need compatibility. * Last night it turned out that Oracle failed miserably with managing signing their repositories and their repositories are now badly signed with expired keys since yesterday. They acknowledged the issue and they apparently started to fix it but we have no idea how long it will take. The problem is that the policy for signing the MySQL repos Oracle is essentially flawed - the expiry date is very short - and we will have to re-release all (!) our historical images - with removed Mysql repository. Currently none of our users can extend our images without applying a workaround where they remove the mysql repository manually or allow unsecure repositories. *Context*: Just about we should have our Airflow 2.8.0 release, we have a high severity breaking change caused by Oracle failing miserably in resigning their MySQL repositories. The key they used to sign the repositories expired yesterday. They have a new key released in October - that has expiry date in 2025 - but they forgot to re-sign their repositories with it and currently it is not possible to securely install MySQL packages via their APT repositories at all. There is simply no way to do it securely (!). If they did it in October, we would have much more time to react and fix it, but they have not and now we are in a difficult situation (and it will likely delay the 2.8.0 release). We track the issue in https://github.com/apache/airflow/issues/36231 There are two severe issues in MySQL bug system (I commented on both of them): https://bugs.mysql.com/bug.php?id=113427 https://bugs.mysql.com/bug.php?id=113428 Basically anyone installing MySQL from the official APT repos has this problem now. *Bigger Problem:* The problem is that the way Oracle policy is set (even if they would not forget to resign the repos), is flawed. It means that all our images that we release have fixed expiry date (2023-12-14 as it turned out) and we will have to re-release the images again when they change the key and basically it requires manually applying the new key when they re-sign the repos anyway. The current workaround is to add this by to their Dockerfile when they are extending any of our past images: *RUN rm /etc/apt/sources.list.d/mysql.list* But this has side effects - for example if someone will run `apt upgrade` as part of their process, mysql clients will get installed from the "distro" packages - and they will be outdated. Another workaround is to add *RUN apt update --allow-insecure-repositories* But this has obvious issues with security and is not at all recommended. We will basically have to rebuild and re-release all our images in order to fix the problem - regardless of what Oracle does. This is something we discuss with Andrey how to address it in the best way. *Options we have for now:* We discussed it with Andrey and we are discussing how to prevent it happening in the future but we have immediate issue to address now - for the upcoming 2.8.0 We have the following options: 1) We wait until Oracle fixes it and signs the repos with new keys. The new keys are valid till 2025. However we can modify our dockerfile to remove the mysql repositories after installing mysql to avoid similar problems in the future. They responded that they are uploading new keys but we are not sure if we have 2) Thanks to what Andrey implemented a few months ago we have an easy way to switch to MariaDB clients. Those are binary compatible with MySQL and they are stable and production ready, and they pass all our tests in CI - https://github.com/apache/airflow/pull/36235 3) Drop MySQL support. Joking here of course - though I suggested it in the past a few times because of all the problems we have with MySQL and this one is by far the worst problem we and our users could have experienced). Option 1) is a bit "safer" in the "corporate" world - generally sticking to the "official" clients of "official" DB that we support is the default. Though my trust with Oracle as MySQL steward had been greatly diminished by that issue and the flawed approach for their policies. But we do not know how long it will take. Option 2) MariaDB is also very popular and solid and exhibits no such issues - their signing key is set to "no expiry" so if we switch to MariaDB, the problem will be gone. We run it for months for all our ARM images - and anyone using ARM Macbooks and running MySQL with Breeze would use those. *Recommendation:* Both myself and Andrey's recommendation is to go to MariaDB for main and 2.8.0. We are pretty fed-up with a number of past issues we had to fight with - and even if Oracle fixes it now, we have no faith Oracle can be trusted as MySQL steward for the repos (Andrey did not have faith for quite some time now and I lost it today). In order to respond to potential edge cases, we can also keep the option that users who really want it, might be able to rebuild the image with MySQL drivers - similarly as we give the users the option to use bullseye in 2.8.0. And we can even run a CI build with it to see that it continues to work. I believe this is the best option and we can get it implemented quickly today and get RC4 of Airflow with mariadb clients. If we see a support/consensus for the community we will **just** proceed with it and start a formal LAZY_CONSENSUS thread. WDYT? J.
