Hey Jarek, Thank you for opening this discussion. I have been working since this morning on a solution to this problem in our Airflow Prod image, which is based on the Airflow docker file retrieved from tag 2.6.3, without duplicating the docker file in our repository.
I agree that option 1) is more secure, but IMHO, if we chose it, we need to provide a way to update the keys in the dockerfile; otherwise, users will need to duplicate it or update to the most recent version of this file, which is not always compatible with their Airflow version. Option 2) seems good to me, and I prefer it if it's fully compatible, but we have to keep the possibility of building the image with the MySql client and testing it in the CI. For now, we may need to update the key (once Oracle fixes the issue) in the stable branch for all minor versions released in the last two years (2.2 to 2.7). On Fri, Dec 15, 2023 at 11:55 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Hello everyone, > > (sorry for capital letters but we need to act quickly). > > *TL;DR; Due to Oracle breaking MySQL repo signing, Together with Andrey we > have a proposal to - immediately - switch to MariaDB clients in Airflow > images for main and the upcoming Airflow 2.8.0 - giving the user an option > to build their own custom images with MySQL clients if they need > compatibility. * > > Last night it turned out that Oracle failed miserably with managing signing > their repositories and their repositories are now badly signed with expired > keys since yesterday. They acknowledged the issue and they apparently > started to fix it but we have no idea how long it will take. > > The problem is that the policy for signing the MySQL repos Oracle is > essentially flawed - the expiry date is very short - and we will have to > re-release all (!) our historical images - with removed Mysql repository. > > Currently none of our users can extend our images without applying a > workaround where they remove the mysql repository manually or allow > unsecure repositories. > > *Context*: > > Just about we should have our Airflow 2.8.0 release, we have a high > severity breaking change caused by Oracle failing miserably in resigning > their MySQL repositories. The key they used to sign the repositories > expired yesterday. > > They have a new key released in October - that has expiry date in 2025 - > but they forgot to re-sign their repositories with it and currently it is > not possible to securely install MySQL packages via their APT repositories > at all. There is simply no way to do it securely (!). > > If they did it in October, we would have much more time to react and fix > it, but they have not and now we are in a difficult situation (and it will > likely delay the 2.8.0 release). > > We track the issue in https://github.com/apache/airflow/issues/36231 > > There are two severe issues in MySQL bug system (I commented on both of > them): > > https://bugs.mysql.com/bug.php?id=113427 > https://bugs.mysql.com/bug.php?id=113428 > > Basically anyone installing MySQL from the official APT repos has this > problem now. > > *Bigger Problem:* > > The problem is that the way Oracle policy is set (even if they would not > forget to resign the repos), is flawed. It means that all our images that > we release have fixed expiry date (2023-12-14 as it turned out) and we will > have to re-release the images again when they change the key and basically > it requires manually applying the new key when they re-sign the repos > anyway. > > The current workaround is to add this by to their Dockerfile when they are > extending any of our past images: > > > *RUN rm /etc/apt/sources.list.d/mysql.list* > > But this has side effects - for example if someone will run `apt upgrade` > as part of their process, mysql clients will get installed from the > "distro" packages - and they will be outdated. Another workaround is to > add > > *RUN apt update --allow-insecure-repositories* > > But this has obvious issues with security and is not at all recommended. > > We will basically have to rebuild and re-release all our images in order to > fix the problem - regardless of what Oracle does. This is something we > discuss with Andrey how to address it in the best way. > > *Options we have for now:* > > We discussed it with Andrey and we are discussing how to prevent it > happening in the future but we have immediate issue to address now - for > the upcoming 2.8.0 > > We have the following options: > > 1) We wait until Oracle fixes it and signs the repos with new keys. The new > keys are valid till 2025. However we can modify our dockerfile to remove > the mysql repositories after installing mysql to avoid similar problems in > the future. They responded that they are uploading new keys but we are not > sure if we have > > 2) Thanks to what Andrey implemented a few months ago we have an easy way > to switch to MariaDB clients. Those are binary compatible with MySQL and > they are stable and production ready, and they pass all our tests in CI - > https://github.com/apache/airflow/pull/36235 > > 3) Drop MySQL support. Joking here of course - though I suggested it in the > past a few times because of all the problems we have with MySQL and this > one is by far the worst problem we and our users could have experienced). > > Option 1) is a bit "safer" in the "corporate" world - generally sticking to > the "official" clients of "official" DB that we support is the default. > Though my trust with Oracle as MySQL steward had been greatly diminished by > that issue and the flawed approach for their policies. But we do not know > how long it will take. > > Option 2) MariaDB is also very popular and solid and exhibits no such > issues - their signing key is set to "no expiry" so if we switch to > MariaDB, the problem will be gone. We run it for months for all our ARM > images - and anyone using ARM Macbooks and running MySQL with Breeze would > use those. > > *Recommendation:* > > Both myself and Andrey's recommendation is to go to MariaDB for main and > 2.8.0. We are pretty fed-up with a number of past issues we had to fight > with - and even if Oracle fixes it now, we have no faith Oracle can be > trusted as MySQL steward for the repos (Andrey did not have faith for quite > some time now and I lost it today). > > In order to respond to potential edge cases, we can also keep the option > that users who really want it, might be able to rebuild the image with > MySQL drivers - similarly as we give the users the option to use bullseye > in 2.8.0. And we can even run a CI build with it to see that it continues > to work. > > I believe this is the best option and we can get it implemented quickly > today and get RC4 of Airflow with mariadb clients. If we see a > support/consensus for the community we will **just** proceed with it and > start a formal LAZY_CONSENSUS thread. > > WDYT? > > J. >
