Still I think we change things we explicitly told our users they can use ... if someone asks, it should not be a single person decision, it should be something we as community agreed to - if only no-one opposed.
It's not "Ash said it's ok" or "Jarek said it's ok". As a community we agreed to it. This is what decision making in the ASF project looks like. Governance is important. J On Fri, Oct 24, 2025 at 10:35 PM Ash Berlin-Taylor <[email protected]> wrote: > I don’t think even a lazy consensus is needed. > > If someone wants to use the Mysql client library specifically, let them > build their own docker image from scratch. We don’t have to, nor can we, > cater for every possible whim in our docker images. > > The Astronomer Runtime images have been built using > https://packages.debian.org/trixie/default-libmysqlclient-dev which > gives us mariadb (and the same on bookworm etc) for years, and none of our > users have noticed (and yes, it _is_ being used, I checked that) > > Cheers, > -ash > > > On 24 Oct 2025, at 18:41, Jarek Potiuk <[email protected]> wrote: > > > > If there are no comments or objections - I will start a lazy consensus > > about it on Monday. MySQL client is already removed in main (to unblock > > CI) - so if lazy consensus passes, it will stay like that. > > > > On Thu, Oct 23, 2025 at 12:56 PM Jarek Potiuk <[email protected]> wrote: > > > >>> And if there are issues, would we be able to fix them in Airflow? > >> > >> No idea - we had no issues of that sort. Yet it's the third time we > >> have issues with Oracle drivers that we actually know we are not able to > >> fix without Oracle fixing their side. > >> > >> On Thu, Oct 23, 2025 at 12:52 PM Jarek Potiuk <[email protected]> wrote: > >> > >>> PR to remove mysql client is here - > >>> https://github.com/apache/airflow/pull/57146 . We should merge it now > to > >>> fix our canary CI builds. We can revert it later if we will find ways > to > >>> make MySQL work properly (though I fail to see how it can be achieved > while > >>> Oracle does what it does). > >>> > >>> On Thu, Oct 23, 2025 at 12:45 PM Jarek Potiuk <[email protected]> > wrote: > >>> > >>>>> What’s the current compatibility situation between MySQL and MariaDB? > >>>> Would the OSS client be able to connect to Oracle-provided MySQL > server > >>>> software? And if there are issues, would we be able to fix them in > Airflow? > >>>> > >>>> As explained above - for two years CI and PROD images of Airflow use > >>>> MariaDB client. There were zero issues reported about compatibility. > >>>> > >>>> On Thu, Oct 23, 2025 at 12:37 PM Tzu-ping Chung via dev < > >>>> [email protected]> wrote: > >>>> > >>>>> What’s the current compatibility situation between MySQL and MariaDB? > >>>>> Would the OSS client be able to connect to Oracle-provided MySQL > server > >>>>> software? And if there are issues, would we be able to fix them in > Airflow? > >>>>> > >>>>> > >>>>> > >>>>>> On 23 Oct 2025, at 18:25, Jarek Potiuk <[email protected]> wrote: > >>>>>> > >>>>>> Hello Everyone, > >>>>>> > >>>>>> *TL;DR; I would like to propose complete dropping of MySQL "Oracle > >>>>>> published" client libraries from our container images.* > >>>>>> > >>>>>> Two years ago [1] - we switched to MariaDB clients by default > because > >>>>> of a > >>>>>> very convoluted (and plain wrong) approach of Oracle for their Apt > >>>>>> repositories and we are back to the situation we faced 2 years ago. > >>>>>> > >>>>>> We protected (nicely) against total disaster (where I had to > manually > >>>>> build > >>>>>> and push 100 of our broken images during the weekend) and switched > to > >>>>>> MariaDB by default. We still left the option to build the image with > >>>>> MySQL > >>>>>> client and we still run it in our CI - this is how we found tonight > >>>>>> that the problem is back, Luckily all that is needed is we need to > >>>>> drop the > >>>>>> optional support we have for MySQL images described in [2]. It > >>>>> requires the > >>>>>> users who wish to use MySQL client to build the images using our > >>>>>> Dockerfiles with specific build arguments. We kept it for > >>>>> compatibility and > >>>>>> convenience of those who would have to use the clients. We never > >>>>> heard back > >>>>>> from anyone if they are using or not - it's very likely, it's used > >>>>>> extremely rarely (if at all). > >>>>>> > >>>>>> The problem is (and you can find many articles, stack overflow > >>>>> issues, blog > >>>>>> posts about it) that Oracle uses a very convoluted and wrong way of > >>>>> making > >>>>>> their apt packages available - they sign their packages and repos > with > >>>>>> expiring keys. No other company I know is using this, this is > against > >>>>>> debian recommendations and every two years it causes the same > problem > >>>>> - the > >>>>>> old packages are not installable, images released in the past that > >>>>> have > >>>>>> their repos added are blocked from installing **anything** (i.e. apt > >>>>>> install fails to install anything unless you remove oracle's repos > and > >>>>>> keys) > >>>>>> > >>>>>> Just to be clear - this is (so far) not a problem for the server > side > >>>>> of > >>>>>> MySQL. We are ok with our tests where MySQL is used as a server - > >>>>> because > >>>>>> we can use images they publish to run the servers) and MariaDB > >>>>> clients work > >>>>>> well with those. But for MySQL clients - every 2 years (it's already > >>>>> the > >>>>>> 3rd time it happened) it makes our images and dockerfiles broken - > >>>>> and our > >>>>>> users who want to use the clients - scrambling to install those. > >>>>>> > >>>>>> To add to that - when it happens, Oracle is surprised. Always. No > >>>>>> exception. It happened tonight and as of tonight, you have no way > >>>>>> installing the packages at all (even if you somehow get hold of the > >>>>> new > >>>>>> key) because their repos are signed with old, expired keys - 2 years > >>>>> ago it > >>>>>> took them almost a week to fix it and the bug created [3] was > created > >>>>> where > >>>>>> I - among others explained them the problem they had and what > >>>>> solution they > >>>>>> can apply. > >>>>>> > >>>>>> They ignored it. Today the story repeated itself. MySQL clients > >>>>> stopped > >>>>>> installing - because they are signed by expired keys and their repo > >>>>> is also > >>>>>> signed by the same expired key. They have learned nothing and did > not > >>>>> fix > >>>>>> the problem. They will have another sh**tstom coming and will > >>>>> scramble to > >>>>>> fix it again. > >>>>>> > >>>>>> But I do not wish our community (and me particularly) to be part of > >>>>> it any > >>>>>> more - my proposal is to simply drop that option and let the users > >>>>> be on > >>>>>> their own if they want to use MySQL client. > >>>>>> > >>>>>> I will proceed with removing it now in a PR (completely) so that we > >>>>> fix our > >>>>>> failing canary builds and If no-one objects, I will call for LAZY > >>>>> CONSENSUS > >>>>>> and will not revert the change. > >>>>>> > >>>>>> J. > >>>>>> > >>>>>> [1] Lazy consensus from 2023 > >>>>>> https://lists.apache.org/thread/rxbyxg11jg7y35k8om0f8wgb2l9h459l > >>>>>> [2] Optional support for MySQL clients > >>>>>> > >>>>> > https://airflow.apache.org/docs/docker-stack/build.html#building-images-with-mysql-client > >>>>>> [3] Bug from 2 years ago - https://bugs.mysql.com/bug.php?id=113432 > >>>>> > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: [email protected] > >>>>> For additional commands, e-mail: [email protected] > >>>>> > >>>>> > >
