Continuing what I wrote in GitHub issue, I vote "-1" to officially support MariaDB since the topic has again diverged to MariaDB instead of MySQL.
We have Postgres, MySQL and now experimental support for MSSQL too. We can't and should keep adding on the list. Between Postgres and MySQL itself therr is enough choices for the community. There will always be a ask for another DB from atleast one user because it suits their needs. But it isn't feasible to keep on adding to the suite of CI test Matrix. And once you "officially" support a DB, it is not as easy as dropping the DB support like you can already see for MySQL users. Hence my apprehension for adding yet another DB. Again I am not opposed to adding PR fixed to make it compatible to an unsupported DB (if it does not break existing DB support) but I am opposed to adding yet another DB. It is a lot of maintenance overhead and needs ton of testing to make sure it is reliable and performant. Regards, Kaxil On Sun, Nov 7, 2021, 15:15 Jarek Potiuk <[email protected]> wrote: > Thanks Alexadre - just some context here. > > Yeah - I personally sympathise with the community-friendliness of > MariaDB for sure. But also reality is such that MySQL is more likely > to be chosen by "big commercial users" I am afraid. ASF is generally a > commercially-friendly organisation - i.e. we provide everything for > free, but we realise that our users make business with our software, > and we do not want to "limit" them in any way there. So the fact that > such db is "community" friendly is sadly (but justifiably), quite > secondary as long as it complies with our licencing policy and it > should not guide our decisions I think. > > We equally support Presto and Trino for example (speaking of similar > situation) and "it's easier to integrate our CI with Trino, because > they publish and maintain docker images" was really the only criteria > we had for switching Presto integration to Trino when they split > (without judging the community friendliness of either of those). > > In the case of MariaDB, only recently the SKIP LOCKED functionality > was added and we've had MySQL tested and used while Airflow 2 was > developed - that is why MariaDB was not even considered before and > MySQL has won the race there. > > J. > > On Sun, Nov 7, 2021 at 2:56 PM Alexandre Vermeerbergen > <[email protected]> wrote: > > > > To further clarify: we use MariaDB rather than MySQL because for us, > > MariaDB is clearly more community-friendly than MySQL. > > > > Remember that MySQL is provided by ORACLE : they have a long track of > > record for making communities' life hard, with closed source items > > (suck as Java TCKs), and bizarre redistribution restrictions aimed > > confusing users between what's allowed for free vs. paid licenses. > > They recent "Java 17 is free" move "but only for 2 years" (in small > > prints) is yet another example of it. > > > > On the other hand, MariaDB is endorsed by its foundation, with diverse > > sponsors including huge companies, which gives us a sense of > > durability and trustfulness. > > > > Disclaimer: all this is based on my impressions, which I share - but > > I'm sure others will disagree and I fully respect that. > > > > Thanks for this debate, in any cases, and whatever the outcome will be! > > > > Alexandre > > > > Le dim. 7 nov. 2021 à 14:41, Jarek Potiuk <[email protected]> a écrit : > > > > > > Yeah - looking at some big users who DO use MySQL, I agree that IF > > > (and this is a huuuge if) we ever decide on removing MySQL, the lead > > > time and preparation to that must be in years rather than months. So > > > It's quite unlikely to happen IMHO. > > > > > > However - maybe another question (since I have some people's attention > > > already :) ) > > > > > > How do people rate the need for MariaDB support (touched by Alexandre > > > briefly) ?? > > > > > > For now our approach is (that's my interpretation of it at least - but > > > maybe others have different views/expectations): > > > > > > "It is very close to MySQL, maybe it works, maybe it does not, but we > > > do not support it nor spend any effort on it. Occasionally some users > > > contribute some fixes specifically for MariaDB, but it does not make > > > us "support it" in any way" > > > > > > In most circumstances where users raise some DB-related issues with > > > MariaDB, rhis (as I see it at least and this is my approach) leads us > > > to: "MariaDB is not supported, please switch to MySQL" (unless of > > > course the problem is trivial). > > > > > > I have a number of "sub-questions" here: > > > > > > * Do other committers understand it the same way? > > > * Do we state it clearly enough? > > > * Is this an acceptable approach from the user's point of view > > > (knowing that it saves a lot of effort for the community as a whole)? > > > * Do the users who have MariaDB understand it the same way? > > > > > > I wonder how many users of MariaDB we have out there and what do they > > > feel about this "statement/level of support"? > > > > > > Or maybe indeed there is a need for more "full support" (with tests), > > > or maybe the idea in the discussion about us stating that MariaDB for > > > us is "best effort" (though we seem to not spend virtually any effort > > > on it). > > > > > > I think we have a range of options here - they are no 0/1 choice and I > > > just wonder if the current "approach" is good and can be improved > > > somehow (and whether it needs to be improved at all)? > > > I also think it would be great to make sure we align expectations of > > > different community members here (in case they are not aligned of > > > course) and be very precise about our statement/level of support for > > > MariaDB > > > > > > WDYT? > > > > > > J. > > > > > > On Sat, Nov 6, 2021 at 11:14 PM Xinbin Huang <[email protected]> > wrote: > > > > > > > > Agreed with Elad and XD. > > > > > > > > Personally, I +100 to get the elephant out of the room, but it'll > also post challenges to some companies. > > > > Here at Stripe. Postgres is not an option provided by the DB team > and switching from MySQL means that the team needs to either push the DB > team to support Postgres or manage Postgres themself. It may not be trivial > to get an approval on either of these. > > > > > > > > If we decide to do this, we'll need to spread the news way ahead of > time. > > > > > > > > Regards > > > > Bin > > > > > > > > On Sat, Nov 6, 2021 at 4:36 AM Elad Kalif <[email protected]> > wrote: > > > >> > > > >> I can tell that for us at Wix - migrating to Postgres is not going > to be simple nor fast. > > > >> We get alot of services from our DBA team because we are using > MySQL. Think of it like a partially/semi managed Airflow. > > > >> Removing support for a database feels to me like something we > should give a heads up. This is not just a feature we are removing. > > > >> Companies may have work to do in "preparing the ground" before they > can actually migrate so if we are notifying about dropping MySQL support > only when we release Airflow 3 this may be a problem. > > > >> > > > >> > > > >> On Sat, Nov 6, 2021 at 12:41 PM Jarek Potiuk <[email protected]> > wrote: > > > >>> > > > >>> Just to be clear. IF we get rid of MySQL this means no support for > > > >>> MariaDB either. > > > >>> > > > >>> On Sat, Nov 6, 2021 at 11:30 AM Alexandre Vermeerbergen > > > >>> <[email protected]> wrote: > > > >>> > > > > >>> > To be more precise: > > > >>> > +1 to get MySQL out > > > >>> > +1 to get MariaDB official support - we're running Airflow with > > > >>> > MariaDB without troubles and we're reluctant to move to > Postgres, as > > > >>> > we have no admin skills on that later and lots on MariaDB > > > >>> > > > > >>> > Le sam. 6 nov. 2021 à 11:24, Alexandre Vermeerbergen > > > >>> > <[email protected]> a écrit : > > > >>> > > > > > >>> > > +1 let the Elephant get out from the room - a jungle in a > better place > > > >>> > > for Elephants that rooms :) > > > >>> > > > > > >>> > > Le sam. 6 nov. 2021 à 11:18, Jarek Potiuk <[email protected]> > a écrit : > > > >>> > > > > > > >>> > > > Hey everyone, > > > >>> > > > > > > >>> > > > Some of us had a discussion about MariaDB support here > > > >>> > > > https://github.com/apache/airflow/pull/18506 and as a > result I think > > > >>> > > > this might be a good time to talk about the Elephant in the > room we > > > >>> > > > have. > > > >>> > > > > > > >>> > > > I would like to know what others think about the potential > of REMOVING > > > >>> > > > MySQL support in future Airflow versions ? > > > >>> > > > > > > >>> > > > I believe for quite some time MySQL is the "Elephant in the > room" for > > > >>> > > > us, and it's one of the things that already slows us down > when we add > > > >>> > > > new features and when at some point we start thinking about > Airflow 3, > > > >>> > > > maybe, just maybe we could think about removing support for > it. > > > >>> > > > > > > >>> > > > Why thinking about removing MySQL? > > > >>> > > > > > > >>> > > > Quoting the quote of Kaxil from our discussion " "Do less > but do them > > > >>> > > > well". We are relying more and more on more sophisticated > features and > > > >>> > > > queries of the underlying DB and this has already hit - > especially the > > > >>> > > > people who developed new features but also those who helped > others > > > >>> > > > with issues. > > > >>> > > > > > > >>> > > > There are multiple problems with MySQL: deadlocks, encoding > problems, > > > >>> > > > support for different query constructs we have and they keep > on > > > >>> > > > reappearing. I personally developed quite negative feelings > for MySQL > > > >>> > > > while working on Airflow. > > > >>> > > > > > > >>> > > > Some more context: > > > >>> > > > > > > >>> > > > * All the Airflow-As-A-Service providers are using Postgres > now as of Airflow 2. > > > >>> > > > * It seems from some discussions with people - that > migration from > > > >>> > > > MySQL to Postgres is possible and we could even develop a > tool for > > > >>> > > > that for users who would like to migrate in Airflow 3. > > > >>> > > > * We also have MsSQL - which is fresh but I think there > might be > > > >>> > > > stronger reasons for people to use it - especially if they > are in > > > >>> > > > Azure/MS "world" (but we could also consider dropping it as > well) > > > >>> > > > * I do not think there are "super-strong" reasons why > people would > > > >>> > > > like to stick to MySQL. Yes, there are people who prefer it > - but in > > > >>> > > > our case the DB is really an "internal" piece of Airflow. I > can > > > >>> > > > imagine people use Postgres only for Airflow even if for the > majority > > > >>> > > > of other things they use MySQL. > > > >>> > > > * MySQL was 25% last time we checked: > > > >>> > > > https://airflow.apache.org/blog/airflow-survey-2020/ but I > bet a lot > > > >>> > > > of that was Composer 1.* (Which with Airflow 2 is gone). > > > >>> > > > > > > >>> > > > I wonder what others think? > > > >>> > > > > > > >>> > > > J. >
