Yeah agreed, I don’t think it is worth keeping the support of MSSQL given the amount of usage vs the maintenance effort required.
On Tue, 24 Oct 2023 at 23:07, Jarek Potiuk <ja...@potiuk.com> wrote: > Yes. Agree with Andrey. I think our experience from the last few years was > "very" bad. The number of mssql users is very small. And the time that > maintainers and community members lose on various problems with it is huge. > Quite often every time we added a new feature requiring some new db > functionality, quite a lot of overhead was spent by the one adding new > features to solve the problems coming just from MSSQL support. It's not > "existing" issues - it's that it generally slows us down with making > changes > > I think there are two options for you when you. It's not "open issues", > it's the maintenance > > * switch to another backend (recommended). And it's not as complex as you > think. You can also use managed DB with all that is needed > (backup/maintenance), you do not have to manage it yourself. There are some > excellent postgres options available. > > * have your own fork airflow and keep the tests running and make your copy > works for MSSQL if you insist on keeping it. Since you already seem to be > ready to spend your engineering time on it, that seems doable. > > The second option I think might even be a business opportunity - for your > company or for anyone who would like to do it. Someone could even offer it > as a service or as a version to support it for others and make a small > business out of it if you are really so committed to it I guess, including > support for any mssql problems. > > That would actually be awesome if someone does it. > > J. > > > On Tue, Oct 24, 2023 at 11:44 PM Andrey Anshin <andrey.ans...@taragol.is> > wrote: > > > I don’t think there is any possibility left to keep MS SQL Server as DB > > backend for Airflow. > > > > I add Elad's message from the original discussion: > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > > Because it cleary describe what is happen with MS SQL as DB backend for > the > > last 1.5 years > > > > > During this time we hoped it would become stable and widely adopted. > > > To my taste MsSQL a backend has left a niche and is *not* worth the > > > maintenance > > of it in our CI. > > > > I also want to note the following points > > - MS SQL have unstable tests in Airflow CI, and some cases we even don't > > run them for the last couple months (or even longer) > > - In additional it taking 2x memory than any other backend > > - Lack of ARM support, this is also quite important because it prevent > > maintainers to check some sort of things in their M1/M2 laptops > > - Additional backend required extra effort for any contributors who want > to > > add new feature that touches DB > > > > > > This has always been an experimental feature which are described in > AIrflow > > Release Process: > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features > > , I would recommend your team focuses on Airflow on Postgres rather than > > hanging on to vague hope that MS SQL keeping in Airflow. > > > > Quite a few companies provide Managed Airflow, see: > > https://airflow.apache.org/ecosystem/#airflow-as-a-service (this is not > > complete list) and AFAIK none of them use any other backend rather than > > Postgres, maybe one exception with Google Composer v1 which seems use > > MySQL, even on Azure Data Factory Managed Airflow use Postgres as DB > > backend, see: > > > > > https://learn.microsoft.com/en-us/azure/data-factory/concept-managed-airflow#architecture > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Mon, 23 Oct 2023 at 23:01, agateaaa <agate...@gmail.com> wrote: > > > > > Hi All: > > > > > > Mssql support was voted to be dropped. > > > https://lists.apache.org/thread/r06j306hldg03g2my1pd4nyjxg78b3h4 > > > > > > One of our product requirements is that we can only use the Mssql > > database. > > > The product that uses airflow is installed with a suite of 8-10 other > > > products that all use Mssql database as their database. Preferably we > do > > > not want our customers to install another database like postgres or > MySQL > > > since it would involve extra overhead on their DBA team to maintain > (with > > > backup/restore functions) yet another database > > > > > > This has been already discussed and voted on but is there any way we > can > > > keep experimental support if we pitch in to fix any mssql related > issues? > > > > > > List of current mssql issues are here > > > * > > > > > > > > > https://github.com/apache/airflow/issues?q=is%3Aissue+label%3Abackend-mssql-experimintal+is%3Aopen > > > > > > Are there any other outstanding issues or can you please let us know a > > way > > > to identify mssql related bugs/problems that need to be addressed? > > > > > > e.g. > > > * https://github.com/apache/airflow/discussions/35114 > > > > > > > > > We are just trying to understand how much effort will be required to > keep > > > supporting Mssql and if we can help with that in any way. > > > > > > Thanks > > > Ashwin > > > > > >
