Since this executor was part of the Airflow core a few months ago, should we maintain security patches for a reasonable period after officially announcing its suspension?
On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Hello Everyone, > > I would like to start discussion about potential suspension and eventually > removal of Daskexecutor provider. > > After some recent changes and moving DaskExecutor from core to provider we > have now an open option to suspend and eventually remove Dask Executor from > our codebase. > > After we have the process tested and tried with Qubole, for me this is the > next candidate to remove and get rid of some burden it involves. > > I believe this executor is hardly used by anyone - IMHO it does not really > add a value to our Executor's offering and it causes a number of CI/ > maintenance issues for us, while we have little to no experience and > incentive and knowledge to manage it. It slows us down, makes our CI flaky, > sometimes delays our release process etc etc. > > Some of the past discussions about it (this is the third time I am (re) > starting this discussion - the first time it was in 2020 when we discussed > removing it for Airflow 2.0. > > * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m > * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh > > Some of the recent problems with Dask executor impacting stability of our > CI / tests/ impacting release process are manifested in those issues and > PRs over the last few years (and this is just a subset of issues and PRs > dealing with the stability of it. > > * https://github.com/apache/airflow/pull/35659 > * https://github.com/apache/airflow/issues/32778 > * https://github.com/apache/airflow/pull/35473 > * https://github.com/apache/airflow/pull/32991 > * https://github.com/apache/airflow/pull/30258 > * https://github.com/apache/airflow/pull/22046 > * https://github.com/apache/airflow/pull/22017 > > > Why now? What changed? > > One of the fantastic properties of the "providers" approach and moving the > executor from core to provider - we can now ACTUALLY remove DaskExecutor > without breaking any of our SemVer promises. This is something I was > working on for quite a while and with some of the recent changes - making > our Executor API stable and part of our public interface, introducing full > life-cycle process for providers this is now entirely possible to remove > all the maintenance burden while keeping our SemVer promises - so that we > can do it without bumping Airflow to version 3. > > How does it work ? Simply - provider will remain released, and it will > remain working in the version it has been released for the last time with > all the future versions of Airflow, But at the same time we can remove the > code from main of Airflow, stop running tests, stop caring about the > provider impacting the constraints and generally speaking. > > If we decide to do somethin about, we have three options: > > 1) Suspend the daskexecutor provider > 2) Remove it > 3) We can also follow Suspend -> wait few months -> remove) pattern > > All of them are easy, all of them bring us much welcome relief in our > release and CI and maintenance burden. All of them are well documented and > tried and all that we need is just to agree which scenario we want to > follow. > > In case 1) we will stop releasing new versions, but in case any new > "development" is done on the dask side that will break currently released > providers, it will be on someone who will take on the task to bring it back > from suspension - fix all dependency issues, make sure tests are green and > then it's just a matter of PR to bring it back. This is really a way to say > "Hey Dask Executor community, if you want to keep any future development, > it's on you to bring it back from suspension and take the task to solve all > the difficulties we currently have to leave with in Airflow community". > > In case 2) we deliberately decide we do not want to maintain it any more > and if someone would like to bring it back, they will have to convince the > community and get the VOTE or LAZY CONSENSUS to bring it back (and then it > could be even different provider package, not compatible etc.). > > WDYT? Are there any DaskExecutor users around or someone you know of who > would rely on it heavily? Are there any reasons why we should keep it as is > ? Which of the above scenarios would you like to see? > > I would love to hear your opinions on that. If we see that we would like to > follow 1) or 2) I am happy to pass the message to the Dask community and > let them know that they will need to be involved in case they will want to > bring back Dask from suspension or removal - I did that before for some of > the past issues and I am happy to be the messenger (hopefull no-one shoots > me). > > J. >
