Maintaining security fixes is independent from suspension/removal. We are obliged to release security fixes regardless if it's something in our code. For example if we have a report for Qubole and we assess it is valid and it will need change in our code, we should still do it for quite some time (we do not have specific expectations for that time - but in the future CRA regulation there is an expectation to set " expected time of life" for anything released and this would be as long as we set it then).
In both cases if we need to do it - we would do it branching off at the TAG when the last release happened and develop/release a fix from there. J. On Wed, Nov 15, 2023 at 9:30 PM Hussein Awala <huss...@awala.fr> wrote: > Since this executor was part of the Airflow core a few months ago, should > we maintain security patches for a reasonable period after officially > announcing its suspension? > > On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Hello Everyone, >> >> I would like to start discussion about potential suspension and eventually >> removal of Daskexecutor provider. >> >> After some recent changes and moving DaskExecutor from core to provider we >> have now an open option to suspend and eventually remove Dask Executor >> from >> our codebase. >> >> After we have the process tested and tried with Qubole, for me this is the >> next candidate to remove and get rid of some burden it involves. >> >> I believe this executor is hardly used by anyone - IMHO it does not really >> add a value to our Executor's offering and it causes a number of CI/ >> maintenance issues for us, while we have little to no experience and >> incentive and knowledge to manage it. It slows us down, makes our CI >> flaky, >> sometimes delays our release process etc etc. >> >> Some of the past discussions about it (this is the third time I am (re) >> starting this discussion - the first time it was in 2020 when we discussed >> removing it for Airflow 2.0. >> >> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m >> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh >> >> Some of the recent problems with Dask executor impacting stability of our >> CI / tests/ impacting release process are manifested in those issues and >> PRs over the last few years (and this is just a subset of issues and PRs >> dealing with the stability of it. >> >> * https://github.com/apache/airflow/pull/35659 >> * https://github.com/apache/airflow/issues/32778 >> * https://github.com/apache/airflow/pull/35473 >> * https://github.com/apache/airflow/pull/32991 >> * https://github.com/apache/airflow/pull/30258 >> * https://github.com/apache/airflow/pull/22046 >> * https://github.com/apache/airflow/pull/22017 >> >> >> Why now? What changed? >> >> One of the fantastic properties of the "providers" approach and moving the >> executor from core to provider - we can now ACTUALLY remove DaskExecutor >> without breaking any of our SemVer promises. This is something I was >> working on for quite a while and with some of the recent changes - making >> our Executor API stable and part of our public interface, introducing full >> life-cycle process for providers this is now entirely possible to remove >> all the maintenance burden while keeping our SemVer promises - so that we >> can do it without bumping Airflow to version 3. >> >> How does it work ? Simply - provider will remain released, and it will >> remain working in the version it has been released for the last time with >> all the future versions of Airflow, But at the same time we can remove the >> code from main of Airflow, stop running tests, stop caring about the >> provider impacting the constraints and generally speaking. >> >> If we decide to do somethin about, we have three options: >> >> 1) Suspend the daskexecutor provider >> 2) Remove it >> 3) We can also follow Suspend -> wait few months -> remove) pattern >> >> All of them are easy, all of them bring us much welcome relief in our >> release and CI and maintenance burden. All of them are well documented and >> tried and all that we need is just to agree which scenario we want to >> follow. >> >> In case 1) we will stop releasing new versions, but in case any new >> "development" is done on the dask side that will break currently released >> providers, it will be on someone who will take on the task to bring it >> back >> from suspension - fix all dependency issues, make sure tests are green and >> then it's just a matter of PR to bring it back. This is really a way to >> say >> "Hey Dask Executor community, if you want to keep any future development, >> it's on you to bring it back from suspension and take the task to solve >> all >> the difficulties we currently have to leave with in Airflow community". >> >> In case 2) we deliberately decide we do not want to maintain it any more >> and if someone would like to bring it back, they will have to convince the >> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it >> could be even different provider package, not compatible etc.). >> >> WDYT? Are there any DaskExecutor users around or someone you know of who >> would rely on it heavily? Are there any reasons why we should keep it as >> is >> ? Which of the above scenarios would you like to see? >> >> I would love to hear your opinions on that. If we see that we would like >> to >> follow 1) or 2) I am happy to pass the message to the Dask community and >> let them know that they will need to be involved in case they will want to >> bring back Dask from suspension or removal - I did that before for some of >> the past issues and I am happy to be the messenger (hopefull no-one shoots >> me). >> >> J. >> >
