OK. Seeing that - I think I will do the next step - I pointed this discussion to at the discord of Dask and see if there is a volunteer there who would like to take on fixing the "Quarantined" issue https://github.com/apache/airflow/issues/32778 -> we have the flaky tests currently marked as "Quarantined - so it is not really running in regular tests anyway. If there is no-one stepping in and wanting to commit to take over and manage fixing of similar issues, I will start a voting thread for removal,
For me that is an ultimate (and very concrete) "checkpoint" if the Dask community is up to the task in our repo. And if they are not - they are always free to fork it and maintain their own provider. I think it's a very fair approach and quite reasonable expectation - taking into account that we have so little usage and so many problems. I announced our intentions in this thread: https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328 I explained the option Dask Community has and gave them a week for reaction. If no-one steps up and starts seriously diagnosing and fixing the flaky issue - I will start the vote on removal in about a week. J. On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <[email protected]> wrote: > +1 for removal if there is no active maintainer on this one > > Best, > Wei > > > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <[email protected]> > wrote: > > > > +1 for suspend and after a while remove > > > > There is a small chance that things would improve after we suspended this > > provider. > > > > However we do not have a lot of statistics, AFAIK (correct me if I'm > wrong) > > it is only two cases: > > * 1 provider suspended in the past and restored (Yandex), there is no > > process of removal at that moment. > > * 1 provider suspended and removed (Qubole) > > > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <[email protected]> > wrote: > > > >> Theres very little incentive in maintaining this if theres no one > actively > >> maintaining it. > >> > >> I am totally for the removal +1 > >> > >> > >> > >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty > >> <[email protected]> > >> wrote: > >> > >>> +1 for removal > >>> > >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <[email protected]> > wrote: > >>> > >>>>> we would do it branching off at the TAG when the last release > >> happened > >>>> and develop/release a fix from there. > >>>> > >>>> Since it would be possible to release security patches, +1 to remove > >> it. > >>>> > >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko > >> <[email protected] > >>>> > >>>> wrote: > >>>> > >>>>> If no one comes forward willing to support the executor long term I'm > >>> +1 > >>>>> for removal. > >>>>> > >>>>> ________________________________ > >>>>> From: Vincent Beck <[email protected]> > >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM > >>>>> To: [email protected] > >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend > >> (Remove?) > >>>>> Daskexecutor provider > >>>>> > >>>>> CAUTION: This email originated from outside of the organization. Do > >> not > >>>>> click links or open attachments unless you can confirm the sender and > >>>> know > >>>>> the content is safe. > >>>>> > >>>>> > >>>>> > >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >>> externe. > >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > >>>> pouvez > >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > >> certain > >>>> que > >>>>> le contenu ne présente aucun risque. > >>>>> > >>>>> > >>>>> > >>>>> +1 for removal > >>>>> > >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote: > >>>>>> More detailed comparison: > >>>>>> > >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day > >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day > >>>>>> > >>>>>> This means that *apache-airflow-providers-daskexecutor * is > >>> downloaded > >>>> in > >>>>>> less of *0.3 %* cases, comparing to *apache-airflow* > >>>>>> > >>>>>> I'd say it's negligible usage. > >>>>>> > >>>>>> My personal vote would go for immediate removal. > >>>>>> > >>>>>> WDYT? > >>>>>> > >>>>>> J. > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <[email protected]> > >>>> wrote: > >>>>>> > >>>>>>> > >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <[email protected]> > >>>> wrote: > >>>>>>> > >>>>>>>> Now that the code is in its own provider we can check the > >> download > >>>>> stats > >>>>>>>> of > >>>>>>>> the library via Pypi stats. > >>>>>>>> > >>>>>>>> > >>>>>>> Good point but this is only an indication. > >>>>>>> > >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit difficult > >>> to > >>>>>>> compare those. The "fixed" numbers are misleading (we still do > >> not > >>>> know > >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day > >> came > >>>> from > >>>>>>> mid-October. > >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) > >> -> > >>>>> ~0.1 > >>>>>>> %. But those are only downloads - and not whether it's used - > >>> likely > >>>>> there > >>>>>>> are a number of cases where CI pulls "all" extras. > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>> > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: [email protected] > >>>>> For additional commands, e-mail: [email protected] > >>>>> > >>>>> > >>>> > >>> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
