I was asked to open the issue in GitHub to get more visibility by Dask developers so here it is https://github.com/dask/community/issues/355
On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiuk <[email protected]> wrote: > OK. Seeing that - I think I will do the next step - I pointed this > discussion to at the discord of Dask and see if there is a volunteer there > who would like to take on fixing the "Quarantined" issue > https://github.com/apache/airflow/issues/32778 -> we have the flaky tests > currently marked as "Quarantined - so it is not really running in regular > tests anyway. If there is no-one stepping in and wanting to commit to take > over and manage fixing of similar issues, I will start a voting thread for > removal, > > For me that is an ultimate (and very concrete) "checkpoint" if the Dask > community is up to the task in our repo. And if they are not - they are > always free to fork it and maintain their own provider. I think it's a very > fair approach and quite reasonable expectation - taking into account that > we have so little usage and so many problems. > > I announced our intentions in this thread: > https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328 > > I explained the option Dask Community has and gave them a week for > reaction. If no-one steps up and starts seriously diagnosing and fixing the > flaky issue - I will start the vote on removal in about a week. > > J. > > > > On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <[email protected]> wrote: > >> +1 for removal if there is no active maintainer on this one >> >> Best, >> Wei >> >> > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <[email protected]> >> wrote: >> > >> > +1 for suspend and after a while remove >> > >> > There is a small chance that things would improve after we suspended >> this >> > provider. >> > >> > However we do not have a lot of statistics, AFAIK (correct me if I'm >> wrong) >> > it is only two cases: >> > * 1 provider suspended in the past and restored (Yandex), there is no >> > process of removal at that moment. >> > * 1 provider suspended and removed (Qubole) >> > >> > >> > ---- >> > Best Wishes >> > *Andrey Anshin* >> > >> > >> > >> > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <[email protected]> >> wrote: >> > >> >> Theres very little incentive in maintaining this if theres no one >> actively >> >> maintaining it. >> >> >> >> I am totally for the removal +1 >> >> >> >> >> >> >> >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty >> >> <[email protected]> >> >> wrote: >> >> >> >>> +1 for removal >> >>> >> >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <[email protected]> >> wrote: >> >>> >> >>>>> we would do it branching off at the TAG when the last release >> >> happened >> >>>> and develop/release a fix from there. >> >>>> >> >>>> Since it would be possible to release security patches, +1 to remove >> >> it. >> >>>> >> >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko >> >> <[email protected] >> >>>> >> >>>> wrote: >> >>>> >> >>>>> If no one comes forward willing to support the executor long term >> I'm >> >>> +1 >> >>>>> for removal. >> >>>>> >> >>>>> ________________________________ >> >>>>> From: Vincent Beck <[email protected]> >> >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM >> >>>>> To: [email protected] >> >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend >> >> (Remove?) >> >>>>> Daskexecutor provider >> >>>>> >> >>>>> CAUTION: This email originated from outside of the organization. Do >> >> not >> >>>>> click links or open attachments unless you can confirm the sender >> and >> >>>> know >> >>>>> the content is safe. >> >>>>> >> >>>>> >> >>>>> >> >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> >>> externe. >> >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne >> >>>> pouvez >> >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >> >> certain >> >>>> que >> >>>>> le contenu ne présente aucun risque. >> >>>>> >> >>>>> >> >>>>> >> >>>>> +1 for removal >> >>>>> >> >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote: >> >>>>>> More detailed comparison: >> >>>>>> >> >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day >> >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day >> >>>>>> >> >>>>>> This means that *apache-airflow-providers-daskexecutor * is >> >>> downloaded >> >>>> in >> >>>>>> less of *0.3 %* cases, comparing to *apache-airflow* >> >>>>>> >> >>>>>> I'd say it's negligible usage. >> >>>>>> >> >>>>>> My personal vote would go for immediate removal. >> >>>>>> >> >>>>>> WDYT? >> >>>>>> >> >>>>>> J. >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <[email protected]> >> >>>> wrote: >> >>>>>> >> >>>>>>> >> >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <[email protected]> >> >>>> wrote: >> >>>>>>> >> >>>>>>>> Now that the code is in its own provider we can check the >> >> download >> >>>>> stats >> >>>>>>>> of >> >>>>>>>> the library via Pypi stats. >> >>>>>>>> >> >>>>>>>> >> >>>>>>> Good point but this is only an indication. >> >>>>>>> >> >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit difficult >> >>> to >> >>>>>>> compare those. The "fixed" numbers are misleading (we still do >> >> not >> >>>> know >> >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day >> >> came >> >>>> from >> >>>>>>> mid-October. >> >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) >> >> -> >> >>>>> ~0.1 >> >>>>>>> %. But those are only downloads - and not whether it's used - >> >>> likely >> >>>>> there >> >>>>>>> are a number of cases where CI pulls "all" extras. >> >>>>>>> >> >>>>>>> J. >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: [email protected] >> >>>>> For additional commands, e-mail: [email protected] >> >>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
