I think we need to ask “who is going to champion this executor.” I see that it is being used (a bit), but am concerned if no one with knowledge of this executor is willing to maintain it.
I’ve personally never used Dask and the DaskExecutor isn’t super high on my priority list compared to things like autoscaling, DAG serialization, etc. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Jan 17, 2020 at 6:07 AM, Jarek Potiuk <jarek.pot...@polidea.com> wrote: Do we have anyone here who uses Dask Executor and would like to test it/fix the tests. They are marked now as xfailed (expected to fail) and it would be great to fix them. J. On Tue, Jan 14, 2020 at 12:18 AM Darren Weber <dweber.consult...@gmail.com> wrote: > +1 for keeping it and fixing tests > > PS, I also noticed the skipped tests while looking at an option to use the > async client feature; if/when I get time to get back on that and figure out > how the test setup needs to work, I might also discover how to enable tests > for the non-async executor. No promises, just noting that I'm aware of it > too. > > On Mon, Jan 13, 2020 at 8:06 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > For now I marked the skipped tests we had (including Dask) as > > pytest.mark.xfail (means - expected to fail). They will be executed and > > summarized as XFail tests and we will have to deal with them at some > point. > > > > I think we will have to decide if we want to keep it or not, and either > > remove both tests and executor or fix the tests. > > > > J. > > > > On Mon, Jan 13, 2020 at 4:40 PM Shaw, Damian P. < > > damian.sha...@credit-suisse.com> wrote: > > > > > FYI I used Dash instead of Local Executor when first starting Airflow, > it > > > was a great way to make sure the Executor and Scheduler weren’t tied to > > > each other with no difficulty in set-up. But once I actually started > > > deploying to multiple boxes I needed queue names pretty quickly. So not > > > going to say it's needed but for me it was a helpful stepping stone. > > > > > > > > > -----Original Message----- > > > From: Ash Berlin-Taylor <a...@apache.org> > > > Sent: Sunday, January 12, 2020 17:38 > > > To: dev@airflow.apache.org > > > Cc: dev@airflow.apache.org > > > Subject: Re: Remove Dask Executor in Airflow 2.0 ? > > > > > > It hasn't been discussed before, but unlike the Mesos one this one was > > > seen a (tiny) bit of activity in 1.10 so at least one person is using > it > > > https://github.com/apache/airflow/pull/5273 > > > > > > On Jan 12 2020, at 9:05 pm, Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > I am finishing the PR on separating integrations and improving our CI > > > > footprint (https://github.com/apache/airflow/pull/7091) but during > > > > this change I have found that we have - apparently - dysfunctional > > > > DaskExecutor in Airflow 2.0. > > > > > > > > There is a "test_dask_executor.py" for which all tests are skipped. > > > > And they fail when I try to run the tests. I tried to look for any > > > > reference in devlist archives but I couldn't find anything about it. > > > > > > > > Can someone shed some light on this? Should we remove Dask executor > > > > completely from Airflow 2.0 ? Or should we fix the tests/executor ? > > > > Has it been discussed ? > > > > > > > > J. > > > > > > > > -- > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > =============================================================================== > > > > > > Please access the attached hyperlink for an important electronic > > > communications disclaimer: > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > > > > > =============================================================================== > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > -- > Darren L. Weber, Ph.D. > http://psdlw.users.sourceforge.net/ > http://psdlw.users.sourceforge.net/wordpress/ > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
