potiuk commented on pull request #16829: URL: https://github.com/apache/airflow/pull/16829#issuecomment-903179502
Absolutely 1) 'default' should be treated as "special" in DaskExecutor. Just o explain a bit of context and set expectations: The whole DaskExecutor integration is (and should be) an afterthought for Airflow - to be perfectly honest. DaskExecutor is very rarely used and I think there are no committers who are in any capacity familiar with DaskExecutor because it is - to be perfectly blunt - a total niche for Airflow. Last time we reached out to Dask Community to "keep it" and the only reason Dask Execuor remained in Airflow 2 was because people from Dask community contributed some fixes to keep the tests from failing. And this is the only way to keep it "in" to be honest. We need people from the Dask user's community to make sure that the DaskExecutor is good to go. So DaskExecutor should definitely follow any of the "Airflow" conventions and adapt to Airflow, rather than Airflow adapting to it. Celery Executor is by far the most popular one in Airflow https://airflow.apache.org/blog/airflow-survey-2020/ and Dask Executor is not even mentioned there. So yeah. adapting `Dask Executor` to accept `default` as special value sounds about right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
