Since we moved the executors to the providers packages and made the executor interface pluggable and extensible, we should move the docs to their corresponding providers. However, we need to keep a doc in Airflow core that explains how to use/configure a provider executor (as we have for the secret managers and the task log handlers) and maybe how to create a new custom one.
On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <elad...@apache.org> wrote: > Hello everyone, > > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes > Executor docs to providers <https://github.com/apache/airflow/issues/33916 > > > > *Background:* > We had a discussion about extracting Celery, Kubernetes, Dask executors > from core to providers (discussion thread > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote > thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8>) > > One of the things we voted on was: > > Also, resulting from the discussion we will keep documentation for > > available executors in Airflow (so they will still be considered as THE > > executors available and will be discoverable in the same way as today). > > > *The problem:* > Airflow Core and Providers do not share the same release cycle nor cadance. > This means that if we add new capabilities to executors or fix an issue > which requires both code and doc update - the code will be delivered > before/ahead of documentation. Both cases are not good. > > > *My proposal:*Now that Celery, Kubernetes, Dask executors are in providers. > The section of core-concept > < > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html > > > should > contain only HIGH level information about the executors. It should not > contain information about executors internals or how to address common > problems. This information should be in the provider docs. The high level > info should be short and on a level that is relevant for all executors > which means it's not likely to have many changes over time. The > core-concept should have links/refers for deep dive read to the provider > docs. This is very similar to what we do with Notifiers > < > https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html > > > core > contains high level information and a list of notifers that are linked to > provider docs. > > WDYT? > > Also, resulting from the discussion we will keep documentation for > available executors in Airflow (so they will still be considered as THE > executors available and will be discoverable in the same way as today).Moe > K8S and Celery Executors (and related) to respective providers? >
