Actually it is not that strightforward, but possibly we can make it works much more easily
In order to make Celery Executor works you need to do a bit more (but should be easy to add as an option to Breeze): * you need to start rabitmq or redis as integration (`--integration rabbitq --integration redis') * you need to start worker(s) (`airflow worker` in the background) * you might want to start flower optionally (the celery monitoring tool) So maybe we could add extra switch to start-airflow command (--use-celery-executor) that could set those integrations and start worker/flower additionally to running webserver/scheduler now in tmux ? WDYT? Maybe Ryan you can check if my recipe works ? I could add it then as an option. BTW. We already have a number of CeleryExecutor tests that use the integrations, so Breeze has all what's needed: https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109 J. On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <[email protected]> wrote: > Hmm, maybe I was just getting twisted around with docker then. I’ll have a > look at what you shared. > > Thanks Bin :) > > On Mar 22, 2021, at 01:52, Xinbin Huang <[email protected]> wrote: > > > Hi Ryan, > > I believe breeze already provides tools for you to do that: > > 1.Would it make sense to allow developers to choose the executor for their > Breeze environment? > > You can set up environment variables and any other custom setup you want > in the file: */files/airflow-breeze-config/variables.env. *To set up > CeleryExecutor, you just need to put `export > AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file. > > 2. Developing using Docker > <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> > would > require an update to the image each time you want to make a change to the > codebase > > Breeze automatically mounts the local source to the container unless you > explicitly skip it with the flag *--skip-mounting-local-sources. *You can > find more details here > https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze > > Best > Bin > > > On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <[email protected]> wrote: > >> I recently had some trouble trying to fix a bug in the CeleryExecutor >> <https://github.com/apache/airflow/pull/14883>. The code change was >> small, but it was really difficult to set up a development environment >> using the CeleryExecutor. I ultimately had to muck around with the test >> case that covers this situation, default_airflow.cfg, and >> default_celery.py. Developing using Docker >> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >> would require an update to the image each time you want to make a change to >> the codebase (or maybe `exec`ing into the relevant container?) which is a >> pain. >> >> This led me to two questions: >> >> 1. Would it make sense to allow developers to choose the executor for >> their Breeze environment? >> 2. If not, how do folks test out changes they make to the >> CeleryExecutor or KubernetesExecutor? >> >> Thanks! >> Ryan >> >> >> -- +48 660 796 129
