Yeah I'll give that a shot. Do you think something with KubernetesExecutor with something like MiniKube would also make sense?
On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <[email protected]> wrote: > Actually it is not that strightforward, but possibly we can make it works > much more easily > > In order to make Celery Executor works you need to do a bit more (but > should be easy to add as an option to Breeze): > > * you need to start rabitmq or redis as integration (`--integration > rabbitq --integration redis') > * you need to start worker(s) (`airflow worker` in the background) > * you might want to start flower optionally (the celery monitoring tool) > > So maybe we could add extra switch to start-airflow command > (--use-celery-executor) that could set those integrations and start > worker/flower additionally to running webserver/scheduler now in tmux ? > > WDYT? Maybe Ryan you can check if my recipe works ? I could add it then as > an option. > > BTW. We already have a number of CeleryExecutor tests that use the > integrations, so Breeze has all what's needed: > > > https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109 > > J. > > On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <[email protected]> wrote: > >> Hmm, maybe I was just getting twisted around with docker then. I’ll have >> a look at what you shared. >> >> Thanks Bin :) >> >> On Mar 22, 2021, at 01:52, Xinbin Huang <[email protected]> wrote: >> >> >> Hi Ryan, >> >> I believe breeze already provides tools for you to do that: >> >> 1.Would it make sense to allow developers to choose the executor for >> their Breeze environment? >> >> You can set up environment variables and any other custom setup you want >> in the file: */files/airflow-breeze-config/variables.env. *To set up >> CeleryExecutor, you just need to put `export >> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file. >> >> 2. Developing using Docker >> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >> would >> require an update to the image each time you want to make a change to the >> codebase >> >> Breeze automatically mounts the local source to the container unless you >> explicitly skip it with the flag *--skip-mounting-local-sources. *You >> can find more details here >> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze >> >> Best >> Bin >> >> >> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <[email protected]> >> wrote: >> >>> I recently had some trouble trying to fix a bug in the CeleryExecutor >>> <https://github.com/apache/airflow/pull/14883>. The code change was >>> small, but it was really difficult to set up a development environment >>> using the CeleryExecutor. I ultimately had to muck around with the test >>> case that covers this situation, default_airflow.cfg, and >>> default_celery.py. Developing using Docker >>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >>> would require an update to the image each time you want to make a change to >>> the codebase (or maybe `exec`ing into the relevant container?) which is a >>> pain. >>> >>> This led me to two questions: >>> >>> 1. Would it make sense to allow developers to choose the executor >>> for their Breeze environment? >>> 2. If not, how do folks test out changes they make to the >>> CeleryExecutor or KubernetesExecutor? >>> >>> Thanks! >>> Ryan >>> >>> >>> > > -- > +48 660 796 129 >
