Oh, I did not write about K8S Executor because we already have K8S executor support in Breeze. It does not use Breeze docker image - it uses KinD (Kubernetes-in-Docker) that has all the bells and whistles (and more).
Here is the part of Breeze video when I showcase it: https://github.com/apache/airflow/blob/master/BREEZE.rst#running-kubernetes-tests And here the whole chapter describing how to test Airflow with K8S: https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes What we have built-in with Breeze (automagically): * configures and start kind cluster (kind is a really nice development-focus installation of K8S that uses K8S-in-Docker approach). * deploys airflow with K8S executor * createss a k8s virtualenv which installs all necessary dependencies that you can run tests against and allows you to enter it with `shell` command * it even has nice k9s integration (which is a fantastic tool for interacting with k8s) * and it even has a step-by-step screenshot guide on how to debug the K8s tests with IntelliJ. So I believe K8S executor is well covered already :) J. On Mon, Mar 22, 2021 at 11:08 PM Ryan Hatter <[email protected]> wrote: > Yeah I'll give that a shot. > > Do you think something with KubernetesExecutor with something like > MiniKube would also make sense? > > On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <[email protected]> wrote: > >> Actually it is not that strightforward, but possibly we can make it works >> much more easily >> >> In order to make Celery Executor works you need to do a bit more (but >> should be easy to add as an option to Breeze): >> >> * you need to start rabitmq or redis as integration (`--integration >> rabbitq --integration redis') >> * you need to start worker(s) (`airflow worker` in the background) >> * you might want to start flower optionally (the celery monitoring tool) >> >> So maybe we could add extra switch to start-airflow command >> (--use-celery-executor) that could set those integrations and start >> worker/flower additionally to running webserver/scheduler now in tmux ? >> >> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then >> as an option. >> >> BTW. We already have a number of CeleryExecutor tests that use the >> integrations, so Breeze has all what's needed: >> >> >> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109 >> >> J. >> >> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <[email protected]> >> wrote: >> >>> Hmm, maybe I was just getting twisted around with docker then. I’ll have >>> a look at what you shared. >>> >>> Thanks Bin :) >>> >>> On Mar 22, 2021, at 01:52, Xinbin Huang <[email protected]> wrote: >>> >>> >>> Hi Ryan, >>> >>> I believe breeze already provides tools for you to do that: >>> >>> 1.Would it make sense to allow developers to choose the executor for >>> their Breeze environment? >>> >>> You can set up environment variables and any other custom setup you want >>> in the file: */files/airflow-breeze-config/variables.env. *To set up >>> CeleryExecutor, you just need to put `export >>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file. >>> >>> 2. Developing using Docker >>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >>> would >>> require an update to the image each time you want to make a change to the >>> codebase >>> >>> Breeze automatically mounts the local source to the container unless you >>> explicitly skip it with the flag *--skip-mounting-local-sources. *You >>> can find more details here >>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze >>> >>> Best >>> Bin >>> >>> >>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <[email protected]> >>> wrote: >>> >>>> I recently had some trouble trying to fix a bug in the CeleryExecutor >>>> <https://github.com/apache/airflow/pull/14883>. The code change was >>>> small, but it was really difficult to set up a development environment >>>> using the CeleryExecutor. I ultimately had to muck around with the test >>>> case that covers this situation, default_airflow.cfg, and >>>> default_celery.py. Developing using Docker >>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >>>> would require an update to the image each time you want to make a change to >>>> the codebase (or maybe `exec`ing into the relevant container?) which is a >>>> pain. >>>> >>>> This led me to two questions: >>>> >>>> 1. Would it make sense to allow developers to choose the executor >>>> for their Breeze environment? >>>> 2. If not, how do folks test out changes they make to the >>>> CeleryExecutor or KubernetesExecutor? >>>> >>>> Thanks! >>>> Ryan >>>> >>>> >>>> >> >> -- >> +48 660 796 129 >> > -- +48 660 796 129
