OH and BTW - this is the very same setup that is used on CI to run the k8s tests ;)
On Tue, Mar 23, 2021 at 12:07 AM Jarek Potiuk <[email protected]> wrote: > > Oh, I did not write about K8S Executor because we already have K8S > executor support in Breeze. It does not use Breeze docker image - it uses > KinD (Kubernetes-in-Docker) that has all the bells and whistles (and more). > > Here is the part of Breeze video when I showcase it: > > https://github.com/apache/airflow/blob/master/BREEZE.rst#running-kubernetes-tests > > And here the whole chapter describing how to test Airflow with K8S: > > https://github.com/apache/airflow/blob/master/TESTING.rst#running-tests-with-kubernetes > > What we have built-in with Breeze (automagically): > > * configures and start kind cluster (kind is a really nice > development-focus installation of K8S that uses K8S-in-Docker approach). > * deploys airflow with K8S executor > * createss a k8s virtualenv which installs all necessary dependencies that > you can run tests against and allows you to enter it with `shell` command > * it even has nice k9s integration (which is a fantastic tool for > interacting with k8s) > * and it even has a step-by-step screenshot guide on how to debug the K8s > tests with IntelliJ. > > So I believe K8S executor is well covered already :) > > J. > > > On Mon, Mar 22, 2021 at 11:08 PM Ryan Hatter <[email protected]> > wrote: > >> Yeah I'll give that a shot. >> >> Do you think something with KubernetesExecutor with something like >> MiniKube would also make sense? >> >> On Mon, Mar 22, 2021 at 5:51 PM Jarek Potiuk <[email protected]> wrote: >> >>> Actually it is not that strightforward, but possibly we can make it >>> works much more easily >>> >>> In order to make Celery Executor works you need to do a bit more (but >>> should be easy to add as an option to Breeze): >>> >>> * you need to start rabitmq or redis as integration (`--integration >>> rabbitq --integration redis') >>> * you need to start worker(s) (`airflow worker` in the background) >>> * you might want to start flower optionally (the celery monitoring tool) >>> >>> So maybe we could add extra switch to start-airflow command >>> (--use-celery-executor) that could set those integrations and start >>> worker/flower additionally to running webserver/scheduler now in tmux ? >>> >>> WDYT? Maybe Ryan you can check if my recipe works ? I could add it then >>> as an option. >>> >>> BTW. We already have a number of CeleryExecutor tests that use the >>> integrations, so Breeze has all what's needed: >>> >>> >>> https://github.com/apache/airflow/blob/master/tests/executors/test_celery_executor.py#L109 >>> >>> J. >>> >>> On Mon, Mar 22, 2021 at 2:24 PM Ryan Hatter <[email protected]> >>> wrote: >>> >>>> Hmm, maybe I was just getting twisted around with docker then. I’ll >>>> have a look at what you shared. >>>> >>>> Thanks Bin :) >>>> >>>> On Mar 22, 2021, at 01:52, Xinbin Huang <[email protected]> wrote: >>>> >>>> >>>> Hi Ryan, >>>> >>>> I believe breeze already provides tools for you to do that: >>>> >>>> 1.Would it make sense to allow developers to choose the executor for >>>> their Breeze environment? >>>> >>>> You can set up environment variables and any other custom setup you >>>> want in the file: */files/airflow-breeze-config/variables.env. *To set >>>> up CeleryExecutor, you just need to put `export >>>> AIRFLOW__CORE__EXECUTOR=CeleryExecutor` in the file. >>>> >>>> 2. Developing using Docker >>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >>>> would >>>> require an update to the image each time you want to make a change to the >>>> codebase >>>> >>>> Breeze automatically mounts the local source to the container unless >>>> you explicitly skip it with the flag *--skip-mounting-local-sources. *You >>>> can find more details here >>>> https://github.com/apache/airflow/blob/master/BREEZE.rst#mounting-local-sources-to-breeze >>>> >>>> Best >>>> Bin >>>> >>>> >>>> On Sun, Mar 21, 2021 at 5:32 PM Ryan Hatter <[email protected]> >>>> wrote: >>>> >>>>> I recently had some trouble trying to fix a bug in the CeleryExecutor >>>>> <https://github.com/apache/airflow/pull/14883>. The code change was >>>>> small, but it was really difficult to set up a development environment >>>>> using the CeleryExecutor. I ultimately had to muck around with the test >>>>> case that covers this situation, default_airflow.cfg, and >>>>> default_celery.py. Developing using Docker >>>>> <https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html> >>>>> would require an update to the image each time you want to make a change >>>>> to >>>>> the codebase (or maybe `exec`ing into the relevant container?) which is a >>>>> pain. >>>>> >>>>> This led me to two questions: >>>>> >>>>> 1. Would it make sense to allow developers to choose the executor >>>>> for their Breeze environment? >>>>> 2. If not, how do folks test out changes they make to the >>>>> CeleryExecutor or KubernetesExecutor? >>>>> >>>>> Thanks! >>>>> Ryan >>>>> >>>>> >>>>> >>> >>> -- >>> +48 660 796 129 >>> >> > > -- > +48 660 796 129 > -- +48 660 796 129
