Re: Helm Chart for Airflow

Congrats! Looking forward to start using it for tests (I am going to use it also for some performance testing soon). J. On Fri, Jun 19, 2020 at 4:09 PM Greg Neiheisel wrote: > Hey everyone, > > If you haven't been following along on the PR > , it wa

Re: Helm Chart for Airflow

Hey everyone, If you haven't been following along on the PR , it was recently merged! The helm chart now lives under the chart directory inside the Airflow repo. Thanks to everyone who's contributed over the years to this chart or any of the other open

Re: Helm Chart for Airflow

Thanks, Greg 🚀 On Fri, May 8, 2020 at 4:19 AM Greg Neiheisel wrote: > Awesome Jarek. I just opened up a draft PR that adds the chart into the > Airflow repo. https://github.com/apache/airflow/pull/8777 > > It still contains some of our packaging, testing and releasing scripts. I > figure they ca

Re: Helm Chart for Airflow

Cool. Wil look into it over weekend. If you think anything can be improved in the image to make it a better fit - fill free to create new issue or comment the existing issues https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image On Fri, May 8, 2020 at 5:1

Re: Helm Chart for Airflow

Awesome Jarek. I just opened up a draft PR that adds the chart into the Airflow repo. https://github.com/apache/airflow/pull/8777 It still contains some of our packaging, testing and releasing scripts. I figure they can serve as a starting point to discuss how we'd want to replicate into Airflow i

Re: Helm Chart for Airflow

Hey Greg, I just wanted to update you with some recent changes I've done to automate most of the image builds stuff we have in Airflow. That might be useful for your efforts. I am adding (and I will fix some of them very soon) issues based on feedback from people on the #prod-docker-image channel

Re: Helm Chart for Airflow

Awesome, thanks for the update! On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel wrote: > Hey everyone, just wanted to post a quick update on the helm chart. We're > in the process of removing any references to Astronomer specific things and > making them more general features. That work is happen

Re: Helm Chart for Airflow

Hey everyone, just wanted to post a quick update on the helm chart. We're in the process of removing any references to Astronomer specific things and making them more general features. That work is happening on this PR here https://github.com/astronomer/airflow-chart/pull/66. Hoping to wrap up the

Re: Helm Chart for Airflow

Hi Jarek, Congrats on getting the image out! This is going to make a lot of things easier :). We’ve actually been working on separating out KinD as well (and in fact moving it out of travis and into GitHub issues). Let me circle back with Steven (the engineer who’s been working on that) and get

Re: Helm Chart for Airflow

Hello Greg and Daniel, Production image is merged and it's also going to be released in 1.10.10. For now, I will manually tag the release in DockerHub when we release but we will automate that in the future. Currently, we have *airflow:master* and *airflow:v1-10-test* images (for all supported pyt

Re: Helm Chart for Airflow

Hello Daniel and Greg. I think the production image for Airflow is quite ready for the final review and merge. I did some tests and I think it's quite complete now. The PR is running it's Travis build: https://github.com/apache/airflow/pull/7832 I already pushed the image to DockerHub - so that

Re: Helm Chart for Airflow

Absolutely! Please do :). The more eyes on this the better! On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman wrote: > @Jarek Steven and I will look into getting KinD to work on GitHub actions > today. If we can get that working, we can donate the helm chart and > possibly even deprecate the old K

Re: Helm Chart for Airflow

@Jarek Steven and I will look into getting KinD to work on GitHub actions today. If we can get that working, we can donate the helm chart and possibly even deprecate the old Kubernetes tests in the process. On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk , wrote: > We have A POC for Github Actions d

Re: Helm Chart for Airflow

We have A POC for Github Actions done by Tomek, and the blocker there was precisely Kubernetes tests and specifically running kind cluster in the Github Actions https://github.com/PolideaInternal/airflow/pull/542 . In the meantime Jiaje Zhong is trying to make it works on GA we discussed this very

Re: Helm Chart for Airflow

<3 On Thu, Mar 26, 2020 at 3:59 PM Daniel Imberman wrote: > @Jarek I think with the helm chart + prod image we can go even further > than that :). We can test CeleryExecutor, with KEDA autoscaling, and a > bunch of other configurations. > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk , > wrote: >

Re: Helm Chart for Airflow

@Steven glad to help you out on that. For the moment we only use travis, but I believe @jarek has been looking to move to GitHub CI/CD. Perhaps we can use this as our first GitHub tests? On Mar 26, 2020, 8:17 AM -0700, Steven Miller , wrote: > Hey I’m happy to PR in the chart and get the CI work

Re: Helm Chart for Airflow

Hey I’m happy to PR in the chart and get the CI working to test it. Where do we want to run it? Do you all just use Travis only? Steven On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman wrote: > @Jarek I think with the helm chart + prod image we can go even further > than that :). We can test Ce

Re: Helm Chart for Airflow

@Jarek I think with the helm chart + prod image we can go even further than that :). We can test CeleryExecutor, with KEDA autoscaling, and a bunch of other configurations. On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk , wrote: > Yeah. I meant Custom Resources  not CRDs in my original email :) > >

Re: Helm Chart for Airflow

Yeah. I meant Custom Resources not CRDs in my original email :) On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman wrote: > We’re not using CRDs for the tests at the moment. We just have deployment > files. If anything having the helm chart as a part of the airflow repo > could mean that the helm

Re: Helm Chart for Airflow

Cool! Everything is here: https://github.com/apache/airflow/tree/master/scripts/ci/in_container/kubernetes In short: In "kubernetes" dir: setup_kind_cluster.sh + kind-cluster-conf.yaml -> they are use to create a kind Kubernetes cluster. In "docker" subdir: rebuild_airflow_im

Re: Helm Chart for Airflow

We’re not using CRDs for the tests at the moment. We just have deployment files. If anything having the helm chart as a part of the airflow repo could mean that the helm chart becomes the defacto system for testing airflow on kubernetes (we can get rid of all the yams files and run multiple k8s

Re: Helm Chart for Airflow

Yep, we can absolutely pull it into the airflow repo. We've also been building up a test suite that currently runs on CircleCI and uses kind (Kubernetes in Docker) to test several kubernetes versions with some different settings. Right now we're mostly testing the different executors since that has

Re: Helm Chart for Airflow

One thing for the donation. Did you you want to have separate repository Greg ? I think we should simply create a folder in Airflow repo and keep it there (similarly as we keep Dockerfile). I am going to switch our Kubernetes Tests to the production image (will make the tests much faster) and I a

Re: Helm Chart for Airflow

+1 on the donation. Always happy to see more useful stuff for the community :) On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel wrote: > Yep, the cleanup_pods script is set up now as an optional Kubernetes > CronJob ( > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/clea

Re: Helm Chart for Airflow

Yep, the cleanup_pods script is set up now as an optional Kubernetes CronJob ( https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml) that we have run periodically to clean failed pods up and could stay separate. The wait_for_migrations script could definit

Re: Helm Chart for Airflow

> > @Tomasz great question. Our images are currently generated from Dockerfiles > in this repo https://github.com/astronomer/ap-airflow and get published to > DockerHub > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. > > For the most part those are typical Airflow images. There

Re: Helm Chart for Airflow

@Tomasz great question. Our images are currently generated from Dockerfiles in this repo https://github.com/astronomer/ap-airflow and get published to DockerHub https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. For the most part those are typical Airflow images. There's an entrypo

Re: Helm Chart for Airflow

@jarek I agree completely. I think that pairing an official helm chart with the official image would make for a REALLY powerful “up and running with airflow” story :). Tomek and I have also been looking into operator-sdk which has the ability to create custom controllers from helm charts. We mig

Re: Helm Chart for Airflow

+1. And it should be paired with the official image we have work in progress on. I looked a lot at the Astronomer's image while preparing my draft and we can make any adjustments needed to make it works with the helm chart - and I am super happy to collaborate on that. PR here: https://github.com/

Re: Helm Chart for Airflow

@Tomasz Urbaszek : Helm Chart Link: https://github.com/astronomer/airflow-chart On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek wrote: > An official helm chart is something our community needs! Using your > chart as the official makes a lot of sens to me because as you > mentioned - it's battle

Re: Helm Chart for Airflow

An official helm chart is something our community needs! Using your chart as the official makes a lot of sens to me because as you mentioned - it's battle tested. One question: what Airflow image do you use? Also, would you mind sharing a link to the chart? Tomek On Tue, Mar 24, 2020 at 2:07 PM