Re: [DISCUSS] Back to (some) dependency pinning

2020-03-24 Thread Driesprong, Fokko
Yes, I'd be in favor of not having two packages, and just pinning the versions then. In this case, all the versions will be pinned, so if a user wants to install a newer version of elastic, they have to do it explicitly. For Java, you have nice packages that will check if you break any public API,

Re: Helm Chart for Airflow

2020-03-24 Thread Aizhamal Nurmamat kyzy
+1 on the donation. Always happy to see more useful stuff for the community :) On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel wrote: > Yep, the cleanup_pods script is set up now as an optional Kubernetes > CronJob ( >

Re: Helm Chart for Airflow

2020-03-24 Thread Greg Neiheisel
Yep, the cleanup_pods script is set up now as an optional Kubernetes CronJob ( https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml) that we have run periodically to clean failed pods up and could stay separate. The wait_for_migrations script could

Re: Helm Chart for Airflow

2020-03-24 Thread Jarek Potiuk
> > @Tomasz great question. Our images are currently generated from Dockerfiles > in this repo https://github.com/astronomer/ap-airflow and get published to > DockerHub > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. > > For the most part those are typical Airflow images.

Re: Let's talk Airflow 2.0

2020-03-24 Thread Daniel Imberman
Hi Robin, > > I feel some of the stuff for instance Schedular HA could wait for a point > > release of version 2 (although maybe this a lot further a long than I am > > aware). Like you mentioned Spark did with K8s. > > I agree on this part. The focus would be more on the breaking changes and

Re: Helm Chart for Airflow

2020-03-24 Thread Greg Neiheisel
@Tomasz great question. Our images are currently generated from Dockerfiles in this repo https://github.com/astronomer/ap-airflow and get published to DockerHub https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. For the most part those are typical Airflow images. There's an

Re: Helm Chart for Airflow

2020-03-24 Thread Daniel Imberman
@jarek I agree completely. I think that pairing an official helm chart with the official image would make for a REALLY powerful “up and running with airflow” story :). Tomek and I have also been looking into operator-sdk which has the ability to create custom controllers from helm charts. We

Re: Helm Chart for Airflow

2020-03-24 Thread Jarek Potiuk
+1. And it should be paired with the official image we have work in progress on. I looked a lot at the Astronomer's image while preparing my draft and we can make any adjustments needed to make it works with the helm chart - and I am super happy to collaborate on that. PR here:

Re: Helm Chart for Airflow

2020-03-24 Thread Kaxil Naik
@Tomasz Urbaszek : Helm Chart Link: https://github.com/astronomer/airflow-chart On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek wrote: > An official helm chart is something our community needs! Using your > chart as the official makes a lot of sens to me because as you > mentioned - it's

Re: Helm Chart for Airflow

2020-03-24 Thread Tomasz Urbaszek
An official helm chart is something our community needs! Using your chart as the official makes a lot of sens to me because as you mentioned - it's battle tested. One question: what Airflow image do you use? Also, would you mind sharing a link to the chart? Tomek On Tue, Mar 24, 2020 at 2:07

Re: Let's talk Airflow 2.0

2020-03-24 Thread Kaxil Naik
+1 On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards wrote: > I feel some of the stuff for instance Schedular HA could wait for a point > release of version 2 (although maybe this a lot further a long than I am > aware). Like you mentioned Spark did with K8s. > > Also does the new API need to be

Re: [DISCUSS] Back to (some) dependency pinning

2020-03-24 Thread Jarek Potiuk
And yet another update: - after seeing how it works I will remove requirement generation from pre-commit - now that it needs to be generated separately for different versions of python it's a bit too much overhead (you'd need to have more images downloaded for different python versions for