Re: Airflow Deployment tools

Thank you all for your responses, this is very helpful! - Andy Hadjigeorgiou > On Nov 15, 2017, at 1:52 PM, Scott Halgrim > wrote: > > I bet the folks at astronomer.io would love to talk to you about Airflow as a > service > >> On Nov 15, 2017, 4:31 AM

Re: Airflow Deployment tools

I bet the folks at astronomer.io would love to talk to you about Airflow as a service On Nov 15, 2017, 4:31 AM -0800, Andrew Maguire , wrote: > Is there any options at all out there for Airflow as a service type > approach? > > I'd love to just be able to define my dags

Re: Airflow Deployment tools

Our Airflow situation: • Development happens in two different repos (a repo that holds a lot of cross-company python tools, for the core app and any plugins we develop on top of it, and our reporting tools/infra repo, for DAGs and related utility files). • The core app & plugins get packaged

Re: Airflow Deployment tools

Infrastructure wise we use docker containers, hosted via Kubernetes on Google Container Engine and deployed with Helm. We bake our DAGs and custom code into the images - so in the end the deployer does a `helm upgrade` command locally, the images are rebuilt with the newest code, and then all the

Re: Airflow Deployment tools

We are also using Ansible for: - Installing/upgrading/configuring Airflow (there are several airflow roles on git) - Deploying the pipelines - Restarting Airflow webserver/scheduler It would be great to have Airflow manageable from Hadoop cluster managers (Cloudera Manager, Ambari). For this a

Re: Airflow Deployment tools

Is there any options at all out there for Airflow as a service type approach? I'd love to just be able to define my dags and load them to some cloud ui and not have to worry about anything else. This looks kinda interesting -

Re: Airflow Deployment tools

I'm using Ansible to deploy the Airflow, the steps are: - First install Airflow using pip (or a rc using curl) - Do an `airflow version` to trigger the creation of the default config - Set the config correctly variables in the config using Ansible. - Deploy the supervisord files - Start everything

Re: Airflow Deployment tools

Samson deploy that runs a script running a Broadside deploy for ECS, which bounces the Web and Scheduler workers, and updates the DAG directory on the workers. Docker images come from a Github -> Travis -> Quay

Re: Airflow Deployment tools

Our TeamCity server detects the master branch has changed, then packages up the repo containing our DAGs as an artifact. We then use SaltStack to trigger a bash script on the targeted servers that downloads the artifact, moves the files to the right place, and restarts the scheduler (on the

Re: Airflow Deployment tools

Saltstack. On Tue, 14 Nov 2017, 18:14 Scott Halgrim, wrote: > We have a bash script. Main features > > > • Uses AWS metadata URL to determine if it’s a prod box or dev box and > pulls appropriate branches > • Gets difference between commit on box and commit on

Re: Airflow Deployment tools

We have a bash script. Main features • Uses AWS metadata URL to determine if it’s a prod box or dev box and pulls appropriate branches • Gets difference between commit on box and commit on remote tip and git logs the difference into Slack • Restarts all the upstart bits That’s on the worker.

Airflow Deployment tools

Hey, Was just wondering what tools & services everyone uses to deploy new versions of their data pipelines (understandably this would vary greatly based on tech stack) but I'd love to hear what the community has been using. - Andy