Thank you all for your responses, this is very helpful! - Andy Hadjigeorgiou
> On Nov 15, 2017, at 1:52 PM, Scott Halgrim <[email protected]> > wrote: > > I bet the folks at astronomer.io would love to talk to you about Airflow as a > service > >> On Nov 15, 2017, 4:31 AM -0800, Andrew Maguire <[email protected]>, >> wrote: >> Is there any options at all out there for Airflow as a service type >> approach? >> >> I'd love to just be able to define my dags and load them to some cloud ui >> and not have to worry about anything else. >> >> This looks kinda interesting - >> http://docs.qubole.com/en/latest/user-guide/airflow/introduction-airflow.html >> >> Cheers, >> Andy >> >> On Wed, Nov 15, 2017 at 10:28 AM Driesprong, Fokko <[email protected] >> wrote: >> >>> I'm using Ansible to deploy the Airflow, the steps are: >>> - First install Airflow using pip (or a rc using curl) >>> - Do an `airflow version` to trigger the creation of the default config >>> - Set the config correctly variables in the config using Ansible. >>> - Deploy the supervisord files >>> - Start everything >>> >>> A separate role is there to deploy Postgres. But if you are working on a >>> cloud environment, you can also get Postgres/MySQL as a service. Hope this >>> helps. >>> >>> Cheers, Fokko >>> >>> 2017-11-15 3:19 GMT+01:00 Marc Bollinger <[email protected]>: >>> >>>> Samson <https://github.com/zendesk/samson> deploy that runs a script >>>> running a Broadside <https://github.com/lumoslabs/broadside> deploy for >>>> ECS, which bounces the Web and Scheduler workers, and updates the DAG >>>> directory on the workers. Docker images come from a Github -> Travis - >>>> Quay >>>> <https://quay.io/> CI setup. >>>> >>>> On Tue, Nov 14, 2017 at 10:18 AM, Alek Storm <[email protected] >>> wrote: >>>> >>>>> Our TeamCity server detects the master branch has changed, then >>> packages >>>> up >>>>> the repo containing our DAGs as an artifact. We then use SaltStack to >>>>> trigger a bash script on the targeted servers that downloads the >>>> artifact, >>>>> moves the files to the right place, and restarts the scheduler (on the >>>>> master). >>>>> >>>>> This allows us to easily revert changes by redeploying a particular >>>>> TeamCity artifact, without touching the git history. >>>>> >>>>> Alek >>>>> >>>>> On Nov 14, 2017 11:02 AM, "Andy Hadjigeorgiou" <[email protected] >>>>> wrote: >>>>> >>>>>> Hey, >>>>>> >>>>>> Was just wondering what tools & services everyone uses to deploy new >>>>>> versions of their data pipelines (understandably this would vary >>>> greatly >>>>>> based on tech stack) but I'd love to hear what the community has been >>>>>> using. >>>>>> >>>>>> - Andy >>>>>> >>>>> >>>> >>>
