Re: updating dags best practice
We're currently using CeleryExecutor ... On 21/08/2018, 15:12, "Driesprong, Fokko" wrote: Hi Aleksander, What kind of executor are you using? This is really important when choosing a strategy. For most of the Airflow deployments that I've worked with simply uses git to deploy the master branch. This is with a LocalExecutor. Cheers, Fokko Op di 21 aug. 2018 om 15:25 schreef Dev Aleksander > Hi all, > in the place I'm currently at we're building and redeploying a new set of > containers with the latest code every time we want to update a DAG. That > doesn't feel like the fastest way. > > Anyone can share their approach? > > Thanks, > Aleksander > The information in this email (and any attachments) is confidential and is intended solely for the use of the individual or entity to whom it is addressed. If you received this email in error please tell us by reply email (or telephone the sender) and delete all electronic copies on your system or other copies known to you. Trainline Investments Holdings Limited (Registered No.5776685), Trainline.com Limited (Registered No. 3846791) and Trainline International Limited (Registered No. 6881309) are all registered in England and Wales with registered office at 3rd floor, 120 Holborn, London, EC1N 2TD.
Re: updating dags best practice
We're currently using CeleryExecutor On Tue, Aug 21, 2018 at 3:11 PM, Driesprong, Fokko wrote: > Hi Aleksander, > > What kind of executor are you using? This is really important when choosing > a strategy. For most of the Airflow deployments that I've worked with > simply uses git to deploy the master branch. This is with a LocalExecutor. > > Cheers, Fokko > > Op di 21 aug. 2018 om 15:25 schreef Dev Aleksander > > > Hi all, > > in the place I'm currently at we're building and redeploying a new set of > > containers with the latest code every time we want to update a DAG. That > > doesn't feel like the fastest way. > > > > Anyone can share their approach? > > > > Thanks, > > Aleksander > > >
Re: updating dags best practice
We follow the strategy to re-build Docker with each deployment, because sometimes DAG changes can include new or updated dependencies. This Docker-first approach makes local development nice too (run same Docker images on local machine during debug as production). This isnt painful for us because we’ve also automated the dockerizarion, deployment, and graceful restart of the Airflow cluster with our CLI and API, and because we’re using a Docker repository, roll back is possible. -Ry Sent from mobile On Aug 21, 2018 at 8:25 AM, Dev Aleksander wrote: Hi all, in the place I'm currently at we're building and redeploying a new set of containers with the latest code every time we want to update a DAG. That doesn't feel like the fastest way. Anyone can share their approach? Thanks, Aleksander
Re: updating dags best practice
Hi Aleksander, What kind of executor are you using? This is really important when choosing a strategy. For most of the Airflow deployments that I've worked with simply uses git to deploy the master branch. This is with a LocalExecutor. Cheers, Fokko Op di 21 aug. 2018 om 15:25 schreef Dev Aleksander > Hi all, > in the place I'm currently at we're building and redeploying a new set of > containers with the latest code every time we want to update a DAG. That > doesn't feel like the fastest way. > > Anyone can share their approach? > > Thanks, > Aleksander >
updating dags best practice
Hi all, in the place I'm currently at we're building and redeploying a new set of containers with the latest code every time we want to update a DAG. That doesn't feel like the fastest way. Anyone can share their approach? Thanks, Aleksander
