We pretty much just shut everything off (fence the db by changing the password), upgrade the db, then turn everything back on and change the db password back. We accept that there will be downtime for maintenance and do it at a low impact time (like 20:00 utc ). Also, the thing with sqlalchemy is a lot more things become breaking changes because of the way it writes queries.
It might be good to have downtime even without schema changes to prevent weird behavior if the changeset is large. But for small stuff, depending on the scope of the change, workers don't really need to be restarted, scheduler can be handled manually, and webserver we do a rolling restart. On Wed, Oct 4, 2017 at 7:54 AM Thoralf Gutierrez <[email protected]> wrote: > Thanks for your answer Alex. > > I guess you mean it won't work if there is a _breaking_ schema change > right? But for new patch and minor versions (i.e. without breaking > changes), I'm guessing that upgrading the db first should do the trick? > > @Airbnb team, how do you all do it? If I remember correctly, you often > update your version of airflow to test out new releases. What does your > runbook for Airflow upgrades look like? > > Cheers, > Thoralf > > On Tue, 3 Oct 2017 at 13:04 Alex Guziel <[email protected]> > wrote: > > > You won't be able to if there's a schema change. > > > > On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez < > > [email protected]> wrote: > > > > > Hey everybody! > > > > > > Does anybody have some kind of runbook to upgrade airflow (with a > Celery > > > backend) without having any downtime (i.e. tasks keep on running as you > > > upgrade)? > > > > > > I have this in mind, but not sure if I am missing something or if I > > should > > > be careful with the order of steps (especially for upgrading the db > > > schema): > > > > > > 1. run airflow upgradedb from anywhere > > > > > > 2. one worker at a time > > > 2a. make sure it doesn't start any new task. > > > 2b. wait for all tasks to be finished > > > 2c. run pip install airflow --upgrade > > > 2d. re-enable worker > > > > > > 3. one webserver at a time > > > 3a. kill webserver > > > 3b. run pip install airflow --upgrade > > > 3c. start webserver > > > > > > 4. scheduler > > > 4a. kill scheduler > > > 4b. run pip intsall airflow --upgrade > > > 4c. start scheduler > > > > > > Thanks, > > > Thoralf > > > > > >
