I see - thanks for the answers! On Wed, 4 Oct 2017 at 13:10 Alex Guziel <[email protected]> wrote:
> We pretty much just shut everything off (fence the db by changing the > password), upgrade the db, then turn everything back on and change the db > password back. We accept that there will be downtime for maintenance and do > it at a low impact time (like 20:00 utc ). Also, the thing with sqlalchemy > is a lot more things become breaking changes because of the way it writes > queries. > > It might be good to have downtime even without schema changes to prevent > weird behavior if the changeset is large. > > But for small stuff, depending on the scope of the change, workers don't > really need to be restarted, scheduler can be handled manually, and > webserver we do a rolling restart. > > On Wed, Oct 4, 2017 at 7:54 AM Thoralf Gutierrez < > [email protected]> > wrote: > > > Thanks for your answer Alex. > > > > I guess you mean it won't work if there is a _breaking_ schema change > > right? But for new patch and minor versions (i.e. without breaking > > changes), I'm guessing that upgrading the db first should do the trick? > > > > @Airbnb team, how do you all do it? If I remember correctly, you often > > update your version of airflow to test out new releases. What does your > > runbook for Airflow upgrades look like? > > > > Cheers, > > Thoralf > > > > On Tue, 3 Oct 2017 at 13:04 Alex Guziel <[email protected]> > > wrote: > > > > > You won't be able to if there's a schema change. > > > > > > On Tue, Oct 3, 2017 at 12:33 PM, Thoralf Gutierrez < > > > [email protected]> wrote: > > > > > > > Hey everybody! > > > > > > > > Does anybody have some kind of runbook to upgrade airflow (with a > > Celery > > > > backend) without having any downtime (i.e. tasks keep on running as > you > > > > upgrade)? > > > > > > > > I have this in mind, but not sure if I am missing something or if I > > > should > > > > be careful with the order of steps (especially for upgrading the db > > > > schema): > > > > > > > > 1. run airflow upgradedb from anywhere > > > > > > > > 2. one worker at a time > > > > 2a. make sure it doesn't start any new task. > > > > 2b. wait for all tasks to be finished > > > > 2c. run pip install airflow --upgrade > > > > 2d. re-enable worker > > > > > > > > 3. one webserver at a time > > > > 3a. kill webserver > > > > 3b. run pip install airflow --upgrade > > > > 3c. start webserver > > > > > > > > 4. scheduler > > > > 4a. kill scheduler > > > > 4b. run pip intsall airflow --upgrade > > > > 4c. start scheduler > > > > > > > > Thanks, > > > > Thoralf > > > > > > > > > >
