Hi Raman, We do much the same that Teresa does. We just have a staging and production instances and separated infrastructure environments for normal testing and deployments. We do one of two things with upgrades
1. Build out a separate environment just for upgrade testing (which doesn't take too long with Docker and cloud infrastructure) 2. Block our CI pipeline while validation of core features is done in staging We also. 1. Read the upgrade guide in the Airflow project 2. Review the JIRA tickets between versions 3. Assess any changes we may need to make to features like logging or our components (subdags, retry handlers, alerting, airflow variables, airflow configuration, UI etc) 4. Do end-to-end tests of our core custom operators and sensors verifying execution and retry cases All-in-all I don't think I'd feel comfortable automating this yet b/c I think problems can exist in hidden places. Cheers, David On Wed, Jun 12, 2019 at 5:57 AM Teresa Martyny < [email protected]> wrote: > Hi Raman, > We have a staging and preproduction instance. We block deploy of the docker > image to production on the successful full run of our core pipeline dags in > preproduction. > > If there's enough risk, we will disable the job that deploys to > preproduction, duplicate it, and deploy a branch to preproduction. Then we > can watch it run against the core pipeline and all external extracts for a > few runs to feel even more confident. At which point we will let it through > to master and our regular CI. > > Updating dependencies is the kind of scenario where we deploy a branch to > preproduction or staging depending on what the dependency is since the risk > is high. Another helpful thing is to wait a little and watch the issues in > Jira. > > Hope that helps, > Teresa > > > On Wed, Jun 12, 2019, 1:14 AM [email protected] <[email protected]> > wrote: > > > Hi All, > > > > Are there any recommended steps/process to upgrade Airflow in production. > > We have a airflow setup in prod which is being actively used by different > > users. So ideally we would want to have a seamless upgrade but it seems > > Airflow upgrade are not backward compatible and might break few things. > > So what are the best practices for Airflow Upgrade in Production. > > > > Thanks, > > Raman Gupta > > > > -- > This email may contain material that is confidential and/or privileged for > the sole use of the intended recipient. Any review, reliance, or > distribution by others or forwarding without express permission is > strictly > prohibited. If you are not the intended recipient, please contact the > sender and delete all copies. Also note that email is not an appropriate > way to send protected health information to Omada Health employees. Please > use your discretion when responding to this email. >
