Hi Raman, are you using Docker? We manage all of that with Docker and Nomad.
On Tue, Jun 18, 2019, 10:54 PM [email protected] <[email protected]> wrote: > Thanks David & Teresa, > We are thinking of similar kind of approach where we would have a > dedicated Test env to test the Airflow upgrades. We have to preserve the > metadata so we will be upgrading the Airflow version using Airflow cmd. > Following steps would be done > -> Setup Mysql Slave to Airflow's Mysql Metastore. > -> Stop/Kill Airflow scheduler & server > -> Detach Mysql Slave(It can be used for rollbacks) > -> Upgrade Airflow Server & Scheduler Image > -> Invoke airflow cli fro server image to upgrade DB > -> Start Airflow server and scheduler. > Restore metastore from slave If anything goes bad while upgrading Airflow > cluster. > > > Thanks, > Raman Gupta > > On 2019/06/12 17:39:11, David Klosowski <[email protected]> wrote: > > Hi Raman, > > > > We do much the same that Teresa does. We just have a staging and > > production instances and separated infrastructure environments for normal > > testing and deployments. We do one of two things with upgrades > > > > 1. Build out a separate environment just for upgrade testing (which > doesn't > > take too long with Docker and cloud infrastructure) > > 2. Block our CI pipeline while validation of core features is done in > > staging > > > > We also. > > > > 1. Read the upgrade guide in the Airflow project > > 2. Review the JIRA tickets between versions > > 3. Assess any changes we may need to make to features like logging or our > > components (subdags, retry handlers, alerting, airflow variables, airflow > > configuration, UI etc) > > 4. Do end-to-end tests of our core custom operators and sensors verifying > > execution and retry cases > > > > All-in-all I don't think I'd feel comfortable automating this yet b/c I > > think problems can exist in hidden places. > > > > Cheers, > > David > > > > On Wed, Jun 12, 2019 at 5:57 AM Teresa Martyny < > > [email protected]> wrote: > > > > > Hi Raman, > > > We have a staging and preproduction instance. We block deploy of the > docker > > > image to production on the successful full run of our core pipeline > dags in > > > preproduction. > > > > > > If there's enough risk, we will disable the job that deploys to > > > preproduction, duplicate it, and deploy a branch to preproduction. > Then we > > > can watch it run against the core pipeline and all external extracts > for a > > > few runs to feel even more confident. At which point we will let it > through > > > to master and our regular CI. > > > > > > Updating dependencies is the kind of scenario where we deploy a branch > to > > > preproduction or staging depending on what the dependency is since the > risk > > > is high. Another helpful thing is to wait a little and watch the > issues in > > > Jira. > > > > > > Hope that helps, > > > Teresa > > > > > > > > > On Wed, Jun 12, 2019, 1:14 AM [email protected] < > [email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > Are there any recommended steps/process to upgrade Airflow in > production. > > > > We have a airflow setup in prod which is being actively used by > different > > > > users. So ideally we would want to have a seamless upgrade but it > seems > > > > Airflow upgrade are not backward compatible and might break few > things. > > > > So what are the best practices for Airflow Upgrade in Production. > > > > > > > > Thanks, > > > > Raman Gupta > > > > > > > > > > -- > > > This email may contain material that is confidential and/or privileged > for > > > the sole use of the intended recipient. Any review, reliance, or > > > distribution by others or forwarding without express permission is > > > strictly > > > prohibited. If you are not the intended recipient, please contact the > > > sender and delete all copies. Also note that email is not an > appropriate > > > way to send protected health information to Omada Health employees. > Please > > > use your discretion when responding to this email. > > > > > > -- This email may contain material that is confidential and/or privileged for the sole use of the intended recipient. Any review, reliance, or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. Also note that email is not an appropriate way to send protected health information to Omada Health employees. Please use your discretion when responding to this email.
