Yup, We are using Docker. Thanks, Raman gupta
On 2019/06/19 13:25:04, Teresa Martyny <[email protected]> wrote: > Hi Raman, are you using Docker? > We manage all of that with Docker and Nomad. > > On Tue, Jun 18, 2019, 10:54 PM [email protected] <[email protected]> > wrote: > > > Thanks David & Teresa, > > We are thinking of similar kind of approach where we would have a > > dedicated Test env to test the Airflow upgrades. We have to preserve the > > metadata so we will be upgrading the Airflow version using Airflow cmd. > > Following steps would be done > > -> Setup Mysql Slave to Airflow's Mysql Metastore. > > -> Stop/Kill Airflow scheduler & server > > -> Detach Mysql Slave(It can be used for rollbacks) > > -> Upgrade Airflow Server & Scheduler Image > > -> Invoke airflow cli fro server image to upgrade DB > > -> Start Airflow server and scheduler. > > Restore metastore from slave If anything goes bad while upgrading Airflow > > cluster. > > > > > > Thanks, > > Raman Gupta > > > > On 2019/06/12 17:39:11, David Klosowski <[email protected]> wrote: > > > Hi Raman, > > > > > > We do much the same that Teresa does. We just have a staging and > > > production instances and separated infrastructure environments for normal > > > testing and deployments. We do one of two things with upgrades > > > > > > 1. Build out a separate environment just for upgrade testing (which > > doesn't > > > take too long with Docker and cloud infrastructure) > > > 2. Block our CI pipeline while validation of core features is done in > > > staging > > > > > > We also. > > > > > > 1. Read the upgrade guide in the Airflow project > > > 2. Review the JIRA tickets between versions > > > 3. Assess any changes we may need to make to features like logging or our > > > components (subdags, retry handlers, alerting, airflow variables, airflow > > > configuration, UI etc) > > > 4. Do end-to-end tests of our core custom operators and sensors verifying > > > execution and retry cases > > > > > > All-in-all I don't think I'd feel comfortable automating this yet b/c I > > > think problems can exist in hidden places. > > > > > > Cheers, > > > David > > > > > > On Wed, Jun 12, 2019 at 5:57 AM Teresa Martyny < > > > [email protected]> wrote: > > > > > > > Hi Raman, > > > > We have a staging and preproduction instance. We block deploy of the > > docker > > > > image to production on the successful full run of our core pipeline > > dags in > > > > preproduction. > > > > > > > > If there's enough risk, we will disable the job that deploys to > > > > preproduction, duplicate it, and deploy a branch to preproduction. > > Then we > > > > can watch it run against the core pipeline and all external extracts > > for a > > > > few runs to feel even more confident. At which point we will let it > > through > > > > to master and our regular CI. > > > > > > > > Updating dependencies is the kind of scenario where we deploy a branch > > to > > > > preproduction or staging depending on what the dependency is since the > > risk > > > > is high. Another helpful thing is to wait a little and watch the > > issues in > > > > Jira. > > > > > > > > Hope that helps, > > > > Teresa > > > > > > > > > > > > On Wed, Jun 12, 2019, 1:14 AM [email protected] < > > [email protected]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Are there any recommended steps/process to upgrade Airflow in > > production. > > > > > We have a airflow setup in prod which is being actively used by > > different > > > > > users. So ideally we would want to have a seamless upgrade but it > > seems > > > > > Airflow upgrade are not backward compatible and might break few > > things. > > > > > So what are the best practices for Airflow Upgrade in Production. > > > > > > > > > > Thanks, > > > > > Raman Gupta > > > > > > > > > > > > > -- > > > > This email may contain material that is confidential and/or privileged > > for > > > > the sole use of the intended recipient. Any review, reliance, or > > > > distribution by others or forwarding without express permission is > > > > strictly > > > > prohibited. If you are not the intended recipient, please contact the > > > > sender and delete all copies. Also note that email is not an > > appropriate > > > > way to send protected health information to Omada Health employees. > > Please > > > > use your discretion when responding to this email. > > > > > > > > > > > -- > This email may contain material that is confidential and/or privileged for > the sole use of the intended recipient. Any review, reliance, or > distribution by others or forwarding without express permission is strictly > prohibited. If you are not the intended recipient, please contact the > sender and delete all copies. Also note that email is not an appropriate > way to send protected health information to Omada Health employees. Please > use your discretion when responding to this email. >
