This one time, at weekly catchup call, Francis and I spoke about this plan ... :)
We both think that its going to take significant time to get all the pieces in place, so we're proposing to make this whole thing even more incrementally adoptable. So - first up, a 'daily staging' environment, which is RT 40482 qa with the stable branch and production schema. This will let us start QA'ing what we rollout on a daily basis (vs the monthly cycle for db-stable). QA seem to be pretty much ready for this - can you confirm that this is in fact the case? We'll also want to fix RT 40685 deploy icing to apache before updating appservers, because rollouts to appservers currently cause a few glitches. When the branch 'stable' is completely QA'd, a rollout of *just the appservers* will be triggered (initially by asking a LOSA to hit the button), which includes stable getting propogated to the 'production' branch as normal. Once these two things are done and we're successfully executing this rollout pattern, we can just get rid of the 'edge' config, 'edge' appservers and so on. We'll keep the edge DNS entry, with an apache config that simply redirects to the same url on production. is_edge in our codebase will start returning False *everywhere*. We discussed an alternative, where we kept edge working as a DNS alias for production, and turned on a feature scope for edge; but we felt that the gains were marginal (users could opt in/out of *one* scope - the edge scope - directly) and something we want a better answer for anyway (users should be able to opt into / out of things on a more granular basis - e.g. participate in A/B testing and choose to get the original format page back because they don't like the new one). With that level of control over scopes, an implicit edge scope started to feel positively primordial, and it has lots of costs - an ongoing redirect to edge causes performance issues in new browser sessions, we'd have more QA work to do, because its a global scope that would be affecting every page, and so forth. This eliminates some complexity from the production environment immediately : rather than 1 revision for all backend servers + some appservers, and a different revision for the edge appservers, we will have one revision for all backend servers and one revision for all the appservers. It also means that changes which you're experimenting with should be hidden by feature flags. These are available in DB-stable now - so please do start using them asap for things you are doing in db-devel; we'll need to develop some proficiency with it (and there are still significant concerns Curtis raised in an earlier thread that we need to put in place preventative measures for :- the sooner we have concrete data on them the easier is it change our path accordingly. The next steps after that are to iterate on the reliability of doing backend rollouts - cronscripts, single points of failure etc - until we have just a single revision everywhere. At that point, we can start working on the database patch deployment story - but expect that to be 2-3 months away for now. I will (honestly!) update the wiki page tomorrow, but I wanted to get this proposal out to you guys asap, so you can tell me how its broken! -Rob _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
