Excerpts from Fox, Kevin M's message of 2017-11-15 00:37:26 +0000: > I can think of a few ideas, though some sound painful on paper.... Not really > recommending anything, just thinking out loud... > > One idea is that at the root of chaos monkey. If something is hard, do it > frequently. If upgrading is hard, we need to be doing it constantly so the > pain gets largely eliminated. One idea would be to discourage the use of > standing up a fresh devstack all the time by devs and have them upgrade them > instead. If its hard, then its likely someone will chip in to make it less > hard. > > Another is devstack in general. the tooling used by devs and that used by ops > are so different as to isolate the devs from ops' pain. If they used more > opsish tooling, then they would hit the same issues and would be more likely > to find solutions that work for both parties. > > A third one is supporting multiple version upgrades in the gate. I rarely > have a problem with a cloud has a database one version back. I have seen lots > of issues with databases that contain data back when the cloud was > instantiated and then upgraded multiple times. > > Another option is trying to unify/detangle the upgrade procedure. upgrading > compute kit should be one or two commands if you can live with the defaults. > Not weeks of poring through release notes, finding correct orders from pages > of text and testing vigorously on test systems.
This sounds like an opportunity for some knowledge sharing. Maybe when the Operators' Guide makes it into the wiki? > > How about some tool that does the: dump database to somewhere temporary, > iterate over all the upgrade job components, and see if it will successfully > not corrupt your database. That takes a while to do manually. Ideally it > could even upload stacktraces back a bug tracker for attention. > > Thanks, > Kevin > ________________________________________ > From: Davanum Srinivas [[email protected]] > Sent: Tuesday, November 14, 2017 4:08 PM > To: OpenStack Development Mailing List (not for usage questions) > Cc: openstack-oper. > Subject: Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases > > On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson <[email protected]> wrote: > > > > > > On 14 Nov 2017, at 15:18, Mathieu Gagné wrote: > > > >> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M <[email protected]> wrote: > >>> The pressure for #2 comes from the inability to skip upgrades and the > >>> fact that upgrades are hugely time consuming still. > >>> > >>> If you want to reduce the push for number #2 and help developers get > >>> their wish of getting features into users hands sooner, the path to > >>> upgrade really needs to be much less painful. > >>> > >> > >> +1000 > >> > >> We are upgrading from Kilo to Mitaka. It took 1 year to plan and > >> execute the upgrade. (and we skipped a version) > >> Scheduling all the relevant internal teams is a monumental task > >> because we don't have dedicated teams for those projects and they have > >> other priorities. > >> Upgrading affects a LOT of our systems, some we don't fully have > >> control over. And it can takes months to get new deployment on those > >> systems. (and after, we have to test compatibility, of course) > >> > >> So I guess you can understand my frustration when I'm told to upgrade > >> more often and that skipping versions is discouraged/unsupported. > >> At the current pace, I'm just falling behind. I *need* to skip > >> versions to keep up. > >> > >> So for our next upgrades, we plan on skipping even more versions if > >> the database migration allows it. (except for Nova which is a huge > >> PITA to be honest due to CellsV1) > >> I just don't see any other ways to keep up otherwise. > > > > ?!?! > > > > What does it take for this to never happen again? No operator should need > > to plan and execute an upgrade for a whole year to upgrade one year's worth > > of code development. > > > > We don't need new policies, new teams, more releases, fewer releases, or > > anything like that. The goal is NOT "let's have an LTS release". The goal > > should be "How do we make sure Mattieu and everyone else in the world can > > actually deploy and use the software we are writing?" > > > > Can we drop the entire LTS discussion for now and focus on "make upgrades > > take less than a year" instead? After we solve that, let's come back around > > to LTS versions, if needed. I know there's already some work around that. > > Let's focus there and not be distracted about the best bureaucracy for not > > deleting two-year-old branches. > > > > > > --John > > John, > > So... Any concrete ideas on how to achieve that? > > Thanks, > Dims > > > > > > > /me puts on asbestos pants > > > >> > >> -- > >> Mathieu > >> > >> _______________________________________________ > >> OpenStack-operators mailing list > >> [email protected] > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: [email protected]?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
