Excerpts from Fox, Kevin M's message of 2017-11-15 00:37:26 +0000:
> I can think of a few ideas, though some sound painful on paper.... Not really 
> recommending anything, just thinking out loud...
> 
> One idea is that at the root of chaos monkey. If something is hard, do it 
> frequently. If upgrading is hard, we need to be doing it constantly so the 
> pain gets largely eliminated. One idea would be to discourage the use of 
> standing up a fresh devstack all the time by devs and have them upgrade them 
> instead. If its hard, then its likely someone will chip in to make it less 
> hard.
> 
> Another is devstack in general. the tooling used by devs and that used by ops 
> are so different as to isolate the devs from ops' pain. If they used more 
> opsish tooling, then they would hit the same issues and would be more likely 
> to find solutions that work for both parties.
> 
> A third one is supporting multiple version upgrades in the gate. I rarely 
> have a problem with a cloud has a database one version back. I have seen lots 
> of issues with databases that contain data back when the cloud was 
> instantiated and then upgraded multiple times.
> 
> Another option is trying to unify/detangle the upgrade procedure. upgrading 
> compute kit should be one or two commands if you can live with the defaults. 
> Not weeks of poring through release notes, finding correct orders from pages 
> of text and testing vigorously on test systems.

This sounds like an opportunity for some knowledge sharing. Maybe when
the Operators' Guide makes it into the wiki?

> 
> How about some tool that does the: dump database to somewhere temporary, 
> iterate over all the upgrade job components, and see if it will successfully 
> not corrupt your database. That takes a while to do manually. Ideally it 
> could even upload stacktraces back a bug tracker for attention.
> 
> Thanks,
> Kevin
> ________________________________________
> From: Davanum Srinivas [dava...@gmail.com]
> Sent: Tuesday, November 14, 2017 4:08 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: openstack-oper.
> Subject: Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases
> 
> On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson <m...@not.mn> wrote:
> >
> >
> > On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
> >
> >> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M <kevin....@pnnl.gov> wrote:
> >>> The pressure for #2 comes from the inability to skip upgrades and the 
> >>> fact that upgrades are hugely time consuming still.
> >>>
> >>> If you want to reduce the push for number #2 and help developers get 
> >>> their wish of getting features into users hands sooner, the path to 
> >>> upgrade really needs to be much less painful.
> >>>
> >>
> >> +1000
> >>
> >> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
> >> execute the upgrade. (and we skipped a version)
> >> Scheduling all the relevant internal teams is a monumental task
> >> because we don't have dedicated teams for those projects and they have
> >> other priorities.
> >> Upgrading affects a LOT of our systems, some we don't fully have
> >> control over. And it can takes months to get new deployment on those
> >> systems. (and after, we have to test compatibility, of course)
> >>
> >> So I guess you can understand my frustration when I'm told to upgrade
> >> more often and that skipping versions is discouraged/unsupported.
> >> At the current pace, I'm just falling behind. I *need* to skip
> >> versions to keep up.
> >>
> >> So for our next upgrades, we plan on skipping even more versions if
> >> the database migration allows it. (except for Nova which is a huge
> >> PITA to be honest due to CellsV1)
> >> I just don't see any other ways to keep up otherwise.
> >
> > ?!?!
> >
> > What does it take for this to never happen again? No operator should need 
> > to plan and execute an upgrade for a whole year to upgrade one year's worth 
> > of code development.
> >
> > We don't need new policies, new teams, more releases, fewer releases, or 
> > anything like that. The goal is NOT "let's have an LTS release". The goal 
> > should be "How do we make sure Mattieu and everyone else in the world can 
> > actually deploy and use the software we are writing?"
> >
> > Can we drop the entire LTS discussion for now and focus on "make upgrades 
> > take less than a year" instead? After we solve that, let's come back around 
> > to LTS versions, if needed. I know there's already some work around that. 
> > Let's focus there and not be distracted about the best bureaucracy for not 
> > deleting two-year-old branches.
> >
> >
> > --John
> 
> John,
> 
> So... Any concrete ideas on how to achieve that?
> 
> Thanks,
> Dims
> 
> >
> >
> > /me puts on asbestos pants
> >
> >>
> >> --
> >> Mathieu
> >>
> >> _______________________________________________
> >> OpenStack-operators mailing list
> >> openstack-operat...@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to