Just a thought, cause I have known/do know what Mathieu is talking about
and find the disconnect still oddly weird. Why aren't developer people
from other companies coming into to where Mathieu works (or where I
work) and seeing how it really works down on the ground here.
I mean if we still have this weird disconnect; why aren't we like
starting some kind of temp. assignments from developers that wish to
learn into the actual companies that are struggling; call it a working
vacation or something if that helps your management buy into it.
After all if its a community, and we should be trying to break down
walls as much as we can...
Just a thought...
-Josh
Mathieu Gagné wrote:
Some clarifications below.
On Wed, Nov 15, 2017 at 4:52 AM, Bogdan Dobrelya<[email protected]> wrote:
Thank you Mathieu for the insights!
To add details to what happened:
* Upgrade was never made a #1 priority. It was a one man show for far
too long. (myself)
I suppose that confirms that upgrades is very nice to have in production
deployments, eventually, maybe... (please read below to continue)
* I also happen to manage and work on other priorities.
* Lot of work made to prepare for multiple versions support in our
deployment tools. (we use Puppet)
* Lot of work in the packaging area to speedup packaging. (we are
still using deb packages but with virtualenv to stay Puppet
compatible)
* We need to forward-port private patches which upstream won't accept
and/or are private business logic.
... yet long time maintaining and landing fixes is the ops' *reality* and
pain #1. And upgrades are only pain #2. LTS can not directly help with #2,
but only indirectly, if the vendors' downstream teams could better cooperate
with #1 and have more time and resources to dedicate for #2, upgrades
stories for shipped products and distros.
We do not have a vendor. (anymore, if you consider Ubuntu
cloud-archive as a vendor)
We package and deploy ourselves.
Let's please to not lower the real value of LTS branches and not substitute
#1 with #2. This topic is not about bureaucracy and policies, it is about
how could the community help vendors to cooperate over maintaining of
commodity things, with as less bureaucracy as possible, to ease the
operators pains in the end.
* Our developer teams didn't have enough free cycles to work right
away on the upgrade. (this means delays)
* We need to test compatibility with 3rd party systems which takes
some time. (and make them compatible)
This confirms perhaps why it is vital to only run 3rd party CI jobs for LTS
branches?
For us, 3rd party systems are internal systems outside our control or
realm of influence.
They are often in-house systems that the outside world would care very
little about.
* We need to update systems ever which we don't have full control.
This means serious delays when it comes to deployment.
* We need to test features/stability during some time in our dev
environment.
* We need to test features/stability during some time in our
staging/pre-prod environment.
* We need to announce and inform our users at least 2 weeks in advance
before performing an upgrade.
* We choose to upgrade one service at a time (in all regions) to avoid
a huge big bang upgrade. (this means more maintenance windows to plan
and you can't stack them too much)
* We need to swiftly respond to bug discovered by our users. This
means change of priorities and delay in other service upgrades.
* We will soon need to upgrade operating systems to support latest
OpenStack versions. (this means we have to stop OpenStack upgrades
until all nodes are upgraded)
It seems that the answer to the question sounded, "Why upgrades are so
painful and take so much time for ops?" is "as upgrades are not the
priority. Long Time Support and maintenance are".
The cost of performing an upgrading is both time and resources
consuming which are both limited.
And you need to sync the world around you to make it happen. It's not
a one man decision/task.
When you remove all the external factors, dependencies, politics,
etc., upgrading can take an afternoon from A to Z from some projects.
We do have an internal cloud for our developers that lives in a
vacuum. Let me tell you that it's not very easy upgrade it. We are
talking about hours/days, not years.
So if I can only afford to upgrade once per year, what are my options?
--
Mathieu
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
