Re: [openstack-dev] [TripleO] our update story: can people live with it?
Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I think in most cases transfer cost is worth it to know you're deploying what you tested. Also it is pretty easy to just do this optimization but still be rsyncing the contents of the image. Instead of downloading the whole thing we could have a box expose the mounted image via rsync and then all of the machines can just rsync changes. Also rsync has a batch mode where if you know for sure the end-state of machines you can pre-calculate that rsync and just ship that. Lots of optimization possible that will work fine in your just- update-one-file scenario. But really, how much does downtime cost? How much do 10Gb NICs and switches cost? It's not as simple as just saying buy better hardware (although I do have a vested interest in that approach ;-) - on a compute node the Network and Disk bandwidth is already doing useful work for paying customers. The more overhead you put into that for updates, the more disruptive it becomes. Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On 01/22/2014 12:17 PM, Dan Prince wrote: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? FWIW, I agree that this is going to be considered unacceptable by most people. Hopefully everyone is on the same page with that. It sounds like that's the case so far in this thread, at least... If you have to reboot the compute node, ideally you also have support for live migrating all running VMs on that compute node elsewhere before doing so. That's not something you want to have to do for *every* little change to *every* compute node. Yep, my reading is the same as yours Russell, everyone agreed that there needs to be an update that avoids the reboot where possible (other parts of the thread seem to be focused on how much further the update can be optimized). What's not clear to me is when the plan is to have that support in TripleO - I tried looking for a matching Blueprint to see if it was targeted for Icehouse but can't match it against the five listed. Perhaps Rob or Clint can clarify ? Feels to me that this is a must have before anyone will really be able to use TripleO beyond a PoC for initial deployment. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Day, Phil's message of 2014-01-24 04:39:10 -0800: On 01/22/2014 12:17 PM, Dan Prince wrote: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? FWIW, I agree that this is going to be considered unacceptable by most people. Hopefully everyone is on the same page with that. It sounds like that's the case so far in this thread, at least... If you have to reboot the compute node, ideally you also have support for live migrating all running VMs on that compute node elsewhere before doing so. That's not something you want to have to do for *every* little change to *every* compute node. Yep, my reading is the same as yours Russell, everyone agreed that there needs to be an update that avoids the reboot where possible (other parts of the thread seem to be focused on how much further the update can be optimized). What's not clear to me is when the plan is to have that support in TripleO - I tried looking for a matching Blueprint to see if it was targeted for Icehouse but can't match it against the five listed. Perhaps Rob or Clint can clarify ? Feels to me that this is a must have before anyone will really be able to use TripleO beyond a PoC for initial deployment. Right now we are focused on the hard case, updates requiring reboot. Avoiding the reboot is a bit more than an optimization, but it is something we will get to once we've nailed the harder case of handling a new kernel and reboot gracefully. I for one have a fear that if we start with the easy case, we'll just avoid the hard one, spend less time on it, and thus do it poorly. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Day, Phil's message of 2014-01-24 04:24:11 -0800: Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I think in most cases transfer cost is worth it to know you're deploying what you tested. Also it is pretty easy to just do this optimization but still be rsyncing the contents of the image. Instead of downloading the whole thing we could have a box expose the mounted image via rsync and then all of the machines can just rsync changes. Also rsync has a batch mode where if you know for sure the end-state of machines you can pre-calculate that rsync and just ship that. Lots of optimization possible that will work fine in your just- update-one-file scenario. But really, how much does downtime cost? How much do 10Gb NICs and switches cost? It's not as simple as just saying buy better hardware (although I do have a vested interest in that approach ;-) - on a compute node the Network and Disk bandwidth is already doing useful work for paying customers. The more overhead you put into that for updates, the more disruptive it becomes. Agreed. The question becomes whether you should reserve a portion of your resources for updates or let them push you into over-subscription. Either way, those are business decisions. And once we have a working system and we can say this costs X bitcoins, we can make a clear justification for somebody to spend developer time to push X downward. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On 22/01/14 20:54, Clint Byrum wrote:
I don't understand the aversion to using existing, well-known tools to handle
this?
These tools are of course available to users and nobody is stopping them
from using them. We are optimizing for not needing them. They are there
and we're not going to explode if you use them. You just lose one aspect
of what we're aiming at. I believe that having image based deploys will
be well received as long as it is simple to understand.
A hybrid model (blending 2 and 3, above) here I think would work best where
TripleO lays down a baseline image and the cloud operator would employ an
well-known
and support configuration tool for any small diffs.
These tools are popular because they control entropy and make it at
least more likely that what you tested ends up on the boxes.
A read-only root partition is a much stronger control on entropy.
The operator would then be empowered to make the call for any major upgrades
that
would adversely impact the infrastructure (and ultimately the users/apps).
He/She
could say, this is a major release, let's deploy the image.
Something logically like this, seems reasonable:
if (system_change 10%) {
use TripleO;
} else {
use Existing_Config_Management;
}
I think we can make deploying minor updates minimally invasive.
We've kept it simple enough, this should be a fairly straight forward
optimization cycle. And the win there is that we also improve things
for the 11% change.
Hi Clint,
For deploying minimally-invasive minor updates, the idea, if I've
understood it correctly, would be to deploy a tarball which replaced
selected files on the (usually read-only) root filesystem. That would
allow for selective restarting of only the services which are directly
affected. The alternative, pushing out a complete root filesystem image,
would necessitate the same amount of disruption in all cases.
There are a handful of costs with that approach which concern me: It
simplifies the deployment itself, but increases the complexity of
preparing the deployment. The administrator is going to have to identify
the services which need to be restarted, based on the particular set of
libraries which are touched in their partial update, and put together
the service restart scripts accordingly.
We're also making the administrator responsible for managing the
sequence in which incremental updates are deployed. Since each
incremetal update will re-write a particular set of files, any machine
which gets updates 1,2, 3, there's an oversight, and then update 5 is
deployed would end up in an odd state, which would require additional
tooling to detect. Package based updates, with versioning and dependency
tracking on each package, mitigate that risk.
Then there's the relationship between the state of running machines,
with applied partial updates, and the images which are put onto new
machines by Ironic. We would need to apply the partial updates to the
images which Ironic writes, or to have the tooling to ensure that newly
deployed machines immediately apply the set of applicable partial
updates, in sequence.
Solving these issues feels like it'll require quite a lot of additional
tooling.
Angus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Hi
Not a tarball. The node would notice from Heat metadata that it should update
to a new image and would fetch that image and sync the contents to its /. This
would leave it bit for bit identical to a fresh deployment of the new image, at
least on disk. The running state would differ and that still requires some
design and implementation to figure out.
Cheers,
--
Chris Jones
On 23 Jan 2014, at 12:57, Angus Thomas atho...@redhat.com wrote:
On 22/01/14 20:54, Clint Byrum wrote:
I don't understand the aversion to using existing, well-known tools to
handle this?
These tools are of course available to users and nobody is stopping them
from using them. We are optimizing for not needing them. They are there
and we're not going to explode if you use them. You just lose one aspect
of what we're aiming at. I believe that having image based deploys will
be well received as long as it is simple to understand.
A hybrid model (blending 2 and 3, above) here I think would work best where
TripleO lays down a baseline image and the cloud operator would employ an
well-known
and support configuration tool for any small diffs.
These tools are popular because they control entropy and make it at
least more likely that what you tested ends up on the boxes.
A read-only root partition is a much stronger control on entropy.
The operator would then be empowered to make the call for any major
upgrades that
would adversely impact the infrastructure (and ultimately the users/apps).
He/She
could say, this is a major release, let's deploy the image.
Something logically like this, seems reasonable:
if (system_change 10%) {
use TripleO;
} else {
use Existing_Config_Management;
}
I think we can make deploying minor updates minimally invasive.
We've kept it simple enough, this should be a fairly straight forward
optimization cycle. And the win there is that we also improve things
for the 11% change.
Hi Clint,
For deploying minimally-invasive minor updates, the idea, if I've understood
it correctly, would be to deploy a tarball which replaced selected files on
the (usually read-only) root filesystem. That would allow for selective
restarting of only the services which are directly affected. The alternative,
pushing out a complete root filesystem image, would necessitate the same
amount of disruption in all cases.
There are a handful of costs with that approach which concern me: It
simplifies the deployment itself, but increases the complexity of preparing
the deployment. The administrator is going to have to identify the services
which need to be restarted, based on the particular set of libraries which
are touched in their partial update, and put together the service restart
scripts accordingly.
We're also making the administrator responsible for managing the sequence in
which incremental updates are deployed. Since each incremetal update will
re-write a particular set of files, any machine which gets updates 1,2, 3,
there's an oversight, and then update 5 is deployed would end up in an odd
state, which would require additional tooling to detect. Package based
updates, with versioning and dependency tracking on each package, mitigate
that risk.
Then there's the relationship between the state of running machines, with
applied partial updates, and the images which are put onto new machines by
Ironic. We would need to apply the partial updates to the images which Ironic
writes, or to have the tooling to ensure that newly deployed machines
immediately apply the set of applicable partial updates, in sequence.
Solving these issues feels like it'll require quite a lot of additional
tooling.
Angus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Angus Thomas's message of 2014-01-23 04:57:20 -0800:
On 22/01/14 20:54, Clint Byrum wrote:
I don't understand the aversion to using existing, well-known tools to
handle this?
These tools are of course available to users and nobody is stopping them
from using them. We are optimizing for not needing them. They are there
and we're not going to explode if you use them. You just lose one aspect
of what we're aiming at. I believe that having image based deploys will
be well received as long as it is simple to understand.
A hybrid model (blending 2 and 3, above) here I think would work best
where
TripleO lays down a baseline image and the cloud operator would employ an
well-known
and support configuration tool for any small diffs.
These tools are popular because they control entropy and make it at
least more likely that what you tested ends up on the boxes.
A read-only root partition is a much stronger control on entropy.
The operator would then be empowered to make the call for any major
upgrades that
would adversely impact the infrastructure (and ultimately the
users/apps). He/She
could say, this is a major release, let's deploy the image.
Something logically like this, seems reasonable:
if (system_change 10%) {
use TripleO;
} else {
use Existing_Config_Management;
}
I think we can make deploying minor updates minimally invasive.
We've kept it simple enough, this should be a fairly straight forward
optimization cycle. And the win there is that we also improve things
for the 11% change.
Hi Clint,
For deploying minimally-invasive minor updates, the idea, if I've
understood it correctly, would be to deploy a tarball which replaced
selected files on the (usually read-only) root filesystem. That would
allow for selective restarting of only the services which are directly
affected. The alternative, pushing out a complete root filesystem image,
would necessitate the same amount of disruption in all cases.
FTR, I have not said that chopping updates into things like tars of only
the difference between old and new image is a bad idea. Its a great idea
to avoid transfer costs. It just doesn't solve the system state problem,
nor do package updates.
How exactly would we determine which running things have imported
/usr/lib/python2.7/wsgiref/__init__.py ? Assuming we have packages only,
are we going to walk through its reverse dependencies and find any init
scripts/systemd unit files, and restart them? Do we just pray there
are no transitive dependencies or do we get really broad and do a first
level deep just to be sure? This is fine, but it is complexity, and for
what gain? If we make it not disruptive to restart any given service,
we don't have to be complex, and we can spend our energy on other
things.
I don't think a tarball or packages make it any easier to know that we
have deployed the software and ensured there are no stale copies of code
or configuration running. Any optimizations that can be done will need
to be able to resolve system state with the whole software collection or
will have to carry the risk of stale software continuing to run.
I'm suggesting we focus our efforts on straight forward, easy to
understand things: HA and tests. Those things will have a much greater
benefit on the user than trying to disect diffs and figure out what
things to restart, IMO.
There are a handful of costs with that approach which concern me: It
simplifies the deployment itself, but increases the complexity of
preparing the deployment. The administrator is going to have to identify
the services which need to be restarted, based on the particular set of
libraries which are touched in their partial update, and put together
the service restart scripts accordingly.
The cloud is for automation. Said administrator should be writing tests
that ensure they're running the software and configuration that they
think they are. We should also be writing tests for them and shipping
said tests with each and every update, as that is something generally
consumable that will have a massive positive impact on trust in OpenStack.
We're also making the administrator responsible for managing the
sequence in which incremental updates are deployed. Since each
incremetal update will re-write a particular set of files, any machine
which gets updates 1,2, 3, there's an oversight, and then update 5 is
deployed would end up in an odd state, which would require additional
tooling to detect. Package based updates, with versioning and dependency
tracking on each package, mitigate that risk.
Yes, a carefully constructed set of packages can know all of the
version combinations and they have asserted their dependencies so
post configurations get run at the right times. I'm well aware of what
Debian and RPM packaging are capable of. I'm also well aware that it
is
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Would docker work for this?
Assume every service gets its own docker container. A deployed node is then a
docker base image with a set of service containers. Updating an image could be:
Check if base part of image updated (kernel, docker). if so, full redeploy the
node.
Sync each container image, restart container only if different?
Maybe the same trick could be done with python virtualenvs instead...
Kevin
From: Chris Jones [c...@tenshu.net]
Sent: Thursday, January 23, 2014 7:17 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [TripleO] our update story: can people live
with it?
Hi
Not a tarball. The node would notice from Heat metadata that it should update
to a new image and would fetch that image and sync the contents to its /. This
would leave it bit for bit identical to a fresh deployment of the new image, at
least on disk. The running state would differ and that still requires some
design and implementation to figure out.
Cheers,
--
Chris Jones
On 23 Jan 2014, at 12:57, Angus Thomas atho...@redhat.com wrote:
On 22/01/14 20:54, Clint Byrum wrote:
I don't understand the aversion to using existing, well-known tools to
handle this?
These tools are of course available to users and nobody is stopping them
from using them. We are optimizing for not needing them. They are there
and we're not going to explode if you use them. You just lose one aspect
of what we're aiming at. I believe that having image based deploys will
be well received as long as it is simple to understand.
A hybrid model (blending 2 and 3, above) here I think would work best where
TripleO lays down a baseline image and the cloud operator would employ an
well-known
and support configuration tool for any small diffs.
These tools are popular because they control entropy and make it at
least more likely that what you tested ends up on the boxes.
A read-only root partition is a much stronger control on entropy.
The operator would then be empowered to make the call for any major
upgrades that
would adversely impact the infrastructure (and ultimately the users/apps).
He/She
could say, this is a major release, let's deploy the image.
Something logically like this, seems reasonable:
if (system_change 10%) {
use TripleO;
} else {
use Existing_Config_Management;
}
I think we can make deploying minor updates minimally invasive.
We've kept it simple enough, this should be a fairly straight forward
optimization cycle. And the win there is that we also improve things
for the 11% change.
Hi Clint,
For deploying minimally-invasive minor updates, the idea, if I've understood
it correctly, would be to deploy a tarball which replaced selected files on
the (usually read-only) root filesystem. That would allow for selective
restarting of only the services which are directly affected. The alternative,
pushing out a complete root filesystem image, would necessitate the same
amount of disruption in all cases.
There are a handful of costs with that approach which concern me: It
simplifies the deployment itself, but increases the complexity of preparing
the deployment. The administrator is going to have to identify the services
which need to be restarted, based on the particular set of libraries which
are touched in their partial update, and put together the service restart
scripts accordingly.
We're also making the administrator responsible for managing the sequence in
which incremental updates are deployed. Since each incremetal update will
re-write a particular set of files, any machine which gets updates 1,2, 3,
there's an oversight, and then update 5 is deployed would end up in an odd
state, which would require additional tooling to detect. Package based
updates, with versioning and dependency tracking on each package, mitigate
that risk.
Then there's the relationship between the state of running machines, with
applied partial updates, and the images which are put onto new machines by
Ironic. We would need to apply the partial updates to the images which Ironic
writes, or to have the tooling to ensure that newly deployed machines
immediately apply the set of applicable partial updates, in sequence.
Solving these issues feels like it'll require quite a lot of additional
tooling.
Angus
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On 23 January 2014 09:27, Keith Basil kba...@redhat.com wrote:
3) Deploy specific application level updates via packages or tarballs.
(only selected applications/packages get deployed)
++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD
environments, so this level of optimization will be frequently used.
And, as I've said before, optimizing for frequently-used scenarios is
worth spending the time on. Optimizing for infrequently-occurring
things... not so much. :)
I don't understand the aversion to using existing, well-known tools to handle
this?
If you're referring to operating system packages, I laid out the
overall case point by point in this talk at LCA -
http://mirror.linux.org.au/linux.conf.au/2014/Friday/111-Diskimage-builder_deep_dive_into_a_machine_compiler_-_Robert_Collins.mp4
- I'm not sure why the video is not linked in the programme
(https://lca2014.linux.org.au/programme/schedule/friday) yet.
A hybrid model (blending 2 and 3, above) here I think would work best where
TripleO lays down a baseline image and the cloud operator would employ an
well-known
and support configuration tool for any small diffs.
In doing that they would sacrifice all testing of the 'small diffs'.
Which is a great way to end up running a combination that doesn't
work.
The operator would then be empowered to make the call for any major upgrades
that
would adversely impact the infrastructure (and ultimately the users/apps).
He/She
could say, this is a major release, let's deploy the image.
Something logically like this, seems reasonable:
if (system_change 10%) {
use TripleO;
} else {
use Existing_Config_Management;
}
It seems disruptive to force compute (or other) nodes to reboot on trivial
updates.
We won't be doing that :), though it is much simpler to reason about,
so I believe some users will want it.
If we are to get further enterprise adoption of OpenStack, this seems like a
huge
blocker. This will be a very hard sell to get traditional IT folk to buy into
this approach:
Wait, *every* time I have to make a system change, I need to reboot
my
entire cloud?
Elastic cloud concepts are already trying enough for the enterprise.
Sure, if you frame it like that it would be a scary and unreasonable
concept - they would quite reasonably ask 'what about all my running
VMs' and other such questions :). Also it would be a massive
exaggeration as we *already* *only* do reboots on software changes,
not on last-mile configuration changes.
With rolling upgrades there would be no end user visible downtime in
the *un*optimised case. The optimised case merely deploys faster.
Finally, remember that the undercloud machines have small (~170K)
numbers of files - which takes 28 seconds to stat cold - so rsyncing a
single changed file should be ~30seconds. Thats comparable (if not
better :)) than the time to run yum or apt.
-Rob
--
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] our update story: can people live with it?
I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. I would call your e-mail a documentation/roadmap bug. This plan may have been recorded somewhere, but for me it has just always been in my head as the end goal (thanks to Robert Collins for drilling the hole and pouring it in there btw ;). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
- Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) I would call your e-mail a documentation/roadmap bug. Fair enough. Thanks for the info. This plan may have been recorded somewhere, but for me it has just always been in my head as the end goal (thanks to Robert Collins for drilling the hole and pouring it in there btw ;). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Another tricky bit left is how to handle service restarts as needed? Thanks, Kevin From: Dan Prince [dpri...@redhat.com] Sent: Wednesday, January 22, 2014 10:15 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) I would call your e-mail a documentation/roadmap bug. Fair enough. Thanks for the info. This plan may have been recorded somewhere, but for me it has just always been in my head as the end goal (thanks to Robert Collins for drilling the hole and pouring it in there btw ;). ___ OpenStack-dev mailing
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On 01/22/2014 12:17 PM, Dan Prince wrote: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? FWIW, I agree that this is going to be considered unacceptable by most people. Hopefully everyone is on the same page with that. It sounds like that's the case so far in this thread, at least... If you have to reboot the compute node, ideally you also have support for live migrating all running VMs on that compute node elsewhere before doing so. That's not something you want to have to do for *every* little change to *every* compute node. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Dan Prince's message of 2014-01-22 10:15:20 -0800: - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. I prefer choosing what to update at image build time. This is the time where it is most clear how, from a developer and deployer standpoint, to influence the update. I can diff images, I can freeze mirrors.. etc. etc, all decoupled from anybody else and from production or test cycles. That said, I do think it would be good for deployers to be able have a way to control when reboots are and aren't allowed. That seems like policy, which may be best handled in Nova.. so we can have a user that can do updates to Heat Metadata/stacks, but not rebuilds in Nova. I have no idea of Heat's trust model will allow us to have such separation though. Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I think in most cases transfer cost is worth it to know you're deploying what you tested. Also it is pretty easy to just do this optimization but still be rsyncing the contents of the image. Instead of downloading the whole thing we could have a box expose the mounted image via rsync and then all of the machines can just rsync changes. Also rsync has a batch mode where if you know for sure the end-state of machines you can pre-calculate that rsync and just ship that. Lots of optimization possible that will work fine in your just-update-one-file scenario. But really, how much does downtime cost? How much do 10Gb NICs and switches cost? I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote: - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. ++ Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. Right. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD environments, so this level of optimization will be frequently used. And, as I've said before, optimizing for frequently-used scenarios is worth spending the time on. Optimizing for infrequently-occurring things... not so much. :) Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Agreed, it is tricky if we try to only restart what we've changed. OR, just restart everything. We can make endpoints HA and use rolling updates to avoid spurious faults. There are complex ways to handle things even smoother.. but I go back to What does complexity cost? Excerpts from Fox, Kevin M's message of 2014-01-22 10:32:02 -0800: Another tricky bit left is how to handle service restarts as needed? Thanks, Kevin From: Dan Prince [dpri...@redhat.com] Sent: Wednesday, January 22, 2014 10:15 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Jay Pipes's message of 2014-01-22 10:53:14 -0800: On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote: - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD environments, so this level of optimization will be frequently used. And, as I've said before, optimizing for frequently-used scenarios is worth spending the time on. Optimizing for infrequently-occurring things... not so much. :) I do understand that little tweaks are more common than whole software updates. I also think that little tweaks must be tested just like big ones. So I would argue that it is more important to optimize for trusting that what you tested is what is in production, and then to address any issues if that work-flow needs optimization. A system that leaves operators afraid to do a big update because it will trigger the bad path is a system that doesn't handle big updates well. Ideally we'd optimize all 3 in all of the obvious ways before determining that the one file update just isn't fast enough. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
I think most of the time taken to reboot is spent in bringing down/up the services though, so I'm not sure what it really buys you if you do it all. It may let you skip the crazy long bootup time on enterprise hardware, but that could be worked around with kexec on the full reboot method too. Thanks, Kevin From: Clint Byrum [cl...@fewbar.com] Sent: Wednesday, January 22, 2014 10:55 AM To: openstack-dev Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Agreed, it is tricky if we try to only restart what we've changed. OR, just restart everything. We can make endpoints HA and use rolling updates to avoid spurious faults. There are complex ways to handle things even smoother.. but I go back to What does complexity cost? Excerpts from Fox, Kevin M's message of 2014-01-22 10:32:02 -0800: Another tricky bit left is how to handle service restarts as needed? Thanks, Kevin From: Dan Prince [dpri...@redhat.com] Sent: Wednesday, January 22, 2014 10:15 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On Jan 22, 2014, at 1:53 PM, Jay Pipes wrote: On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote: - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. ++ Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. Right. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD environments, so this level of optimization will be frequently used. And, as I've said before, optimizing for frequently-used scenarios is worth spending the time on. Optimizing for infrequently-occurring things... not so much. :) I don't understand the aversion to using existing, well-known tools to handle this? A hybrid model (blending 2 and 3, above) here I think would work best where TripleO lays down a baseline image and the cloud operator would employ an well-known and support configuration tool
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Fox, Kevin M's message of 2014-01-22 12:19:56 -0800: I think most of the time taken to reboot is spent in bringing down/up the services though, so I'm not sure what it really buys you if you do it all. It may let you skip the crazy long bootup time on enterprise hardware, but that could be worked around with kexec on the full reboot method too. If we could get kexec reliable.. but I have no evidence that it is anything but a complete flake. What it saves you is losing running processes that you don't end up killing, which is expensive on many types of services.. Nova Compute being a notable example. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
On Wed, 2014-01-22 at 12:12 -0800, Clint Byrum wrote: Excerpts from Jay Pipes's message of 2014-01-22 10:53:14 -0800: On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote: - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people livewith it? Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD environments, so this level of optimization will be frequently used. And, as I've said before, optimizing for frequently-used scenarios is worth spending the time on. Optimizing for infrequently-occurring things... not so much. :) I do understand that little tweaks are more common than whole software updates. I also think that little tweaks must be tested just like big ones. So I would argue that it is more important to optimize for trusting that what you tested is what is in production, and then to address any issues if that work-flow needs optimization. A system that leaves operators afraid to do a big update because it will trigger the bad path is a system that doesn't handle big updates well. Ideally we'd optimize all 3 in all of the obvious ways before determining that the one file update just isn't fast enough. Well said. No disagreement from me. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Excerpts from Keith Basil's message of 2014-01-22 12:27:50 -0800: On Jan 22, 2014, at 1:53 PM, Jay Pipes wrote: On Wed, 2014-01-22 at 13:15 -0500, Dan Prince wrote: - Original Message - From: Clint Byrum cl...@fewbar.com To: openstack-dev openstack-dev@lists.openstack.org Sent: Wednesday, January 22, 2014 12:45:45 PM Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Dan Prince's message of 2014-01-22 09:17:24 -0800: I've been thinking a bit more about how TripleO updates are developing specifically with regards to compute nodes. What is commonly called the update story I think. As I understand it we expect people to actually have to reboot a compute node in the cluster in order to deploy an update. This really worries me because it seems like way overkill for such a simple operation. Lets say all I need to deploy is a simple change to Nova's libvirt driver. And I need to deploy it to *all* my compute instances. Do we really expect people to actually have to reboot every single compute node in their cluster for such a thing. And then do this again and again for each update they deploy? Agreed, if we make everybody reboot to push out a patch to libvirt, we have failed. And thus far, we are failing to do that, but with good reason. Right at this very moment, we are leaning on 'rebuild' in Nova, which reboots the instance. But this is so that we handle the hardest thing well first (rebooting to have a new kernel). For small updates we need to decouple things a bit more. There is a notion of the image ID in Nova, versus the image ID that is actually running. Right now we update it with a nova rebuild command only. But ideally we would give operators a tool to optimize and avoid the reboot when it is appropriate. The heuristic should be as simple as comparing kernels. When we get to implementing such a thing I might prefer it not to be auto-magic. I can see a case where I want the new image but maybe not the new kernel. Perhaps this should be addressed when building the image (by using the older kernel)... but still. I could see a case for explicitly not wanting to reboot here as well. ++ Once we have determined that a new image does not need a reboot, we can just change the ID in Metadata, and an os-refresh-config script will do something like this: if [ $(cat /etc/image_id) != $(os-apply-config --key image_id) ] ; then; download_new_image mount_image /tmp/new_image mount / -o remount,rw # Assuming we've achieved ro root rsync --one-file-system -a /tmp/new_image/ / mount / -o remount,ro # ditto fi No reboot required. This would run early in configure.d, so that any pre-configure.d scripts will have run to quiesce services that can't handle having their binaries removed out from under them (read: non-Unix services). Then configure.d runs as usual, configures things, restarts services, and we are now running the new image. Cool. I like this a good bit better as it avoids the reboot. Still, this is a rather large amount of data to copy around if I'm only changing a single file in Nova. Right. I understand the whole read only images thing plays into this too... but I'm wondering if there is a middle ground where things might work better. Perhaps we have a mechanism where we can tar up individual venvs from /opt/stack/ or perhaps also this is an area where real OpenStack packages could shine. It seems like we could certainly come up with some simple mechanisms to deploy these sorts of changes with Heat such that compute host reboot can be avoided for each new deploy. Given the scenario above, that would be a further optimization. I don't think it makes sense to specialize for venvs or openstack services though, so just ensure the root filesystems match seems like a workable, highly efficient system. Note that we've talked about having highly efficient ways to widely distribute the new images as well. Yes. Optimization! In the big scheme of things I could see 3 approaches being useful: 1) Deploy a full image and reboot if you have a kernel update. (entire image is copied) 2) Deploy a full image if you change a bunch of things and/or you prefer to do that. (entire image is copied) 3) Deploy specific application level updates via packages or tarballs. (only selected applications/packages get deployed) ++. FWIW, #3 happens a heck of a lot more often than #1 or #2 in CD environments, so this level of optimization will be frequently used. And, as I've said before, optimizing for frequently-used scenarios is worth spending the time on. Optimizing for infrequently-occurring things... not so much. :) I don't understand the aversion to using existing, well-known tools to handle this? These tools
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Maybe I misunderstand, but I thought: kexec - lets you boot a new kernel/initrd starting at the point a boot loader would skipping the bios init. All previous running processes are not running in the new boot just like a normal reboot. CRIU - Lets you snapshot/restart running processes. While you could use both together to upgrades kernel while leaving all the processes running after the reboot, I don't think that's very tested at the moment. checkpointing the system memory is not without cost too. Restarting the services may be faster. I think we're pretty far off in a tangent though. My main point was, if you can't selectively restart services as needed, I'm not sure how useful patching the image really is over a full reboot. It should take on the same order of magnitude service unavailability I think. Thanks, Kevin From: Clint Byrum [cl...@fewbar.com] Sent: Wednesday, January 22, 2014 12:36 PM To: openstack-dev Subject: Re: [openstack-dev] [TripleO] our update story: can people live with it? Excerpts from Fox, Kevin M's message of 2014-01-22 12:19:56 -0800: I think most of the time taken to reboot is spent in bringing down/up the services though, so I'm not sure what it really buys you if you do it all. It may let you skip the crazy long bootup time on enterprise hardware, but that could be worked around with kexec on the full reboot method too. If we could get kexec reliable.. but I have no evidence that it is anything but a complete flake. What it saves you is losing running processes that you don't end up killing, which is expensive on many types of services.. Nova Compute being a notable example. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] our update story: can people live with it?
Hi On 22 January 2014 21:33, Fox, Kevin M kevin@pnnl.gov wrote: I think we're pretty far off in a tangent though. My main point was, if you can't selectively restart services as needed, I'm not sure how useful patching the image really is over a full reboot. It should take on the same order of magnitude service unavailability I think. The in-place upgrade (currently known as takeovernode) is not yet well designed and while there is a script in tripleo-incubator called takeovernode, nobody is likely to resume working on it until a little later in our roadmap. The crude hack we have atm does no detection of services that need to be restarted, but that is not intended to suggest that we don't care about such a feature :) I think Clint has covered pretty much all the bases here, but I would re-iterate that in no way do we think the kind of upgrade we're working on at the moment (i.e. a nova rebuild driven instance reboot) is the only one that should exist. We know that in-place upgrades need to happen for tripleo's full story to be taken seriously, and we will get to it. If folk have suggestions for behaviours/techniques/tools, those would be great to capture, probably in https://etherpad.openstack.org/p/tripleo-image-updates . http://manpages.ubuntu.com/manpages/oneiric/man1/checkrestart.1.html is one such tool that we turned up in earlier research about how to detect services that need to be restarted after an upgrade. It's not a complete solution on its own, but it gets us some of the way. (Also, just because we favour low-entropy golden images for all software changes, doesn't mean that any given user can't choose to roll out an upgrade to some piece(s) of software via any other mechanism they choose, if that is what they feel is right for their operation. A combination of the two strategies is entirely possible). -- Cheers, Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
