Re: [Openstack-operators] [openstack-dev] [nova] Supporting force live-migrate and force evacuate with nested allocations
>> I disagree on this. I'd rather just do a simple check for >1 >> provider in the allocations on the source and if True, fail hard. >> >> The reverse (going from a non-nested source to a nested destination) >> will hard fail anyway on the destination because the POST >> /allocations won't work due to capacity exceeded (or failure to have >> any inventory at all for certain resource classes on the >> destination's root compute node). > > I agree with Jay here. If we know the source has allocations on >1 > provider, just fail fast, why even walk the tree and try to claim > those against the destination - the nested providers aren't going to > be the same UUIDs on the destination, *and* trying to squash all of > the source nested allocations into the single destination root > provider and hope it works is super hacky and I don't think we should > attempt that. Just fail if being forced and nested allocations exist > on the source. Same, yeah. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] Open letter/request to TC candidates (and existing elected officials)
> I'm just a bit worried to limit that role to the elected TC members. If > we say "it's the role of the TC to do cross-project PM in OpenStack" > then we artificially limit the number of people who would sign up to do > that kind of work. You mention Ildiko and Lance: they did that line of > work without being elected. Why would saying that we _expect_ the TC members to do that work limit such activities only to those that are on the TC? I would expect the TC to take on the less-fun or often-neglected efforts that we all know are needed but don't have an obvious champion or sponsor. I think we expect some amount of widely-focused technical or project leadership from TC members, and certainly that expectation doesn't prevent others from leading efforts (even in the areas of proposing TC resolutions, etc) right? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [upgrade] request for pre-upgrade check for db purge
> How do people feel about this? It seems pretty straight-forward to > me. If people are generally in favor of this, then the question is > what would be sane defaults - or should we not assume a default and > force operators to opt into this? I dunno, adding something to nova.conf that is only used for nova-status like that seems kinda weird to me. It's just a warning/informational sort of thing so it just doesn't seem worth the complication to me. Moving it to an age thing set at one year seems okay, and better than making the absolute limit more configurable. Any reason why this wouldn't just be a command line flag to status if people want it to behave in a specific way from a specific tool? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova][placement][upgrade][qa] Some upgrade-specific news on extraction
> The other obvious thing is the database. The placement repo code as-is > today still has the check for whether or not it should use the > placement database but falls back to using the nova_api database > [5]. So technically you could point the extracted placement at the > same nova_api database and it should work. However, at some point > deployers will clearly need to copy the placement-related tables out > of the nova_api DB to a new placement DB and make sure the > 'migrate_version' table is dropped so that placement DB schema > versions can reset to 1. I think it's wrong to act like placement and nova-api schemas are the same. One is a clone of the other at a point in time, and technically it will work today. However the placement db sync tool won't do the right thing, and I think we run the major risk of operators not fully grokking what is going on here, seeing that pointing placement at nova-api "works" and move on. Later, when we add the next placement db migration (which could technically happen in stein), they will either screw their nova-api schema, or mess up their versioning, or be unable to apply the placement change. > With respect to grenade and making this work in our own upgrade CI > testing, we have I think two options (which might not be mutually > exclusive): > > 1. Make placement support using nova.conf if placement.conf isn't > found for Stein with lots of big warnings that it's going away in > T. Then Rocky nova.conf with the nova_api database configuration just > continues to work for placement in Stein. I don't think we then have > any grenade changes to make, at least in Stein for upgrading *from* > Rocky. Assuming fresh devstack installs in Stein use placement.conf > and a placement-specific database, then upgrades from Stein to T > should also be OK with respect to grenade, but likely punts the > cut-over issue for all other deployment projects (because we don't CI > with grenade doing Rocky->Stein->T, or FFU in other words). As I have said above and in the review, I really think this is the wrong approach. At the current point of time, the placement schema is a clone of the nova-api schema, and technically they will work. At the first point that placement evolves its schema, that will no longer be a workable solution, unless we also evolve nova-api's database in lockstep. > 2. If placement doesn't support nova.conf in Stein, then grenade will > require an (exceptional) [6] from-rocky upgrade script which will (a) > write out placement.conf fresh and (b) run a DB migration script, > likely housed in the placement repo, to create the placement database > and copy the placement-specific tables out of the nova_api > database. Any script like this is likely needed regardless of what we > do in grenade because deployers will need to eventually do this once > placement would drop support for using nova.conf (if we went with > option 1). Yep, and I'm asserting that we should write that script, make grenade do that step, and confirm that it works. I think operators should do that step during the stein upgrade because that's where the fork/split of history and schema is happening. I'll volunteer to do the grenade side at least. Maybe it would help to call out specifically that, IMHO, this can not and should not follow the typical config deprecation process. It's not a simple case of just making sure we "find" the nova-api database in the various configs. The problem is that _after_ the split, they are _not_ the same thing and should not be considered as the same. Thus, I think to avoid major disaster and major time sink for operators later, we need to impose the minor effort now to make sure that they don't take the process of deploying a new service lightly. Jay's original relatively small concern was that deploying a new placement service and failing to properly configure it would result in a placement running with the default, empty, sqlite database. That's a valid concern, and I think all we need to do is make sure we fail in that case, explaining the situation. We just had a hangout on the topic and I think we've come around to the consensus that just removing the default-to-empty-sqlite behavior is the right thing to do. Placement won't magically find nova.conf if it exists and jump into its database, and it also won't do the silly thing of starting up with an empty database if the very important config step is missed in the process of deploying placement itself. Operators will have to deploy the new package and do the database surgery (which we will provide instructions and a script for) as part of that process, but there's really no other sane alternative without changing the current agreed-to plan regarding the split. Is everyone okay with the above summary of the outcome? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org
Re: [Openstack-operators] [openstack-dev] [nova] [placement] extraction (technical) update
> I think there was a period in time where the nova_api database was created > where entires would try to get pulled out from the original nova database and > then checking nova_api if it doesn't exist afterwards (or vice versa). One > of the cases that this was done to deal with was for things like instance > types > or flavours. > > I don't know the exact details but I know that older instance types exist in > the nova db and the newer ones are sitting in nova_api. Something along > those lines? Yep, we've moved entire databases before in nova with minimal disruption to the users. Not just flavors, but several pieces of data came out of the "main" database and into the api database transparently. It's doable, but with placement being split to a separate project/repo/whatever, there's not really any option for being graceful about it in this case. > At this point, I'm thinking turn off placement, setup the new one, do > the migration > of the placement-specific tables (this can be a straightforward documented > task > OR it would be awesome if it was a placement command (something along > the lines of `placement-manage db import_from_nova`) which would import all > the right things > > The idea of having a command would be *extremely* useful for deployment tools > in automating the process and it also allows the placement team to selectively > decide what they want to onboard? Well, it's pretty cut-and-dried as all the tables in nova-api are either for nova or placement, so there's not much confusion about what belongs. I'm not sure that doing this import in python is really the most efficient way. I agree a placement-manage command would be ideal from an "easy button" point of view, but I think a couple lines of bash that call mysqldump are likely to vastly outperform us doing it natively in python. We could script exec()s of those commands from python, but.. I think I'd rather just see that as a shell script that people can easily alter/test on their own. Just curious, but in your case would the service catalog entry change at all? If you stand up the new placement in the exact same spot, it shouldn't, but I imagine some people will have the catalog entry change slightly (even if just because of a VIP or port change). Am I remembering correctly that the catalog can get cached in various places such that much of nova would need a restart to notice? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration
> - The VMs to be migrated are not generally not expensive > configurations, just hardware lifecycles where boxes go out of > warranty or computer centre rack/cooling needs re-organising. For > CERN, this is a 6-12 month frequency of ~10,000 VMs per year (with a > ~30% pet share) > - We make a cell from identical hardware at a single location, this > greatly simplifies working out hardware issues, provisioning and > management > - Some cases can be handled with the 'please delete and > re-create'. Many other cases need much user support/downtime (and > require significant effort or risk delaying retirements to get > agreement) Yep, this is the "organizational use case" of cells I refer to. I assume that if one aisle (cell) is being replaced, it makes sense to stand up the new one as its own cell, migrate the pets from one to the other and then decommission the old one. Being only an aisle away, it's reasonable to think that *this* situation might not suffer from the complexity of needing to worry about heavyweight migrate network and storage. > From my understanding, these models were feasible with Cells V1. I don't think cellsv1 supported any notion of moving things between cells at all, unless you had some sort of external hack for doing it. Being able to migrate between cells at all was always one of the things we touted as a "future feature" for cellsv2. Unless of course you mean migration in terms of snapshot-to-glance-and-redeploy? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration
> A release upgrade dance involves coordination of multiple moving > parts. It's about as similar to this scenario as I can imagine. And > there's a reason release upgrades are not done entirely within Nova; > clearly an external upgrade tool or script is needed to orchestrate > the many steps and components involved in the upgrade process. I'm lost here, and assume we must be confusing terminology or something. > The similar dance for cross-cell migration is the coordination that > needs to happen between Nova, Neutron and Cinder. It's called > orchestration for a reason and is not what Nova is good at (as we've > repeatedly seen) Most other operations in Nova meet this criteria. Boot requires coordination between Nova, Cinder, and Neutron. As do migrate, start, stop, evacuate. We might decide that (for now) the volume migration thing is beyond the line we're willing to cross, and that's cool, but I think it's an arbitrary limitation we shouldn't assume is impossible. Moving instances around *is* what nova is (supposed to be) good at. > The thing that makes *this* particular scenario problematic is that > cells aren't user-visible things. User-visible things could much more > easily be orchestrated via external actors, as I still firmly believe > this kind of thing should be done. I'm having a hard time reconciling these: 1. Cells aren't user-visible, and shouldn't be (your words and mine). 2. Cross-cell migration should be done by an external service (your words). 3. External services work best when things are user-visible (your words). You say the user-invisible-ness makes orchestrating this externally difficult and I agree, but...is your argument here just that it shouldn't be done at all? >> As we discussed in YVR most recently, it also may become an important >> thing for operators and users where expensive accelerators are committed >> to instances with part-time usage patterns. > > I don't think that's a valid use case in respect to this scenario of > cross-cell migration. You're right, it has nothing to do with cross-cell migration at all. I was pointing to *other* legitimate use cases for shelve. > Also, I'd love to hear from anyone in the real world who has > successfully migrated (live or otherwise) an instance that "owns" > expensive hardware (accelerators, SR-IOV PFs, GPUs or otherwise). Again, the accelerator case has nothing to do with migrating across cells, but merely demonstrates another example of where shelve may be the thing operators actually desire. Maybe I shouldn't have confused the discussion by bringing it up. > The patterns that I have seen are one of the following: > > * Applications don't move. They are pets that stay on one or more VMs > or baremetal nodes and they grow roots. > > * Applications are designed to *utilize* the expensive hardware. They > don't "own" the hardware itself. > > In this latter case, the application is properly designed and stores > its persistent data in a volume and doesn't keep state outside of the > application volume. In these cases, the process of "migrating" an > instance simply goes away. You just detach the application persistent > volume, shut down the instance, start up a new one elsewhere (allowing > the scheduler to select one that meets the resource constraints in the > flavor/image), attach the volume again and off you go. No messing > around with shelving, offloading, migrating, or any of that nonsense > in Nova. Jay, you know I sympathize with the fully-ephemeral application case, right? Can we agree that pets are a thing and that migrations are not going to be leaving Nova's scope any time soon? If so, I think we can get back to the real discussion, and if not, I think we probably, er, can't :) > We should not pretend that what we're discussing here is anything > other than hacking orchestration workarounds into Nova to handle > poorly-designed applications that have grown roots on some hardware > and think they "own" hardware resources in a Nova deployment. I have no idea how we got to "own hardware resources" here. The point of this discussion is to make our instance-moving operations work across cells. We designed cellsv2 to be invisible and baked into the core of Nova. We intended for it to not fall into the trap laid by cellsv1, where the presence of multiple cells meant that a bunch of regular operations don't work like they would otherwise. If we're going to discuss removing move operations from Nova, we should do that in another thread. This one is about making existing operations work :) > If that's the case, why are we discussing shelve at all? Just stop the > instance, copy/migrate the volume data (if needed, again it completely > depends on the deployment, network topology and block storage > backend), to a new location (new cell, new AZ, new host agg, does it > really matter?) and start a new instance, attaching the volume after > the instance starts or supplying the volume in the boot/create >
Re: [Openstack-operators] [nova][cinder][neutron] Cross-cell cold migration
>> * Cells can shard across flavors (and hardware type) so operators >> would like to move users off the old flavors/hardware (old cell) to >> new flavors in a new cell. > > So cell migrations are kind of the new release upgrade dance. Got it. No, cell migrations are about moving instances between cells for whatever reason. If you have small cells for organization, then it's about not building arbitrary barriers between aisles. If you use it for hardware refresh, then it might be related to long term lifecycle. I'm not sure what any of this has to do with release upgrades or dancing. > shelve was and continues to be a hack in order for users to keep an > IPv4 address while not consuming compute resources for some amount of > time. [1] As we discussed in YVR most recently, it also may become an important thing for operators and users where expensive accelerators are committed to instances with part-time usage patterns. It has also come up more than once in the realm of "but I need to detach my root volume" scenarios. I love to hate on shelve as well, but recently a few more legit (than merely keeping an IPv4 address) use-cases have come out for it, and I don't think Matt is wrong that cross-cell migration *might* be easier as a shelve operation under the covers. > If cross-cell cold migration is similarly just about the user being > able to keep their instance's IPv4 address while allowing an admin to > move an instance's storage to another physical location, then my firm > belief is that this kind of activity needs to be coordinated > *externally to Nova*. I'm not sure how you could make that jump, but no, I don't think that's the case. In any sort of large cloud that uses cells to solve problems of scale, I think it's quite likely to expect that your IPv4 address physically can't be honored in the target cell, and/or requires some less-than-ideal temporary tunneling for bridging the gap. > Since we're not talking about live migration (thank all that is holy), Oh it's coming. Don't think it's not. > I believe the safest and most effective way to perform such a > cross-cell "migration" would be the following basic steps: > > 0. ensure that each compute node is associated with at least one nova > host aggregate that is *only* in a single cell > 1. shut down the instance (optionally snapshotting required local disk > changes if the user is unfortunately using their root disk for > application data) > 2. "save" the instance's IP address by manually creating a port in > Neutron and assigning the IP address manually to that port. this of > course will be deployment-dependent since you will need to hope the > saved IP address for the migrating instance is in a subnet range that > is available in the target cell > 3. migrate the volume manually. this will be entirely deployment and > backend-dependent as smcginnis alluded to in a response to this thread > 4. have the admin boot the instance in a host aggregate that is known > to be in the target cell, passing --network > port_id=$SAVED_PORT_WITH_IP and --volume $MIGRATED_VOLUME_UUID > arguments as needed. the admin would need to do this because users > don't know about host aggregates and, frankly, the user shouldn't know > about host aggregates, cells, or any of this. What you just described here is largely shelve, ignoring the volume migration part and the fact that such a manual process means the user loses the instance's uuid and various other elements about it (such as create time, action/event history, etc). Oh, and ignoring the fact that the user no longer owns their instance (the admin does) :) Especially given that migrating across a cell may mean "one aisle over, same storage provider and network" to a lot of people, the above being a completely manual process seems a little crazy to me. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova][cinder][neutron] Cross-cell cold migration
> I think Nova should never have to rely on Cinder's hosts/backends > information to do migrations or any other operation. > > In this case even if Nova had that info, it wouldn't be the solution. > Cinder would reject migrations if there's an incompatibility on the > Volume Type (AZ, Referenced backend, capabilities...) I think I'm missing a bunch of cinder knowledge required to fully grok this situation and probably need to do some reading. Is there some reason that a volume type can't exist in multiple backends or something? I guess I think of volume type as flavor, and the same definition in two places would be interchangeable -- is that not the case? > I don't know anything about Nova cells, so I don't know the specifics of > how we could do the mapping between them and Cinder backends, but > considering the limited range of possibilities in Cinder I would say we > only have Volume Types and AZs to work a solution. I think the only mapping we need is affinity or distance. The point of needing to migrate the volume would purely be because moving cells likely means you moved physically farther away from where you were, potentially with different storage connections and networking. It doesn't *have* to mean that, but I think in reality it would. So the question I think Matt is looking to answer here is "how do we move an instance from a DC in building A to building C and make sure the volume gets moved to some storage local in the new building so we're not just transiting back to the original home for no reason?" Does that explanation help or are you saying that's fundamentally hard to do/orchestrate? Fundamentally, the cells thing doesn't even need to be part of the discussion, as the same rules would apply if we're just doing a normal migration but need to make sure that storage remains affined to compute. > I don't know how the Nova Placement works, but it could hold an > equivalency mapping of volume types to cells as in: > > Cell#1Cell#2 > > VolTypeA <--> VolTypeD > VolTypeB <--> VolTypeE > VolTypeC <--> VolTypeF > > Then it could do volume retypes (allowing migration) and that would > properly move the volumes from one backend to another. The only way I can think that we could do this in placement would be if volume types were resource providers and we assigned them traits that had special meaning to nova indicating equivalence. Several of the words in that sentence are likely to freak out placement people, myself included :) So is the concern just that we need to know what volume types in one backend map to those in another so that when we do the migration we know what to ask for? Is "they are the same name" not enough? Going back to the flavor analogy, you could kinda compare two flavor definitions and have a good idea if they're equivalent or not... --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] increasing the number of allowed volumes attached per instance > 26
> Some ideas that have been discussed so far include: FYI, these are already in my order of preference. > A) Selecting a new, higher maximum that still yields reasonable > performance on a single compute host (64 or 128, for example). Pros: > helps prevent the potential for poor performance on a compute host > from attaching too many volumes. Cons: doesn't let anyone opt-in to a > higher maximum if their environment can handle it. I prefer this because I think it can be done per virt driver, for whatever actually makes sense there. If powervm can handle 500 volumes in a meaningful way on one instance, then that's cool. I think libvirt's limit should likely be 64ish. > B) Creating a config option to let operators choose how many volumes > allowed to attach to a single instance. Pros: lets operators opt-in to > a maximum that works in their environment. Cons: it's not discoverable > for those calling the API. This is a fine compromise, IMHO, as it lets operators tune it per compute node based on the virt driver and the hardware. If one compute is using nothing but iSCSI over a single 10g link, then they may need to clamp that down to something more sane. Like the per virt driver restriction above, it's not discoverable via the API, but if it varies based on compute node and other factors in a single deployment, then making it discoverable isn't going to be very easy anyway. > C) Create a configurable API limit for maximum number of volumes to > attach to a single instance that is either a quota or similar to a > quota. Pros: lets operators opt-in to a maximum that works in their > environment. Cons: it's yet another quota? Do we have any other quota limits that are per-instance like this would be? If not, then this would likely be weird, but if so, then this would also be an option, IMHO. However, it's too much work for what is really not a hugely important problem, IMHO, and both of the above are lighter-weight ways to solve this and move on. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] Concern about trusted certificates API change
> Maybe it wasn't clear but I'm not advocating that we block the change > until volume-backed instances are supported with trusted certs. I'm > suggesting we add a policy rule which allows deployers to at least > disable it via policy if it's not supported for their cloud. That's fine with me, and provides an out for another issue I pointed out on the code review. Basically, the operator has no way to disable this feature. If they haven't set this up properly and have no desire to, a user reading the API spec and passing trusted certs will not be able to boot an instance and not really understand why. > I agree. I'm the one that noticed the issue and pointed out in the > code review that we should explicitly fail the request if we can't > honor it. I agree for the moment for sure, but it would obviously be nice not to open another gap we're not going to close. There's no reason this can't be supported for volume-backed instances, it just requires some help from cinder. I would think that it'd be nice if we could declare the "can't do this for reasons" response as a valid one regardless of the cause so we don't need another microversion for the future where volume-backed instances can do this. > Again, I'm not advocating that we block until boot from volume is > supported. However, we have a lot of technical debt for "good > functionality" added over the years that failed to consider > volume-backed instances, like rebuild, rescue, backup, etc and it's > painful to deal with that after the fact, as can be seen from the > various specs proposed for adding that support to those APIs. Totes agree. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] about rebuild instance booted from volume
> Deleting all snapshots would seem dangerous though... > > 1. I want to reset my instance to how it was before > 2. I'll just do a snapshot in case I need any data in the future > 3. rebuild > 4. oops Yep, for sure. I think if there are snapshots, we have to refuse to do te thing. My comment was about the "does nova have authority to destroy the root volume during a rebuild" and I think it does, if delete_on_termination=True, and if there are no snapshots. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] AggregateMultiTenancyIsolation with multiple (many) projects
> 2. Dan Smith mentioned another idea such that we could index the > aggregate metadata keys like filter_tenant_id0, filter_tenant_id1, > ... filter_tenant_idN and then combine those so you have one host > aggregate filter_tenant_id* key per tenant. Yep, and that's what I've done in my request_filter implementation: https://review.openstack.org/#/c/545002/9/nova/scheduler/request_filter.py Basically it allows any suffix to 'filter_tenant_id' to be processed as a potentially-matching key. Note that I'm hoping we can deprecate/remove the post filter and replace it with this much more efficient version. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] Does anyone rely on PUT /os-services/disable for non-compute services?
Are we allowed to cheat and say auto-disabling non-nova-compute services on startup is a bug and just fix it that way for #2? :) Because (1) it doesn't make sense, as far as we know, and (2) it forces the operator to have to use the API to enable them later just to fix their nova service-list output. Yes, definitely. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] Does anyone rely on PUT /os-services/disable for non-compute services?
So it seems our options are:
1. Allow PUT /os-services/{service_uuid} on any type of service, even if
doesn't make sense for non-nova-compute services.
2. Change the behavior of [1] to only disable new "nova-compute" services.
Please, #2. Please.
--Dan
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [upgrades][skip-level][leapfrog] - RFC - Skipping releases when upgrading
> As most of the upgrade issues center around database migrations, we > discussed some of the potential pitfalls at length. One approach was to > roll-up all DB migrations into a single repository and run all upgrades > for a given project in one step. Another was to simply have mutliple > python virtual environments and just run in-line migrations from a > version specific venv (this is what the OSA tooling does). Does one way > work better than the other? Any thoughts on how this could be better? IMHO, and speaking from a Nova perspective, I think that maintaining a separate repo of migrations is a bad idea. We occasionally have to fix a migration to handle a case where someone is stuck and can't move past a certain revision due to some situation that was not originally understood. If you have a separate copy of our migrations, you wouldn't get those fixes. Nova hasn't compacted migrations in a while anyway, so there's not a whole lot of value there I think. The other thing to consider is that our _schema_ migrations often require _data_ migrations to complete before moving on. That means you really have to move to some milestone version of the schema, then move/transform data, and then move to the next milestone. Since we manage those according to releases, those are the milestones that are most likely to be successful if you're stepping through things. I do think that the idea of being able to generate a small utility container (using the broad sense of the word) from each release, and using those to step through N, N+1, N+2 to arrive at N+3 makes the most sense. Nova has offline tooling to push our data migrations (even though the command is intended to be runnable online). The concern I would have would be over how to push Keystone's migrations mechanically, since I believe they moved forward with their proposal to do data migrations in stored procedures with triggers. Presumably there is a need for something similar to nova's online-data-migrations command which will trip all the triggers and provide a green light for moving on? In the end, projects support N->N+1 today, so if you're just stepping through actual 1-version gaps, you should be able to do as many of those as you want and still be running "supported" transitions. There's a lot of value in that, IMHO. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Anyone else use vendordata_driver in nova.conf?
>> The current sticking point is that nova-core doesn't want to pass >> system-metadata to plugins, as the internal format of that changes >> between releases and isn't intended to be versioned. Instead, we want to >> pass just the specific bits people need. >> >> So -- what things are you using from system-metadata specifically? > > Right now just image_os_distro and image_os_version. Which may very well go away in the near future because of other ways we handle this. IMHO, including system_metadata should not be on the table. If you need things about the image, you should go ask the image service for that information, given what we tell you about the instance (i.e. image-id in this case). If you want to chase our internals, then dig it out of the database. Looking at system_metadata is looking at our internals, just like poking in our database. It's not something we should (effectively) codify in this interface because we WILL break it. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Upgrade OpenStack Juno to Mitaka
> +1 to everything Daniel said. Nova really expects release-to-release > upgrades. We do online data migrations between releases. Maybe one > reason you're getting this to work is we have a nova-manage command to > force the migration of data between releases rather than doing the > online data migration as resources are accessed. But there are some DB > migrations where we do a full stop until you've migrated the data. Yeah, this ^ We try to put blocker migrations into the stream at critical synchronization points to make sure that anything that *has* to be online migrated will have been complete before we roll forward (usually because we're about to drop something or introduce a constraint). However, I'm sure there are some subtleties we don't catch with that. To echo and expand on what Matt said, if you're going to do an accelerated upgrade, I would _at least_ recommend deploying the intermediate code an running all the online migrations to completion (after applying the intermediate schema updates) before rolling to the next. This means running this after "db sync" for releases after it was added: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L823 Just deploying the target code and running online migrations isn't really enough, because we often remove the migration code once we know it (should have been) run to completion. Since we can't know everyone's upgrade cadence, and since our official support is "N-1 to N", that's the price you pay for skipping a release. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] Is verification of images in the image cache necessary?
> It was my impression we were trying to prevent bitrot, not defend > against an attacker that has gained control over the compute node. I think we've established that addressing bitrot at the nova layer is (far) out of scope and not something we want or need to do in nova. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] Is verification of images in the image cache necessary?
> I like the idea of checking the md5 matches before each boot, as it > mirrors the check we do after downloading from glance. Its possible > thats very unlikely to spot anything that shouldn't already be worried > about by something else. It may just be my love of symmetry that makes > me like that idea? IMHO, checking this at boot after we've already checked it on download is not very useful. It supposes that the attacker was kind enough to visit our system before an instance was booted and not after. If I have rooted the system, it's far easier for me to show up after a bunch of instances are booted and modify the base images (or even better, the instance images themselves which are hard to validate from the host side). I would also point out that if I'm going to root a compute node, the first thing I'm going to do is disable the feature in nova-compute or in some other way cripple it so it can't do its thing. --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] [nova] Removing seeded flavors
Hi all, I just wanted to float this past the operators list for visibility: Historically Nova has seeded some default flavors in an initial install. The way it has done this is really atypical of anything else we do, as it's embedded in the initial database schema migration. Since we're moving where we store those flavors now, leaving their creation in the original migration means even new deploys will just have to move them to the new location. That, and we don't even use them for our own testing as they're too large. So, this will involve removing them from that migration, making sure that devstack creates you some flavors to use if you're going that route, and updates to the manuals describing the creation of base flavors alongside getting an image set up to use. For real deployments, there should be little or no effect, but PoC type deploys that are used to those flavors being present may need to run a couple of flavor-create commands when bootstrapping, just like you have to do for images. Thanks! --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova][vmware] Minimum VC version
But 4.x was EOL over a year ago: http://kb.vmware.com/selfservice/microsites/search.do?language=en_UScmd=displayKCexternalId=2039567 ...and was released in 2010. We're supporting a minimum version of libvirt from 2014, so I think that dropping support for five-year-old EOL'd VMware is good. We don't test it, which means it's probably broken. I also feel like this is a thing we can do without a deprecation cycle, assuming there aren't a ton of users still using unsupported commercial software out there. --Dan signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova][vmware] Minimum VC version
The proposed patch also drops support for 5.0, which as I understand it is not EOL'd? The documentation appears to indicate that some functionality will not work with 5.1, but it's not explicitly clear what that it is. Yeah, I guess I assumed that anyone on 5.0 was just late moving to =5.1, where people on 4.x might have a license standing in their way. Maybe refusing 5.0 and warning about 5.5 (which is what is now being tested) is the right thing to do? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
