Re: [openstack-dev] [ironic] The scenary to rolling upgrade Ironic
On Wed, Oct 14, 2015 at 04:09:07PM -0700, Dan Smith wrote: > > Conductors will always need to talk to the database. APIs may not need > > to talk to the database. I think we can just roll conductor > > upgrades through, and then update ironic-api after that. This should > > just work, as long as we're very careful about schema changes (this is > > where the expand/contract thing comes into play). Different versions of > > conductors are only a problem if the database schema is not compatible > > with one of the versions. > > Yep, this seems like it's probably the right approach, assuming that > your API depending on RPC is still reasonable, performance-wise. It > might be confusing for people to hear that ironic's conductor goes last, > but nova's goes first. In reality, it was probably not a great idea to > have these named the same thing as they're not very similar, but oh well. Sorry, if it wasn't clear, I said conductor should go first here, and then API... // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] The scenary to rolling upgrade Ironic
> Conductors will always need to talk to the database. APIs may not need > to talk to the database. I think we can just roll conductor > upgrades through, and then update ironic-api after that. This should > just work, as long as we're very careful about schema changes (this is > where the expand/contract thing comes into play). Different versions of > conductors are only a problem if the database schema is not compatible > with one of the versions. Yep, this seems like it's probably the right approach, assuming that your API depending on RPC is still reasonable, performance-wise. It might be confusing for people to hear that ironic's conductor goes last, but nova's goes first. In reality, it was probably not a great idea to have these named the same thing as they're not very similar, but oh well. If you expand your DB schema separate from contract, it's not hard to just have a flag that controls the online migration activity in your objects so that it doesn't start happening until everything is upgraded that needs to care. With that, you could upgrade all the conductors one by one, and when the last one is upgraded, there is some indication that it's okay to start actually writing things into the new column(s), or whatever else. This is something I want nova's conductor to do so that we don't have to upgrade all the conductors at once, which is currently a requirement. We now have a global service_version counter in the database that will provide us the info necessary to control this change. --Dan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic] The scenary to rolling upgrade Ironic
On Wed, Oct 14, 2015 at 08:44:08AM +, Tan, Lin wrote: > Hi guys, > > I am looking at https://bugs.launchpad.net/ironic/+bug/1502903 which > is related to rolling upgrade and here is Jim's patch > https://review.openstack.org/#/c/234450 I really have a concern or > question about how to do Ironic doing rolling upgrades. It might be my > mistake, but I would like to discuss here and get some feedback. > > I manually did a rolling upgrade for a private OpenStack Cloud before. > There are three main tasks for upgrade: 1. upgrade the code of > service. 2. change configuration. 3. the upgrade of DB Schema in DB, > which is the most difficult and time-consuming part. > > The current rolling upgrade solution or live upgrade are highly > depends on upgrade different services in place one-by-one while make > new service A can still communicate with old service B. The ideal > case is after we upgrade one of the services, others can still work > without break. This is can be done by using versionedobject and RPC > version. For example, new Nova-API and new Nova-conductor can talk to > old Nova-compute. In the case of Nova services, it was suggests to > follow below steps: > 1. expand DB schema > 2. pin RPC versions and object version at current > 3. upgrade all nova-conductor servers because it will talk with DB > 4. upgrade all nova services on controller nodes like nova-api > 5. upgrade all nova-compute nodes > 6. unpin RPC versions > 7. shrink DB schema. > This is perfect for Nova. Because it has many > nova-compute nodes, and few nova-conductor nodes and nova-api nodes. > It's not necessary to upgrade nova-compute services at one time, which > is time consuming. > > For Ironic, we only have ir-conductor and ir-api. So the question is > should we upgrade ir-conductor first or ir-api? In my opinion, the > ideal case is that we can have old ir-conductor and new ir-conductors > coexist, which means we should upgrade ir-api to latest at first. But > it's impossible at the moment, because ir-conductor will talk to DB > directly and we only have one DB schema. That's a large difference > between Ironic and Nova. We are missing a layer like nova-conductor. > The second case is upgrade ir-conductors first. That means if we > upgrade the DB Schema, we have to upgrade all ir-conductors at once. > During the upgrade, we could not provide Ironic service at all. > > So I would suggest to stop all Ironic service, and upgrade ir-api > first, and then upgrade ir-conductor one by one. Only enable the > ir-conductor which has done the upgrade. Or upgrade ir-api and > ir-conductors at once, although it sounds stupid a little bit. Hey Tan, thanks for bringing this up. I've been thinking about this stuff a lot lately, and I'd like us to get it working during the Mitaka cycle, so deployers can do a rolling upgrade from Liberty to Mitaka. Conductors will always need to talk to the database. APIs may not need to talk to the database. I think we can just roll conductor upgrades through, and then update ironic-api after that. This should just work, as long as we're very careful about schema changes (this is where the expand/contract thing comes into play). Different versions of conductors are only a problem if the database schema is not compatible with one of the versions. We also need to remote the objects layer to the conductor from the api service, so that the API service is no longer talking to the DB. And allow RPC version pinning. Beyond that, I think the Nova model should work fine for us. There's some work to do in our objects layer, and then lots of documentation for developers, reviewers, and deployers. I think it's totally reasonable to complete this during Mitaka, though. I opened this blueprint[0] yesterday to track this work. I'd like to get the developer/reviewer docs done first, so we don't accidentally land any changes that break assumptions here (for example, the bug you linked before). Is this something you're willing to take the lead on? // jim [0] https://blueprints.launchpad.net/ironic/+spec/online-upgrade-support __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ironic] The scenary to rolling upgrade Ironic
Hi guys, I am looking at https://bugs.launchpad.net/ironic/+bug/1502903 which is related to rolling upgrade and here is Jim's patch https://review.openstack.org/#/c/234450 I really have a concern or question about how to do Ironic doing rolling upgrades. It might be my mistake, but I would like to discuss here and get some feedback. I manually did a rolling upgrade for a private OpenStack Cloud before. There are three main tasks for upgrade: 1. upgrade the code of service. 2. change configuration. 3. the upgrade of DB Schema in DB, which is the most difficult and time-consuming part. The current rolling upgrade solution or live upgrade are highly depends on upgrade different services in place one-by-one while make new service A can still communicate with old service B. The ideal case is after we upgrade one of the services, others can still work without break. This is can be done by using versionedobject and RPC version. For example, new Nova-API and new Nova-conductor can talk to old Nova-compute. In the case of Nova services, it was suggests to follow below steps: 1. expand DB schema 2. pin RPC versions and object version at current 3. upgrade all nova-conductor servers because it will talk with DB 4. upgrade all nova services on controller nodes like nova-api 5. upgrade all nova-compute nodes 6. unpin RPC versions 7. shrink DB schema. This is perfect for Nova. Because it has many nova-compute nodes, and few nova-conductor nodes and nova-api nodes. It's not necessary to upgrade nova-compute services at one time, which is time consuming. For Ironic, we only have ir-conductor and ir-api. So the question is should we upgrade ir-conductor first or ir-api? In my opinion, the ideal case is that we can have old ir-conductor and new ir-conductors coexist, which means we should upgrade ir-api to latest at first. But it's impossible at the moment, because ir-conductor will talk to DB directly and we only have one DB schema. That's a large difference between Ironic and Nova. We are missing a layer like nova-conductor. The second case is upgrade ir-conductors first. That means if we upgrade the DB Schema, we have to upgrade all ir-conductors at once. During the upgrade, we could not provide Ironic service at all. So I would suggest to stop all Ironic service, and upgrade ir-api first, and then upgrade ir-conductor one by one. Only enable the ir-conductor which has done the upgrade. Or upgrade ir-api and ir-conductors at once, although it sounds stupid a little bit. What do you guys think? Best Regards, Tan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
