Re: [openstack-dev] [neutron] upgrades PTG recap
On Mon, Mar 6, 2017 at 10:50 AM, Morales, Victor wrote: > Regarding this testing, is there a place where it was documented, scripted or > shared? Something that helps to someone else can take a look. And in the > other hand, is there a way to simulate a “fake change” for similar releases > where didn’t add significant features but we still need to guarantees its > functionality. I don't think Artur shared any write-up. It would be nice to have one for later reference. Artur, what d'you think? As for 'fake' change, I am not sure I follow what you mean. If/when we have gate job covering the scenario, it will be validated on each patch. Ihar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] upgrades PTG recap
On 3/6/17, 8:28 AM, "Ihar Hrachyshka" wrote: >Hi all, > >This is a report on upgrades related topics discussed during PTG in >Atlanta. A general PTG report from PTL can be found at: > >http://lists.openstack.org/pipermail/openstack-dev/2017-February/113032.html > >Topics discussed: >1. OVO progress; >2. running mixed server versions; >3. online data migration framework; >4. impact of new features (multiple port bindings) for mixed >server execution, and how to deal with it; >5. gate. > >=> OVO progress >The effort is a bit stalled but slowly progressing. Progress is >sometimes blocked by external ongoing initiatives: new oslo.db >enginefacade; other code refactoring. Sometimes we need to back off >and fix plugin database logic before switching to objects (think of >lock_for_update removal in all places in current code). The initiative >is critical only as long as there are other initiatives that would >benefit from it. At this point, we are mainly looking at multiple port >bindings feature and its potential impact on our ability to run mixed >server versions at the same time. Other objects are worth an effort >but are not critical. There may be other objects during the cycle that >may be determined critical for rolling upgrades support. > >Specifically talking about lock_for_update support in objects layer: >https://review.openstack.org/#/c/435598/ , Kevin is hesitant to add >it. Instead we are going to clean up remaining code that uses the >locking feature, like in https://review.openstack.org/#/c/438144/ . We >need the same approach applied for quotas. > >=> Running mixed server versions >Initial testing done by Artur showed that it's now possible to execute >Newton and Ocata server versions in the same environment with some >success. This is partly the result of OVO work, but also the fact that >Ocata cycle was short and not rich on new features. We should build on >top of that to deliver the same for Pike. We should give special >attention to initiatives that may disrupt the work (multiple port >bindings and others). Regarding this testing, is there a place where it was documented, scripted or shared? Something that helps to someone else can take a look. And in the other hand, is there a way to simulate a “fake change” for similar releases where didn’t add significant features but we still need to guarantees its functionality. > >=> Online data migration framework >We discussed that in general there are 3 steps needed for online >migration 1) Declaration of the intent and creating a general >framework for online migration, 2) CLI for online migration (example >patch https://review.openstack.org/#/c/432494/) and, 3) The changes >made in the object layer for online data migration needs to be >backward compatible. We sat down and looked at specific contract >migrations from the past as a matter of case study excercise. During >our discussion we found out that all the migrations might not be >addressed by the general framework and will require case-by-case code >to implement the needed transition. Actually, intent for most >non-obvious contractions would be hard to express in a general way. It seems like we can completely discard the point number one(the creation of a common framework), in other words, online data migration will be analyzed case-by-case. > >=> Impact of multiple port bindings >Discussions suggest that the feature may be of high impact for our >ability to deliver mixed server versions for Ocata->Pike upgrade. This >is not because of database access not being properly aligned for the >newly expected feature. Instead, we may hit issues because of the >nature of the extension usage, that is usually consumed by compute >component. Once nova switches to using the new extension to drive port >bindings if the extension is present, and if neutron replies to nova >that it indeed supports the new extension before all neutron-servers >are upgraded to Pike, then it may turn out that some data stored in >the database may not be easily implemented/acted on by older >neutron-servers. > >Ideally, we would instead block any new API requests till the moment >when all neutron-servers are running the new code, at which point they >could start advertising the fact on /extensions/ endpoint. That would >imply that neutron-servers become aware of each other, and extensions >each of them support and expose. In that case, they could negotiate to >use the common subset of extensions, and avoid exposing any extensions >that are not implemented by any other neutron-server. In that case, >newer servers would still behave as if they don't support new API >features. Then once the upgrade to the new release is complete, >neutron-servers could detect the end of upgrade phase (either through >some external event, like admin initiated signals; or by periodic >inspection of the server db table) and reload extensions to enable new >API. I am going to report the proposal as RFE and write i
[openstack-dev] [neutron] upgrades PTG recap
Hi all, This is a report on upgrades related topics discussed during PTG in Atlanta. A general PTG report from PTL can be found at: http://lists.openstack.org/pipermail/openstack-dev/2017-February/113032.html Topics discussed: 1. OVO progress; 2. running mixed server versions; 3. online data migration framework; 4. impact of new features (multiple port bindings) for mixed server execution, and how to deal with it; 5. gate. => OVO progress The effort is a bit stalled but slowly progressing. Progress is sometimes blocked by external ongoing initiatives: new oslo.db enginefacade; other code refactoring. Sometimes we need to back off and fix plugin database logic before switching to objects (think of lock_for_update removal in all places in current code). The initiative is critical only as long as there are other initiatives that would benefit from it. At this point, we are mainly looking at multiple port bindings feature and its potential impact on our ability to run mixed server versions at the same time. Other objects are worth an effort but are not critical. There may be other objects during the cycle that may be determined critical for rolling upgrades support. Specifically talking about lock_for_update support in objects layer: https://review.openstack.org/#/c/435598/ , Kevin is hesitant to add it. Instead we are going to clean up remaining code that uses the locking feature, like in https://review.openstack.org/#/c/438144/ . We need the same approach applied for quotas. => Running mixed server versions Initial testing done by Artur showed that it's now possible to execute Newton and Ocata server versions in the same environment with some success. This is partly the result of OVO work, but also the fact that Ocata cycle was short and not rich on new features. We should build on top of that to deliver the same for Pike. We should give special attention to initiatives that may disrupt the work (multiple port bindings and others). => Online data migration framework We discussed that in general there are 3 steps needed for online migration 1) Declaration of the intent and creating a general framework for online migration, 2) CLI for online migration (example patch https://review.openstack.org/#/c/432494/) and, 3) The changes made in the object layer for online data migration needs to be backward compatible. We sat down and looked at specific contract migrations from the past as a matter of case study excercise. During our discussion we found out that all the migrations might not be addressed by the general framework and will require case-by-case code to implement the needed transition. Actually, intent for most non-obvious contractions would be hard to express in a general way. => Impact of multiple port bindings Discussions suggest that the feature may be of high impact for our ability to deliver mixed server versions for Ocata->Pike upgrade. This is not because of database access not being properly aligned for the newly expected feature. Instead, we may hit issues because of the nature of the extension usage, that is usually consumed by compute component. Once nova switches to using the new extension to drive port bindings if the extension is present, and if neutron replies to nova that it indeed supports the new extension before all neutron-servers are upgraded to Pike, then it may turn out that some data stored in the database may not be easily implemented/acted on by older neutron-servers. Ideally, we would instead block any new API requests till the moment when all neutron-servers are running the new code, at which point they could start advertising the fact on /extensions/ endpoint. That would imply that neutron-servers become aware of each other, and extensions each of them support and expose. In that case, they could negotiate to use the common subset of extensions, and avoid exposing any extensions that are not implemented by any other neutron-server. In that case, newer servers would still behave as if they don't support new API features. Then once the upgrade to the new release is complete, neutron-servers could detect the end of upgrade phase (either through some external event, like admin initiated signals; or by periodic inspection of the server db table) and reload extensions to enable new API. I am going to report the proposal as RFE and write it up in form of a spec where we will be able to discuss it in more details. Since multiple port bindings work is scheduled to land this cycle, we should work on that proposal in Pike too. => Gating It's still not clear what to do with mixed server version gating, and apparently we will need to sync again with folks driving it for nova gate. In the meantime, we may want to continue periodic local validations of mixed setup to assure nothing is borked. As for grenade multinode linuxbridge job, there is lack of clear interest in the subgroup, so I asked Kevin if he is interested in making it working. Kevin said he will tak