Re: [openstack-dev] [nova] [placement] Modeling SR-IOV with nested resource providers
On 09/13/2017 03:01 PM, Chris Dent wrote: On Wed, 13 Sep 2017, Jay Pipes wrote: We still need a way to represent a request to placement to find allocation candidates for like resources, though. As you pointed out, I've thought about using multiple requests to placement from the conductor or scheduler. We could also do something like this: GET /allocation_candidates?resources=VCPU:1,MEMORY_MB:1024&resources1=SRIOV_NET_VF:1&required1=CUSTOM_PHYSNET_A,CUSTOM_SWITCH_1&resources2=SRIOV_NET_VF:1&required2=CUSTOM_PHYSNET_A,CUSTOM_SWITCH_2 To clarify, this translates to: * give me one compute node with 1 VCPU and 1024 MEMORY_MB that has * 2 vf * both on physnet A * one on switch 1 * one on switch 2 Yup, zactly. -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] Modeling SR-IOV with nested resource providers
On Wed, 13 Sep 2017, Jay Pipes wrote: We still need a way to represent a request to placement to find allocation candidates for like resources, though. As you pointed out, I've thought about using multiple requests to placement from the conductor or scheduler. We could also do something like this: GET /allocation_candidates?resources=VCPU:1,MEMORY_MB:1024&resources1=SRIOV_NET_VF:1&required1=CUSTOM_PHYSNET_A,CUSTOM_SWITCH_1&resources2=SRIOV_NET_VF:1&required2=CUSTOM_PHYSNET_A,CUSTOM_SWITCH_2 To clarify, this translates to: * give me one compute node with 1 VCPU and 1024 MEMORY_MB that has * 2 vf * both on physnet A * one on switch 1 * one on switch 2 -- Chris Dent (⊙＿⊙') https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] Modeling SR-IOV with nested resource providers
On 09/05/2017 11:02 AM, Andrey Volkov wrote: For example, I have SR-IOV PF with four ports (P_i), two of them are connected to one switch (SW_1) and other two to another (SW_2). I would like to get VFs from distinct ports connected to distinct switches (more details can be found in spec [1]), how it can be modeled with nested resource providers? Several possible solutions I see: 1) --- compute node - / / \ \-- -/ /\\--- / / \ \ SR-IOV PF SR-IOV PF SR-IOV PF SR-IOV PF (traits:P1,SW1) (traits:P2,SW1) (traits:P3,SW2) (traits:P4,SW2) : : : : / \ / \ / \ / \ / \ / \ / \ / \ VF1VF2VF3VF4VF5VF6VF7VF8 2) compute node / \ / \ SR-IOV PF SR-IOV PF (traits:SW1)(traits:SW2) / \/ \ /\ /\ SR-IOV PF SR-IOV PF SR-IOV PF SR-IOV PF (traits:P1) (traits:P2) (traits:P3) (traits:P4) : : : : / \ / \ / \ / \ / \ / \ / \ / \ VF1VF2VF3VF4VF5VF6VF7VF8 3) Use tags for inventories, so the problem can be solved without complex structures. Are the described options applicable or there are other to solve the issue? Hey Andrey, sorry for the long delay in getting back to you on this. A few points to consider. First, you would only need to have VFs represented as separate nodes in the tree (i.e. separate resource providers) if the VFs had different traits associated with them (for instance, if some hardware offloads were programmable on some of the VFs but not others). Secondly, the port should not be a trait. I think perhaps you're referring to a physical network as a port, though. If this is the case, then yes, you'd want that physical network to be a custom trait (e.g. CUSTOM_PHYSNET_INTRANET or something like that). So, with that said, you're looking instead at a structure like this: CN |--> PF1 |--> PF2 |--> PF3 |--> PF4 with the following records in the inventories table: RPRC Total -- CNVCPU4 CNMEMORY_MB 8192 RP1 SRIOV_NET_VF2 RP2 SRIOV_NET_VF2 RP3 SRIOV_NET_VF2 RP4 SRIOV_NET_VF2 and the following in the resource_provider_traits table: RPTRAIT --- RP1 CUSTOM_PHYSNET_A RP1 CUSTOM_SWITCH_1 RP2 CUSTOM_PHYSNET_B RP2 CUSTOM_SWITCH_1 RP3 CUSTOM_PHYSNET_A RP3 CUSTOM_SWITCH_2 RP4 CUSTOM_PHYSNET_B RP4 CUSTOM_SWITCH_2 We discussed yesterday that the Neutron ML2 agent would be responsible for inserting trait associations for the resource providers representing the SRIOV physical functions. We still need a way to represent a request to placement to find allocation candidates for like resources, though. As you pointed out, I've thought about using multiple requests to placement from the conductor or scheduler. We could also do something like this: GET /allocation_candidates?resources=VCPU:1,MEMORY_MB:1024&resources1=SRIOV_NET_VF:1&required1=CUSTOM_PHYSNET_A,CUSTOM_SWITCH_1&resources2=SRIOV_NET_VF:1&required2=CUSTOM_PHYSNET_A,CUSTOM_SWITCH_2 Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] Modeling SR-IOV with nested resource providers
Ed, Thanks for the response. I'm also interested how those models can be used from two points of view. First, how I can request the desired configuration. I thought about some anti-affinity logic based on traits in Placement, but probably that's not a task for Placement. Solution Jay Pipes proposed [1] is to make several requests to /allocation_candidates and then combine a new request from responses. Second, how complicated it would be to update resource provider structure if some conditions are changed (port was connected to a different switch). I agree that simple structure is preferable here, for me having PFs as resource providers and VFs as inventories with tags (third option in the previos post) is closer to reality than hierarchical resource providers. What do you think? FYI Eric Fried started an etherpad about generic device management [2]. [1] http://paste.openstack.org/show/620456/ [2] https://etherpad.openstack.org/p/nova-ptg-queens-generic-device-management On Wed, Sep 6, 2017 at 11:17 PM, Ed Leafe wrote: > On Sep 5, 2017, at 10:02 AM, Andrey Volkov wrote: > > > For example, I have SR-IOV PF with four ports (P_i), two of them are > > connected to one switch (SW_1) and other two to another (SW_2). I > > would like to get VFs from distinct ports connected to distinct > > switches (more details can be found in spec [1]), how it can be > > modeled with nested resource providers? > > You should make it as complicated as it needs to be, but no more. The > first model nests one deep, while the second goes two levels deep, yet they > both provide the same granularity for accessing the VFs, so I would go for > the first. And I’m not sure that we will be able to get the “inherited” > traits used in the second model implemented any time soon. > > -- Ed Leafe > > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Thanks, Andrey Volkov, Software Engineer, Mirantis, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] [placement] Modeling SR-IOV with nested resource providers
On Sep 5, 2017, at 10:02 AM, Andrey Volkov wrote: > For example, I have SR-IOV PF with four ports (P_i), two of them are > connected to one switch (SW_1) and other two to another (SW_2). I > would like to get VFs from distinct ports connected to distinct > switches (more details can be found in spec [1]), how it can be > modeled with nested resource providers? You should make it as complicated as it needs to be, but no more. The first model nests one deep, while the second goes two levels deep, yet they both provide the same granularity for accessing the VFs, so I would go for the first. And I’m not sure that we will be able to get the “inherited” traits used in the second model implemented any time soon. -- Ed Leafe __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
