Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic
Hi Julia, Apologize we were not able to be there to better represent the use case. PSB From: Julia Kreger Sent: Monday, October 1, 2018 11:07 PM To: OpenStack Development Mailing List (not for usage questions) Cc: isaku.yamah...@intel.com; Eyal Lavee Subject: Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic Greetings, Comments in-line. Thanks, -Julia On Sat, Sep 29, 2018 at 11:27 PM Moshe Levi mailto:mosh...@mellanox.com>> wrote: Hi Julia, I don't mind to update the ironic spec [1]. Unfortunately, I wasn't in the PTG but I had a sync meeting with Isuku. As I see it there is 2 use-cases: 1. Running the neutron ovs agent in the smartnic 2. Running the neutron super ovs agent which manage the ovs running on the smartnic. My takeaway from the meeting with neutron is that there would not be a neutron ovs agent running on the smartnic. That the configuration would need to be pushed at all times, which is ultimately better security wise if the tenant NIC is somehow compromised it reduces the control plane exposure. [ML] - Can you elaborate on the security concerns with running the neutron ovs agent on the smart NIC? If you compare this to the standard virtualization use case, this is as secure if not more secure. The tenant image runs in the bare metal host and receives only a network interface/port. The host has no way to access the OS/services/agents running on the smart NIC CPUs, in the same way that a tenant image running in a VM has no way to access the services/agents running in the hypervisor. It is in fact event more secure, as they are running in physically disjoint hardware and memory (thus not accessible even through side-channel vulnerabilities such as meltdown/spectre). 1. It seem that most of the discussion was around the second use-case. By the time Ironic and Neutron met together, it seemed like the first use case was no longer under consideration. I may be wrong, but very strong preference existed for the second scenario when we met the next day. [ML] – We are seeing great interest on smart NICs for bare metal use cases to allow to provide services (networking, storage and others) to bare metal servers that were previously only possible for VMs. Conceptually the smart NIC can be thought of as an isolated hypervisor layer for the bare metal host. The first service we are targeting in this spec is aligning the bare metal networking with the standard neutron ovs agent. The target is to try to align (as possible) the bare metal implementation to the virtualization use case, up to the point of actually running (as possible) the same agents on the smart NIC (again acting as a hypervisor for the bare metal host use case). This allows to reuse/align the implementation, and naturally scales with the number of bare metal servers, as opposed to running the agents on controller nodes, requiring potentially scaling the controllers to match the number of bare metal servers. It also provides a path to providing more advanced services in the smart NIC in the next steps (not limiting the implementation to be OVSDB protocol specific). This is my understanding on the ironic neutron PTG meeting: 1. Ironic cores don't want to change the deployment interface as proposed in [1]. 2. We should a new network_interface for use case 2. But what about the first use case? Should it be a new network_interface as well? 3. We should delay the port binding until the baremetal is powered on the ovs is running. * For the first use case I was thinking to change the neutron server to just keep the port binding information in the neutron DB. Then when the neutron ovs agent is a live it will retrieve all the baremeal port , add them to the ovsdb and start the neutron ovs agent fullsync. * For the second use case the agent is alive so the agent itself can monitor the ovsdb of the baremetal and configure it when it up 1. How to notify that neutron agent successfully/unsuccessfully bind the port ? * In both use-cases we should use neutron-ironic notification to make sure the port binding was done successfully. Is my understanding correct? Not quite. 1) We as in Ironic recognize that there would need to be changes, it is the method as to how that we would prefer to be explicit and have chosen by the interface. The underlying behavior needs to be different, and the new network_interface should support both cases 1 and 2 because that interface contain needed logic for the conductor to determine the appropriate path forward. We should likely also put some guards in to prevent non-smart interfaces from being used in the same configuration due to the security issues that creates. 3) I believe this would be more of a matter of the network_interface knowing that the machine is powered up, and attempting to assert configuration through Neutron to push configuration to the smartnic. 3a) The con
Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic
Greetings, Comments in-line. Thanks, -Julia On Sat, Sep 29, 2018 at 11:27 PM Moshe Levi wrote: > Hi Julia, > > > > I don't mind to update the ironic spec [1]. Unfortunately, I wasn't in the > PTG but I had a sync meeting with Isuku. > > > > As I see it there is 2 use-cases: > >1. Running the neutron ovs agent in the smartnic >2. Running the neutron super ovs agent which manage the ovs running on >the smartnic. > > My takeaway from the meeting with neutron is that there would not be a neutron ovs agent running on the smartnic. That the configuration would need to be pushed at all times, which is ultimately better security wise if the tenant NIC is somehow compromised it reduces the control plane exposure. >1. > > > > It seem that most of the discussion was around the second use-case. > By the time Ironic and Neutron met together, it seemed like the first use case was no longer under consideration. I may be wrong, but very strong preference existed for the second scenario when we met the next day. > > This is my understanding on the ironic neutron PTG meeting: > >1. Ironic cores don't want to change the deployment interface as >proposed in [1]. >2. We should a new network_interface for use case 2. But what about >the first use case? Should it be a new network_interface as well? >3. We should delay the port binding until the baremetal is powered on >the ovs is running. > 1. For the first use case I was thinking to change the neutron > server to just keep the port binding information in the neutron DB. Then > when the neutron ovs agent is a live it will retrieve all the baremeal > port > , add them to the ovsdb and start the neutron ovs agent fullsync. > 2. For the second use case the agent is alive so the agent itself > can monitor the ovsdb of the baremetal and configure it when it up >4. How to notify that neutron agent successfully/unsuccessfully bind >the port ? > 1. In both use-cases we should use neutron-ironic notification to > make sure the port binding was done successfully. > > > > Is my understanding correct? > > > Not quite. 1) We as in Ironic recognize that there would need to be changes, it is the method as to how that we would prefer to be explicit and have chosen by the interface. The underlying behavior needs to be different, and the new network_interface should support both cases 1 and 2 because that interface contain needed logic for the conductor to determine the appropriate path forward. We should likely also put some guards in to prevent non-smart interfaces from being used in the same configuration due to the security issues that creates. 3) I believe this would be more of a matter of the network_interface knowing that the machine is powered up, and attempting to assert configuration through Neutron to push configuration to the smartnic. 3a) The consensus is that the information to access the smartnic is hardware configuration metadata and that ironic should be the source of truth for information about that hardware. The discussion was push that as needed into neutron to help enable the attachment. I proposed just including it in the binding profile as a possibility, since it is transient information. 3b) As I understood it, this would ultimately be the default operating behavior. 4) Was not discussed, but something along the path is going to have to check and retry as necessary. That item could be in the network_interface code. 4a) This doesn't exist yet. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic
Hello. I'm willing to help it. For detailed tech discussion, gerrit with updated spec would be better, though. I'd like to supplement on Neutron port binding. On Sun, Sep 30, 2018 at 06:25:58AM +, Moshe Levi wrote: > Hi Julia, > > I don't mind to update the ironic spec [1]. Unfortunately, I wasn't in the > PTG but I had a sync meeting with Isuku. > > As I see it there is 2 use-cases: > > 1. Running the neutron ovs agent in the smartnic > 2. Running the neutron super ovs agent which manage the ovs running on the > smartnic. > > It seem that most of the discussion was around the second use-case. > > This is my understanding on the ironic neutron PTG meeting: > > 1. Ironic cores don't want to change the deployment interface as proposed > in [1]. > 2. We should a new network_interface for use case 2. But what about the > first use case? Should it be a new network_interface as well? * Which component, Ironic or Neutron, should take care that SmartNIC is up/ready? * How is the up/readiness of smartnic defined? Common way or device-dependent way. For example, - able to connect via ovsdb/openflow - agent responsible for smartNIC has reported to Neutron agentdb. - able to ssh into smartnic - device specific way with driver for each device. - etc. > 3. We should delay the port binding until the baremetal is powered on the > ovs is running. > * For the first use case I was thinking to change the neutron server > to just keep the port binding information in the neutron DB. Then when the > neutron ovs agent is a live it will retrieve all the baremeal port , add them > to the ovsdb and start the neutron ovs agent fullsync. > * For the second use case the agent is alive so the agent itself can > monitor the ovsdb of the baremetal and configure it when it up Can you please elaborate why port binding for smartnic should be delayed? I'm failing to see the necessity. Probably I'm missing something. Since we can skip the check of agent liveness with the assumption that hostid given by Ironic is correct, we don't have to delay the port binding for both case, 1 and 2 as above. > 4. How to notify that neutron agent successfully/unsuccessfully bind the > port ? > * In both use-cases we should use neutron-ironic notification to make > sure the port binding was done successfully. I agree that neutron-ironic notification is necessary. There seems the confusion of port-binding and ovs-programing. The success/failure of port-binding is the result of neutron port-update. The current code synchronously checks the ovs-agent liveness on the host and parameter validity. port-binding doesn't directly/synchronously program ovs. It only triggers to start ovs programming eventually. On the other hand, the success/failure of ovs programming is asynchronous regarding to neutron REST API because ovs-agent does it asynchronously on behalf of neutron server. So here neutron-ironic notification is necessary. When the ovs programming is done, the agent sets port::status = UP from DOWN. (and nova is notified that port is ready to use.) In the case of the failure, port::status is set to ERROR. Thanks, > > Is my understanding correct? > > > > [1] - https://review.openstack.org/#/c/582767/ > > From: Miguel Lavalle > Sent: Sunday, September 30, 2018 3:20 AM > To: OpenStack Development Mailing List > Subject: Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic > > Hi, > > Yes, this also matches the recollection of the joint conversation in Denver. > Please look at the "Ironic x-project discussion - Smartnics" section in > http://lists.openstack.org/pipermail/openstack-dev/2018-September/135032.html<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fpipermail%2Fopenstack-dev%2F2018-September%2F135032.html=02%7C01%7Cmoshele%40mellanox.com%7Ce5607928a41c44bf065508d6266a9435%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636738636550233328=ZvAsZoTkk%2FFGlsig54J0d1EejcDiYdqsGlfRkuKBJTg%3D=0> > > Regards > > Miguel > > > > On Thu, Sep 27, 2018 at 1:31 PM Julia Kreger > mailto:juliaashleykre...@gmail.com>> wrote: > Greetings everyone, > > Now that the PTG is over, I would like to go ahead and get the > specification that was proposed to ironic-specs updated to represent > the discussions that took place at the PTG. > > A few highlights from my recollection: > > * Ironic being the source of truth for the hardware configuration for > the neutron agent to determine where to push configuration to. This > would include the address and credential information (certificates > right?). > * The information required is somehow sent to neutron (possibly as > part of t
Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic
Hi Julia, I don't mind to update the ironic spec [1]. Unfortunately, I wasn't in the PTG but I had a sync meeting with Isuku. As I see it there is 2 use-cases: 1. Running the neutron ovs agent in the smartnic 2. Running the neutron super ovs agent which manage the ovs running on the smartnic. It seem that most of the discussion was around the second use-case. This is my understanding on the ironic neutron PTG meeting: 1. Ironic cores don't want to change the deployment interface as proposed in [1]. 2. We should a new network_interface for use case 2. But what about the first use case? Should it be a new network_interface as well? 3. We should delay the port binding until the baremetal is powered on the ovs is running. * For the first use case I was thinking to change the neutron server to just keep the port binding information in the neutron DB. Then when the neutron ovs agent is a live it will retrieve all the baremeal port , add them to the ovsdb and start the neutron ovs agent fullsync. * For the second use case the agent is alive so the agent itself can monitor the ovsdb of the baremetal and configure it when it up 4. How to notify that neutron agent successfully/unsuccessfully bind the port ? * In both use-cases we should use neutron-ironic notification to make sure the port binding was done successfully. Is my understanding correct? [1] - https://review.openstack.org/#/c/582767/ From: Miguel Lavalle Sent: Sunday, September 30, 2018 3:20 AM To: OpenStack Development Mailing List Subject: Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic Hi, Yes, this also matches the recollection of the joint conversation in Denver. Please look at the "Ironic x-project discussion - Smartnics" section in http://lists.openstack.org/pipermail/openstack-dev/2018-September/135032.html<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fpipermail%2Fopenstack-dev%2F2018-September%2F135032.html=02%7C01%7Cmoshele%40mellanox.com%7Ce5607928a41c44bf065508d6266a9435%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636738636550233328=ZvAsZoTkk%2FFGlsig54J0d1EejcDiYdqsGlfRkuKBJTg%3D=0> Regards Miguel On Thu, Sep 27, 2018 at 1:31 PM Julia Kreger mailto:juliaashleykre...@gmail.com>> wrote: Greetings everyone, Now that the PTG is over, I would like to go ahead and get the specification that was proposed to ironic-specs updated to represent the discussions that took place at the PTG. A few highlights from my recollection: * Ironic being the source of truth for the hardware configuration for the neutron agent to determine where to push configuration to. This would include the address and credential information (certificates right?). * The information required is somehow sent to neutron (possibly as part of the binding profile, which we could send at each time port actions are requested by Ironic.) * The Neutron agent running on the control plane connects outbound to the smartnic, using information supplied to perform the appropriate network configuration. * In Ironic, this would likely be a new network_interface driver module, with some additional methods that help facilitate the work-flow logic changes needed in each deploy_interface driver module. * Ironic would then be informed or gain awareness that the configuration has been completed and that the deployment can proceed. (A different spec has been proposed regarding this.) I have submitted a forum session based upon this and the agreed upon goal at the PTG was to have the ironic spec written up to describe the required changes. I guess the next question is, who wants to update the specification? -Julia __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2FOpenStack-dev-request%40lists.openstack.org%3Fsubject%3Aunsubscribe=02%7C01%7Cmoshele%40mellanox.com%7Ce5607928a41c44bf065508d6266a9435%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636738636550233328=PexKWkCH7u4kRWjzs2kOIZsHFmzSL%2BCl6bqzY2B5KWA%3D=0> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack-dev=02%7C01%7Cmoshele%40mellanox.com%7Ce5607928a41c44bf065508d6266a9435%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636738636550243338=dHTuFDlyKrA8ouPU7JV4GKokCKNBg%2Fzw9KlpIrZW5x0%3D=0> __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][neutron] SmartNics with Ironic
Hi, Yes, this also matches the recollection of the joint conversation in Denver. Please look at the "Ironic x-project discussion - Smartnics" section in http://lists.openstack.org/pipermail/openstack-dev/2018-September/135032.html Regards Miguel On Thu, Sep 27, 2018 at 1:31 PM Julia Kreger wrote: > Greetings everyone, > > Now that the PTG is over, I would like to go ahead and get the > specification that was proposed to ironic-specs updated to represent > the discussions that took place at the PTG. > > A few highlights from my recollection: > > * Ironic being the source of truth for the hardware configuration for > the neutron agent to determine where to push configuration to. This > would include the address and credential information (certificates > right?). > * The information required is somehow sent to neutron (possibly as > part of the binding profile, which we could send at each time port > actions are requested by Ironic.) > * The Neutron agent running on the control plane connects outbound to > the smartnic, using information supplied to perform the appropriate > network configuration. > * In Ironic, this would likely be a new network_interface driver > module, with some additional methods that help facilitate the > work-flow logic changes needed in each deploy_interface driver module. > * Ironic would then be informed or gain awareness that the > configuration has been completed and that the deployment can proceed. > (A different spec has been proposed regarding this.) > > I have submitted a forum session based upon this and the agreed upon > goal at the PTG was to have the ironic spec written up to describe the > required changes. > > I guess the next question is, who wants to update the specification? > > -Julia > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
