Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Hi Eric and all,     Thank you very much for considering my concerns and coming back with an improved solution. Glad that no blood was shed in the process. I took this proposal and worked out its details, as I understand them, in this etherpad:

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

On Mar 29, 2018, at 12:57 PM, Eric Fried wrote: > >> That means that for the (re)-programming scenarios you need to >> dynamically adjust the inventory of a particular FPGA resource provider. > > Oh, see, this is something I had *thought* was a non-starter. I need to work

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

> ==> Fully dynamic: You can program one region with one function, and > then still program a different region with a different function, etc. Note that this is also the case if you don't have virtualized multi-slot devices. Like, if you had one that only has one region. Consuming it consumes the

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

> That means that for the (re)-programming scenarios you need to > dynamically adjust the inventory of a particular FPGA resource provider. Oh, see, this is something I had *thought* was a non-starter. This makes the "single program" case way easier to deal with, and allows it to be handled on

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

On 03/28/2018 07:03 PM, Nadathur, Sundar wrote: Thanks, Eric. Looks like there are no good solutions even as candidates, but only options with varying levels of unacceptability. It is funny that that the option that is considered the least unacceptable is to let the problem happen and then

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

We discussed this on IRC [1], hangout, and etherpad [2]. Here is the summary, which we mostly seem to agree on: There are two different classes of device we're talking about modeling/managing. (We don't know the real nomenclature, so forgive errors in that regard.) ==> Fully dynamic: You can

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Sundar- To be clear, *all* of the solutions will have race conditions. There's no getting around the fact that we need to account for situations where an allocation is made, but then can't be satisfied by cyborg (or neutron, or nova, or cinder, or whoever). That failure has to bubble up

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Agree with that, whatever the tweak inventory or traits, none of them works. Same as VGPU, we can support pre-programmed mode for multiple-functions region, and each region only can support one type function. There are two reasons why Cyborg has a filter: * records the usage of functions in a

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Thanks, Eric. Looks like there are no good solutions even as candidates, but only options with varying levels of unacceptability. It is funny that that the option that is considered the least unacceptable is to let the problem happen and then fail the request (last one in your list). Could I

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Sundar- We're running across this issue in several places right now. One thing that's definitely not going to get traction is automatically/implicitly tweaking inventory in one resource class when an allocation is made on a different resource class (whether in the same or different

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Hi Shaohe,   I have responded in the Etherpad. The Cyborg/Nova scheduling spec details the 4 types of user requests . I believe you are looking for more details on

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Hi Eric and all,     I should have clarified that this race condition happens only for the case of devices with multiple functions. There is a prior thread about it. I was trying to get a solution within Cyborg, but

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

I have summarize some scenarios for fpga devices request. https://etherpad.openstack.org/p/cyborg-fpga-request-scenarios Please add more more scenarios to find out the exceptions that placement can not satisfy the filter and weight. IMOH, I refer placement to do filter and weight. If we

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

As I know placement and nova scheduler dedicate to filter and weight. Placement and nova scheduler is responsible for avoiding race. Nested provider + traits should cover most scenarios. Any special case please let the nova developer and cyborg developer know, let work together to get a

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

On 3/23/2018 12:44 PM, Eric Fried wrote: Sundar- First thought is to simplify by NOT keeping inventory information in the cyborg db at all. The provider record in the placement service already knows the device (the provider ID, which you can look up in the cyborg db) the host (the

Re: [openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Sundar- First thought is to simplify by NOT keeping inventory information in the cyborg db at all. The provider record in the placement service already knows the device (the provider ID, which you can look up in the cyborg db) the host (the root_provider_uuid of the provider representing

[openstack-dev] [nova] [cyborg] Race condition in the Cyborg/Nova flow

Hi all,     There seems to be a possibility of a race condition in the Cyborg/Nova flow. Apologies for missing this earlier. (You can refer to the proposed Cyborg/Nova spec for details.) Consider the scenario