Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
Sorry for delay in responding on this. Comments inline. On 05/29/2018 07:33 PM, Nadathur, Sundar wrote: Hi all, The Cyborg/Nova scheduling spec [1] details what traits will be applied to the resource providers that represent devices like GPUs. Some of the traits referred to vendor names. I got feedback that traits must not refer to products or specific models of devices. It's not that traits are not allowed to reference vendor names or identifiers. Just take a look at the entire module in os-traits that is designated with x86 CPU instruction set extensions: https://github.com/openstack/os-traits/blob/master/os_traits/hw/cpu/x86.py Clearly x86 references the vendor_id for Intel, as you know. The primary issue has never been having vendor identifiers in traits. The primary issue has always been the proposed (ab)use of traits as string categories -- in other words, using traits as "types". That isn't what traits are for. Traits are specifically for boolean values -- capabilities that a provider either has or doesn't have. That is why there's no key/value pairing for traits. There isn't a value. The capability is either available or not available. What you are trying to do is make a key/value pair where the key is "VGPU TYPE" and the value is the vendor's model name or moniker. And that isn't appropriate for traits. The string "M60-0Q" doesn't refer to a single capability. Instead, that string is a moniker that NVIDIA uses to represent a set of capabilities and random requirements together: * a max of 2 vGPU "display heads" * a max resolution of 2560x1600 * 512M framebuffer per vGPU * the host requires a Quadro vDWS license installed * support for the following graphics APIs: * DirectX 12 * Direct2D * DirectX Video Acceleration (DXVA) * OpenGL 4.5 * Vulkan 1.0 * support for the following parallel programming platforms: * OpenCL (<= 2.1 I think?) * CUDA (<=4.0 I think?) It's virtually impossible to tell what the actual capabilities of these vendor monikers are without help from the few people at NVIDIA that actually know these things, partly because the documentation from NVIDIA is so poor (or completely lacking), partly because the installation of the various host and guest drivers is an entirely manual process, partly because NVIDIA and most of the other hardware vendors are more interested in enabling their latest and greatest technology instead of documenting their "old" (read: <6 months ago released) stuff. I agree. However, we need some reference to device types to enable matching the VM driver with the device. Well, no, you don't need to match the device type to the VM driver. You need to match the host (or specific pGPU)'s supported CUDA driver version(s) (NVIDIA calls this "Compute Capability") with the *required minimum CUDA driver version for the guest*. The solution here is to have a big hash table of vendor product name (vGPU type) to sets of standard traits, and have the guest specify CUDA driver version requirements as one or more required=HW_GPU_API_CUDA_XXX extra specs. In other words, we need to break down this "vGPU type" (which even NVIDIA admits is nothing more than a "product profile" of a set of capabilities) into its respective set of standardized os-traits. I've recommended in Sylvain's multi-vgpu-types spec that we put this hash table in nova/virt/vgpu_capabilities.py but if Cyborg needs to use this as well, we could just as easily make it a module in os-traits. This way, when the nova-compute or Cyborg worker starts up, it can query the sysfs mdev_supported_types bucket of randomness, take the values that show up in /sys/class/mdev_bus/$device/mdev_supported_types and look up the actual capabilities that the strings like "nvidia-35" represent. TL;DR We need some reference to device types, but we don't need product names. I will update the spec [1] to clarify that. Rest of this email clarifies why we need device types in traits, and what traits we propose to include. In general, an accelerator device is operated by two pieces of software: a driver in the kernel (which may discover and handle the PF for SR-IOV devices), and a driver/library in the guest (which may handle the assigned VF). The device assigned to the VM must match the driver/library packaged in the VM. For this, the request must explicitly state what category of devices it needs. For example, if the VM needs a GPU, it needs to say whether it needs an AMD GPU or an Nvidia GPU, since it may have the driver/libraries for that vendor alone. Placement's traits and resource classes are absolutely *not* intended to be the vehicle by which guest *configuration details* (like proprietary driver setup and versioning in the guest or license activation, etc) are conveyed to the guest. We already have a vehicle for that: it's called the metadata API and userdata, vendor data, device metadata. Let's limit the traits that are set as r
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
After reading the spec https://review.openstack.org/#/c/554717/14/doc/specs/rocky/cyborg-nova-sched.rst , I confuse on the CUSTOM_ACCELERATOR_FPGA meaning. Initially, I thought it means a region. But after reading the spec, it can be a device, a region or a function. Is it on purpose design? Sounds like we need to have agreement on the naming also. We already have resource class `VGPU`, so we only need to add another resource class 'FPGA'(but same as above question, I thought it should be FPGA_REGION?), is it right? I didn't see any requirement on the prefix 'ACCELERATOR'. 2018-05-31 4:18 GMT+08:00 Eric Fried : > This all sounds fully reasonable to me. One thing, though... > > >> * There is a resource class per device category e.g. > >> CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. > > Let's propose standard resource classes for these ASAP. > > https://github.com/openstack/nova/blob/d741f624c81baf89fc8b6b94a2bc20 > eb5355a818/nova/rc_fields.py > > -efried > . > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
I can help on it. 2018-05-31 21:49 GMT+08:00 Eric Fried : > Yup. I'm sure reviewers will bikeshed the names, but the review is the > appropriate place for that to happen. > > A couple of test changes will also be required. You can have a look at > [1] as an example to follow. > > -efried > > [1] https://review.openstack.org/#/c/511180/ > > On 05/31/2018 01:02 AM, Nadathur, Sundar wrote: > > On 5/30/2018 1:18 PM, Eric Fried wrote: > >> This all sounds fully reasonable to me. One thing, though... > >> > * There is a resource class per device category e.g. > CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. > >> Let's propose standard resource classes for these ASAP. > >> > >> https://github.com/openstack/nova/blob/d741f624c81baf89fc8b6b94a2bc20 > eb5355a818/nova/rc_fields.py > >> > >> > >> -efried > > Makes sense, Eric. The obvious names would be ACCELERATOR_GPU and > > ACCELERATOR_FPGA. Do we just submit a patch to rc_fields.py? > > > > Thanks, > > Sundar > > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: > unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
Yup. I'm sure reviewers will bikeshed the names, but the review is the appropriate place for that to happen. A couple of test changes will also be required. You can have a look at [1] as an example to follow. -efried [1] https://review.openstack.org/#/c/511180/ On 05/31/2018 01:02 AM, Nadathur, Sundar wrote: > On 5/30/2018 1:18 PM, Eric Fried wrote: >> This all sounds fully reasonable to me. One thing, though... >> * There is a resource class per device category e.g. CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. >> Let's propose standard resource classes for these ASAP. >> >> https://github.com/openstack/nova/blob/d741f624c81baf89fc8b6b94a2bc20eb5355a818/nova/rc_fields.py >> >> >> -efried > Makes sense, Eric. The obvious names would be ACCELERATOR_GPU and > ACCELERATOR_FPGA. Do we just submit a patch to rc_fields.py? > > Thanks, > Sundar > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
On 5/30/2018 1:18 PM, Eric Fried wrote: This all sounds fully reasonable to me. One thing, though... * There is a resource class per device category e.g. CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. Let's propose standard resource classes for these ASAP. https://github.com/openstack/nova/blob/d741f624c81baf89fc8b6b94a2bc20eb5355a818/nova/rc_fields.py -efried Makes sense, Eric. The obvious names would be ACCELERATOR_GPU and ACCELERATOR_FPGA. Do we just submit a patch to rc_fields.py? Thanks, Sundar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
This all sounds fully reasonable to me. One thing, though... >> * There is a resource class per device category e.g. >> CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. Let's propose standard resource classes for these ASAP. https://github.com/openstack/nova/blob/d741f624c81baf89fc8b6b94a2bc20eb5355a818/nova/rc_fields.py -efried . __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
Hi Sylvain, Glad to know we are on the same page. I haven't updated the spec with this proposal yet, in case I got more comments :). I will do so by today. Thanks, Sundar On 5/30/2018 12:34 AM, Sylvain Bauza wrote: On Wed, May 30, 2018 at 1:33 AM, Nadathur, Sundar mailto:sundar.nadat...@intel.com>> wrote: Hi all, The Cyborg/Nova scheduling spec [1] details what traits will be applied to the resource providers that represent devices like GPUs. Some of the traits referred to vendor names. I got feedback that traits must not refer to products or specific models of devices. I agree. However, we need some reference to device types to enable matching the VM driver with the device. TL;DR We need some reference to device types, but we don't need product names. I will update the spec [1] to clarify that. Rest of this email clarifies why we need device types in traits, and what traits we propose to include. In general, an accelerator device is operated by two pieces of software: a driver in the kernel (which may discover and handle the PF for SR-IOV devices), and a driver/library in the guest (which may handle the assigned VF). The device assigned to the VM must match the driver/library packaged in the VM. For this, the request must explicitly state what category of devices it needs. For example, if the VM needs a GPU, it needs to say whether it needs an AMD GPU or an Nvidia GPU, since it may have the driver/libraries for that vendor alone. It may also need to state what version of Cuda is needed, if it is a Nvidia GPU. These aspects are necessarily vendor-specific. FWIW, the vGPU implementation for Nova also has the same concern. We want to provide traits for explicitly say "use this vGPU type" but given it's related to a specific vendor, we can't just say "ask for this frame buffer size, or just for the display heads", but rather "we need a vGPU accepting Quadro vDWS license". Further, one driver/library version may handle multiple devices. Since a new driver version may be backwards compatible, multiple driver versions may manage the same device. The development/release of the driver/library inside the VM should be independent of the kernel driver for that device. I agree. For FPGAs, there is an additional twist as the VM may need specific bitstream(s), and they match only specific device/region types. The bitstream for a device from a vendor will not fit any other device from the same vendor, let alone other vendors. IOW, the region type is specific not just to a vendor but to a device type within the vendor. So, it is essential to identify the device type. So, the proposed set of RCs and traits are as below. As we learn more about actual usages by operators, we may need to evolve this set. * There is a resource class per device category e.g. CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. * The resource provider that represents a device has the following traits: o Vendor/Category trait: e.g. CUSTOM_GPU_AMD, CUSTOM_FPGA_XILINX. o Device type trait which is a refinement of vendor/category trait e.g. CUSTOM_FPGA_XILINX_VU9P. NOTE: This is not a product or model, at least for FPGAs. Multiple products may use the same FPGA chip. NOTE: The reason for having both the vendor/category and this one is that a flavor may ask for either, depending on the granularity desired. IOW, if one driver can handle all devices from a vendor (*eye roll*), the flavor can ask for the vendor/category trait alone. If there are separate drivers for different device families from the same vendor, the flavor must specify the trait for the device family. NOTE: The equivalent trait for GPUs may be like CUSTOM_GPU_NVIDIA_P90, but I'll let others decide if that is a product or not. I was about to propose the same for vGPUs in Nova, ie. using custom traits. The only concern is that we need operators to set the traits directly using osc-placement instead of having Nova magically provide those traits. But anyway, given operators need to set the vGPU types they want, I think it's acceptable. o For FPGAs, we have additional traits: + Functionality trait: e.g. CUSTOM_FPGA_COMPUTE, CUSTOM_FPGA_NETWORK, CUSTOM_FPGA_STORAGE + Region type ID. e.g. CUSTOM_FPGA_INTEL_REGION_. + Optionally, a function ID, indicating what function is currently programmed in the region RP. e.g. CUSTOM_FPGA_INTEL_FUNCTION_. Not all implementations may provide it. The function trait may change on reprogramming, but it is not expected to be
Re: [openstack-dev] [Cyborg] [Nova] Cyborg traits
On Wed, May 30, 2018 at 1:33 AM, Nadathur, Sundar wrote: > Hi all, >The Cyborg/Nova scheduling spec [1] details what traits will be applied > to the resource providers that represent devices like GPUs. Some of the > traits referred to vendor names. I got feedback that traits must not refer > to products or specific models of devices. I agree. However, we need some > reference to device types to enable matching the VM driver with the device. > > TL;DR We need some reference to device types, but we don't need product > names. I will update the spec [1] to clarify that. Rest of this email > clarifies why we need device types in traits, and what traits we propose to > include. > > In general, an accelerator device is operated by two pieces of software: a > driver in the kernel (which may discover and handle the PF for SR-IOV > devices), and a driver/library in the guest (which may handle the assigned > VF). > > The device assigned to the VM must match the driver/library packaged in > the VM. For this, the request must explicitly state what category of > devices it needs. For example, if the VM needs a GPU, it needs to say > whether it needs an AMD GPU or an Nvidia GPU, since it may have the > driver/libraries for that vendor alone. It may also need to state what > version of Cuda is needed, if it is a Nvidia GPU. These aspects are > necessarily vendor-specific. > > FWIW, the vGPU implementation for Nova also has the same concern. We want to provide traits for explicitly say "use this vGPU type" but given it's related to a specific vendor, we can't just say "ask for this frame buffer size, or just for the display heads", but rather "we need a vGPU accepting Quadro vDWS license". > Further, one driver/library version may handle multiple devices. Since a > new driver version may be backwards compatible, multiple driver versions > may manage the same device. The development/release of the driver/library > inside the VM should be independent of the kernel driver for that device. > > I agree. > For FPGAs, there is an additional twist as the VM may need specific > bitstream(s), and they match only specific device/region types. The > bitstream for a device from a vendor will not fit any other device from the > same vendor, let alone other vendors. IOW, the region type is specific not > just to a vendor but to a device type within the vendor. So, it is > essential to identify the device type. > > So, the proposed set of RCs and traits are as below. As we learn more > about actual usages by operators, we may need to evolve this set. > >- There is a resource class per device category e.g. >CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. >- The resource provider that represents a device has the following >traits: > - Vendor/Category trait: e.g. CUSTOM_GPU_AMD, CUSTOM_FPGA_XILINX. > - Device type trait which is a refinement of vendor/category trait > e.g. CUSTOM_FPGA_XILINX_VU9P. > > NOTE: This is not a product or model, at least for FPGAs. Multiple > products may use the same FPGA chip. > NOTE: The reason for having both the vendor/category and this one is that > a flavor may ask for either, depending on the granularity desired. IOW, if > one driver can handle all devices from a vendor (*eye roll*), the flavor > can ask for the vendor/category trait alone. If there are separate drivers > for different device families from the same vendor, the flavor must specify > the trait for the device family. > NOTE: The equivalent trait for GPUs may be like CUSTOM_GPU_NVIDIA_P90, but > I'll let others decide if that is a product or not. > > I was about to propose the same for vGPUs in Nova, ie. using custom traits. The only concern is that we need operators to set the traits directly using osc-placement instead of having Nova magically provide those traits. But anyway, given operators need to set the vGPU types they want, I think it's acceptable. > >- For FPGAs, we have additional traits: > - Functionality trait: e.g. CUSTOM_FPGA_COMPUTE, > CUSTOM_FPGA_NETWORK, CUSTOM_FPGA_STORAGE > - Region type ID. e.g. CUSTOM_FPGA_INTEL_REGION_. > - Optionally, a function ID, indicating what function is > currently programmed in the region RP. e.g. > CUSTOM_FPGA_INTEL_FUNCTION_. > Not all implementations may provide it. The function trait may > change on > reprogramming, but it is not expected to be frequent. > - Possibly, CUSTOM_PROGRAMMABLE as a separate trait. > > [1] https://review.openstack.org/#/c/554717/ > I'll try to review the spec as soon as I can. -Sylvain > > > Thanks. > > Regards, > Sundar > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___
[openstack-dev] [Cyborg] [Nova] Cyborg traits
Hi all, The Cyborg/Nova scheduling spec [1] details what traits will be applied to the resource providers that represent devices like GPUs. Some of the traits referred to vendor names. I got feedback that traits must not refer to products or specific models of devices. I agree. However, we need some reference to device types to enable matching the VM driver with the device. TL;DR We need some reference to device types, but we don't need product names. I will update the spec [1] to clarify that. Rest of this email clarifies why we need device types in traits, and what traits we propose to include. In general, an accelerator device is operated by two pieces of software: a driver in the kernel (which may discover and handle the PF for SR-IOV devices), and a driver/library in the guest (which may handle the assigned VF). The device assigned to the VM must match the driver/library packaged in the VM. For this, the request must explicitly state what category of devices it needs. For example, if the VM needs a GPU, it needs to say whether it needs an AMD GPU or an Nvidia GPU, since it may have the driver/libraries for that vendor alone. It may also need to state what version of Cuda is needed, if it is a Nvidia GPU. These aspects are necessarily vendor-specific. Further, one driver/library version may handle multiple devices. Since a new driver version may be backwards compatible, multiple driver versions may manage the same device. The development/release of the driver/library inside the VM should be independent of the kernel driver for that device. For FPGAs, there is an additional twist as the VM may need specific bitstream(s), and they match only specific device/region types. The bitstream for a device from a vendor will not fit any other device from the same vendor, let alone other vendors. IOW, the region type is specific not just to a vendor but to a device type within the vendor. So, it is essential to identify the device type. So, the proposed set of RCs and traits are as below. As we learn more about actual usages by operators, we may need to evolve this set. * There is a resource class per device category e.g. CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA. * The resource provider that represents a device has the following traits: o Vendor/Category trait: e.g. CUSTOM_GPU_AMD, CUSTOM_FPGA_XILINX. o Device type trait which is a refinement of vendor/category trait e.g. CUSTOM_FPGA_XILINX_VU9P. NOTE: This is not a product or model, at least for FPGAs. Multiple products may use the same FPGA chip. NOTE: The reason for having both the vendor/category and this one is that a flavor may ask for either, depending on the granularity desired. IOW, if one driver can handle all devices from a vendor (*eye roll*), the flavor can ask for the vendor/category trait alone. If there are separate drivers for different device families from the same vendor, the flavor must specify the trait for the device family. NOTE: The equivalent trait for GPUs may be like CUSTOM_GPU_NVIDIA_P90, but I'll let others decide if that is a product or not. o For FPGAs, we have additional traits: + Functionality trait: e.g. CUSTOM_FPGA_COMPUTE, CUSTOM_FPGA_NETWORK, CUSTOM_FPGA_STORAGE + Region type ID. e.g. CUSTOM_FPGA_INTEL_REGION_. + Optionally, a function ID, indicating what function is currently programmed in the region RP. e.g. CUSTOM_FPGA_INTEL_FUNCTION_. Not all implementations may provide it. The function trait may change on reprogramming, but it is not expected to be frequent. + Possibly, CUSTOM_PROGRAMMABLE as a separate trait. [1] https://review.openstack.org/#/c/554717/ Thanks. Regards, Sundar __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
