Re: [openstack-dev] [cyborg][glance][nova]cyborg FPGA management flow disscusion.

[Feng, Shaohe]

Hi all.
This is the agent of today’s discussion.
https://etherpad.openstack.org/p/cyborg-nova-poc

BR
Feng, Shaohe

From: Zhipeng Huang [mailto:zhipengh...@gmail.com]
Sent: 2018年3月8日 12:08
To: Feng, Shaohe <shaohe.f...@intel.com>
Cc: openstack-dev@lists.openstack.org; openstack-operat...@lists.openstack.org; 
Du, Dolpher <dolpher...@intel.com>; Ding, Jian-feng <jian-feng.d...@intel.com>; 
Sun, Yih Leong <yih.leong@intel.com>; Nadathur, Sundar 
<sundar.nadat...@intel.com>; Dutch <dalth...@xilinx.com>; Rushil Chugh 
<rushil.ch...@gmail.com>; Nguyen Hung Phuong <phuon...@vn.fujitsu.com>; Justin 
Kilpatrick <jkilp...@redhat.com>; Ranganathan, Shobha 
<shobha.ranganat...@intel.com>; zhuli <zhul...@huawei.com>; 
bao.yum...@zte.com.cn; Li Liu <liliue...@gmail.com>; xiaodong...@tencent.com; 
kong.w...@zte.com.cn; li.xia...@zte.com.cn
Subject: Re: [openstack-dev][cyborg][glance][nova]cyborg FPGA management flow 
disscusion.

Thanks Shaohe,

Let's schedule a video conf session next week.

On Thu, Mar 8, 2018 at 11:41 AM, Feng, Shaohe 
<shaohe.f...@intel.com<mailto:shaohe.f...@intel.com>> wrote:
Hi All:

The POC is here:
https://github.com/shaohef/cyborg

BR
Shaohe Feng

_
From: Feng, Shaohe
Sent: 2018年2月12日 15:06
To: 
openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>; 
openstack-operat...@lists.openstack.org<mailto:openstack-operat...@lists.openstack.org>
Cc: Du, Dolpher <dolpher...@intel.com<mailto:dolpher...@intel.com>>; Zhipeng 
Huang <zhipengh...@gmail.com<mailto:zhipengh...@gmail.com>>; Ding, Jian-feng 
<jian-feng.d...@intel.com<mailto:jian-feng.d...@intel.com>>; Sun, Yih Leong 
<yih.leong@intel.com<mailto:yih.leong@intel.com>>; Nadathur, Sundar 
<sundar.nadat...@intel.com<mailto:sundar.nadat...@intel.com>>; Dutch 
<dalth...@xilinx.com<mailto:dalth...@xilinx.com>>; Rushil Chugh 
<rushil.ch...@gmail.com<mailto:rushil.ch...@gmail.com>>; Nguyen Hung Phuong 
<phuon...@vn.fujitsu.com<mailto:phuon...@vn.fujitsu.com>>; Justin Kilpatrick 
<jkilp...@redhat.com<mailto:jkilp...@redhat.com>>; Ranganathan, Shobha 
<shobha.ranganat...@intel.com<mailto:shobha.ranganat...@intel.com>>; zhuli 
<zhul...@huawei.com<mailto:zhul...@huawei.com>>; 
bao.yum...@zte.com.cn<mailto:bao.yum...@zte.com.cn>; 
xiaodong...@tencent.com<mailto:xiaodong...@tencent.com>; 
kong.w...@zte.com.cn<mailto:kong.w...@zte.com.cn>; 
li.xia...@zte.com.cn<mailto:li.xia...@zte.com.cn>; Feng, Shaohe 
<shaohe.f...@intel.com<mailto:shaohe.f...@intel.com>>
Subject: [openstack-dev][cyborg][glance][nova]cyborg FPGA management flow 
disscusion.


Now I am working on an FPGA management POC with Dolpher.
We have finished some code, and have discussion with Li Liu and some cyborg 
developer guys.

Here are some discussions:

image management
1. User should upload the FPGA image to glance and set the tags as follow:
There are two suggestions to upload an FPGA image.
A. use raw glance api like:
   $ openstack image create --file mypath/FPGA.img  fpga.img
   $ openstack image set --tag FPGA --property vendor=intel --property 
type=crypto 58b813db-1fb7-43ec-b85c-3b771c685d22
   The image must have "FPGA" tag and accelerator type(such as type=crypto).
B. cyborg support a new api to upload a image.
   This API will wrap glance api and include the above steps, also make image 
record in it's local DB.

2. Cyborg agent/conductor get the FPGA image info from glance.
There are also two suggestions to get the FPGA image info.
A. use raw glance api.
Cyborg will get the images by FPGA tag and timestamp periodically and store 
them in it's local cache.
It will use the images tags and properties to form placement taits and 
resource_class name.
B. store the imformations when call cybort's new upload API.

3. Image download.
call glance image download API to local file. and make a corresponding md5 
files for checksum.

GAP in image management：
missing related glance image client in cyborg.

resource report management for scheduler.
1.  Cyborg agent/conductor need synthesize all useful information from FPGA 
driver and image information.
The traits will be like:
CUSTOM_FPGA, CUSTOM_ACCELERATOR_CRYPTO,
The resource_class will be like:
CUSTOM_FPGA_INTEL_PF, CUSTOM_FPGA_INTEL_VF
{"inventories":
"CUSTOM_FPGA_INTEL_PF": {
"allocation_ratio": 1.0,
"max_unit": 4,
"min_unit": 1,
"reserved": 0,
"step_size": 1,
"total": 4
}
}


Accelerator claim and release:
1. Cybort will support the releated API for accelerator claim and release.
It can pass the follow parameters:
  nodename: Which host that accelerator loc

Re: [openstack-dev] [cyborg][glance][nova]cyborg FPGA management flow disscusion.

[Feng, Shaohe]

Hi All:

The POC is here:
https://github.com/shaohef/cyborg

BR
Shaohe Feng

_
From: Feng, Shaohe
Sent: 2018年2月12日 15:06
To: openstack-dev@lists.openstack.org; openstack-operat...@lists.openstack.org
Cc: Du, Dolpher <dolpher...@intel.com>; Zhipeng Huang <zhipengh...@gmail.com>; 
Ding, Jian-feng <jian-feng.d...@intel.com>; Sun, Yih Leong 
<yih.leong@intel.com>; Nadathur, Sundar <sundar.nadat...@intel.com>; Dutch 
<dalth...@xilinx.com>; Rushil Chugh <rushil.ch...@gmail.com>; Nguyen Hung 
Phuong <phuon...@vn.fujitsu.com>; Justin Kilpatrick <jkilp...@redhat.com>; 
Ranganathan, Shobha <shobha.ranganat...@intel.com>; zhuli <zhul...@huawei.com>; 
bao.yum...@zte.com.cn; xiaodong...@tencent.com; kong.w...@zte.com.cn; 
li.xia...@zte.com.cn; Feng, Shaohe <shaohe.f...@intel.com>
Subject: [openstack-dev][cyborg][glance][nova]cyborg FPGA management flow 
disscusion.


Now I am working on an FPGA management POC with Dolpher.
We have finished some code, and have discussion with Li Liu and some cyborg 
developer guys.

Here are some discussions:

image management
1. User should upload the FPGA image to glance and set the tags as follow:
There are two suggestions to upload an FPGA image.
A. use raw glance api like:
   $ openstack image create --file mypath/FPGA.img  fpga.img
   $ openstack image set --tag FPGA --property vendor=intel --property 
type=crypto 58b813db-1fb7-43ec-b85c-3b771c685d22
   The image must have "FPGA" tag and accelerator type(such as type=crypto).
B. cyborg support a new api to upload a image.
   This API will wrap glance api and include the above steps, also make image 
record in it's local DB.

2. Cyborg agent/conductor get the FPGA image info from glance.
There are also two suggestions to get the FPGA image info.
A. use raw glance api.
Cyborg will get the images by FPGA tag and timestamp periodically and store 
them in it's local cache.
It will use the images tags and properties to form placement taits and 
resource_class name.
B. store the imformations when call cybort's new upload API.

3. Image download.
call glance image download API to local file. and make a corresponding md5 
files for checksum.

GAP in image management：
missing related glance image client in cyborg.

resource report management for scheduler.
1.  Cyborg agent/conductor need synthesize all useful information from FPGA 
driver and image information.
The traits will be like:
CUSTOM_FPGA, CUSTOM_ACCELERATOR_CRYPTO,
The resource_class will be like:
CUSTOM_FPGA_INTEL_PF, CUSTOM_FPGA_INTEL_VF
{"inventories":
"CUSTOM_FPGA_INTEL_PF": {
"allocation_ratio": 1.0,
"max_unit": 4,
"min_unit": 1,
"reserved": 0,
"step_size": 1,
"total": 4
}
}


Accelerator claim and release:
1. Cybort will support the releated API for accelerator claim and release.
It can pass the follow parameters:
  nodename: Which host that accelerator located on, it is required.
  type: This accelerator type, cyborg can get image uuid by it. it is optional.
  image uuid: the uuid of FPGA bitstream image, . it is optional.
  traits: the traits info that cyborg reports to placement.
  resource_class: the resource_class name that reports to placement.
And return the address for the accelerator. At present, it is the PCIE_ADDRESS.
2. When claim an accelerator, type and image is None, cybort will not program 
the fpga for user.

FPGA accelerator program API:
We still need to support an independent program API for some specific scenarios.
Such as as a FPGA developer, I will change my verilog logical frequently and 
need to do verification on my guest.
I upload my new bitstream image to glance, and call cyborg to program my FPGA 
accelerator.

End user operations follow:
1. upload an bitstream image to glance if necessary and set its tags(at least 
FPGA is requied) and property.
   sucn as: --tag FPGA --property vendor=intel --property type=crypto
2. list the FPGA related traits and resource_class names by placement API.
   such as get "CUSTOM_FPGA_INTEL_PF" resource_class names and 
"CUSTOM_HW_INTEL,CUSTOM_HW_CRYPTO" traits.
3. create a new falvor wiht his expected traits and resource_class as extra 
spec.
   such as:
   "resourcesn:CUSTOM_FPGA_INTEL_PF=2"  n is an integer or empty string.
   "required:CUSTOM_HW_INTEL,CUSTOM_HW_CRYPTO".
4. create the VM with this flavor.


BR
Shaohe Feng


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cyborg][glance][nova]cyborg FPGA management flow disscusion.

[Feng, Shaohe]

Now I am working on an FPGA management POC with Dolpher.
We have finished some code, and have discussion with Li Liu and some cyborg 
developer guys.

Here are some discussions:

image management
1. User should upload the FPGA image to glance and set the tags as follow:
There are two suggestions to upload an FPGA image.
A. use raw glance api like:
   $ openstack image create --file mypath/FPGA.img  fpga.img
   $ openstack image set --tag FPGA --property vendor=intel --property 
type=crypto 58b813db-1fb7-43ec-b85c-3b771c685d22
   The image must have "FPGA" tag and accelerator type(such as type=crypto).
B. cyborg support a new api to upload a image.
   This API will wrap glance api and include the above steps, also make image 
record in it's local DB.

2. Cyborg agent/conductor get the FPGA image info from glance.
There are also two suggestions to get the FPGA image info.
A. use raw glance api.
Cyborg will get the images by FPGA tag and timestamp periodically and store 
them in it's local cache.
It will use the images tags and properties to form placement taits and 
resource_class name.
B. store the imformations when call cybort's new upload API.

3. Image download.
call glance image download API to local file. and make a corresponding md5 
files for checksum.

GAP in image management：
missing related glance image client in cyborg.

resource report management for scheduler.
1.  Cyborg agent/conductor need synthesize all useful information from FPGA 
driver and image information.
The traits will be like:
CUSTOM_FPGA, CUSTOM_ACCELERATOR_CRYPTO,
The resource_class will be like:
CUSTOM_FPGA_INTEL_PF, CUSTOM_FPGA_INTEL_VF
{"inventories":
"CUSTOM_FPGA_INTEL_PF": {
"allocation_ratio": 1.0,
"max_unit": 4,
"min_unit": 1,
"reserved": 0,
"step_size": 1,
"total": 4
}
}


Accelerator claim and release:
1. Cybort will support the releated API for accelerator claim and release.
It can pass the follow parameters:
  nodename: Which host that accelerator located on, it is required.
  type: This accelerator type, cyborg can get image uuid by it. it is optional.
  image uuid: the uuid of FPGA bitstream image, . it is optional.
  traits: the traits info that cyborg reports to placement.
  resource_class: the resource_class name that reports to placement.
And return the address for the accelerator. At present, it is the PCIE_ADDRESS.
2. When claim an accelerator, type and image is None, cybort will not program 
the fpga for user.

FPGA accelerator program API:
We still need to support an independent program API for some specific scenarios.
Such as as a FPGA developer, I will change my verilog logical frequently and 
need to do verification on my guest.
I upload my new bitstream image to glance, and call cyborg to program my FPGA 
accelerator.

End user operations follow:
1. upload an bitstream image to glance if necessary and set its tags(at least 
FPGA is requied) and property.
   sucn as: --tag FPGA --property vendor=intel --property type=crypto
2. list the FPGA related traits and resource_class names by placement API.
   such as get "CUSTOM_FPGA_INTEL_PF" resource_class names and 
"CUSTOM_HW_INTEL,CUSTOM_HW_CRYPTO" traits.
3. create a new falvor wiht his expected traits and resource_class as extra 
spec.
   such as:
   "resourcesn:CUSTOM_FPGA_INTEL_PF=2"  n is an integer or empty string.
   "required:CUSTOM_HW_INTEL,CUSTOM_HW_CRYPTO".
4. create the VM with this flavor.


BR
Shaohe Feng


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cyborg] minutes of FPGA introduction meeting

[Feng, Shaohe]

Date: 2017-12-12


Shaohe and Dolpher gave an introduction about FPGA.

They also do simple POC for FPGA base on Placement schedule and He 
Yongli's PCI management mechanism in nova.


1.  We define a new resource class CUSTOM_FPGA_INTEL_ECHO for a kind of 
FPGA accelerator which is ECHO.


The Cyborg need to implement an interface to get the inventory, then to 
update the provider inventory of placement whit it.


The Cyborg API need to get the inventory data by its FPGA agent running 
on every node.


2.  We need to extend related PCI info of CUSTOM_FPGA_INTEL_ECHO in the 
PCI whitelist, so this kind of PCI device(CUSTOM_FPGA_INTEL_ECHO) can be 
allocated.


Cyborg needs to support to get the related PCI info.

3. we define a new  property "resources:CUSTOM_FPGA_INTEL_ECHO='1' " in 
flavor.


Cyborg needs to translate abstract property to concrete device with PCI 
spec.



The flow is:

Create a VM with a flavor which has a FPGA accelerator resource in it's 
property.


The scheduler will schedule a host with this kind of FPGA accelerator 
base on the placement provider information.  and claim an abstract 
accelerator.


The nova-cpu will claim an related concrete PCI devices from the 
abstract accelerator information.


The driver will start a VM with the PCI devices.


Open

1. Should Cybory will leverage Yongli's PCI management mechanism in 
nova?  or it will maintain a new PCI management mechanism for PCI device?


2. beside, for VGPU accelerator, it is not a PCI device, it is a mdev 
device.


Should Cybory maintain this new kind  device mechanism?

3. The FPGA can be programmed. So should the Cyborg  update the 
whitelist on the fly?



NOTE: the POC does not consider the how to program FPGA.

AR:  Zhu Li  will introduce how to cyborg FPGA model and agent.


BR

Shaohe Feng


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mogan] Nominating liusheng for Mogan core

[Feng, Shaohe]

Liusheng deserved it for his great contribution.

＋１
Thanks.

BR
Ｓｈａｏｈｅ


On Mon, Mar 20, 2017 at 4:19 PM, Zhenguo Niu > wrote:


   Hi team,

   I would like to nominate liusheng to Mogan core. Liusheng has been a
   significant code contributor since the project's creation providing
   high quality reviews.

   Please feel free to respond in public or private your support or any
   concerns.


   Thanks,
   Zhenguo

   __
   OpenStack Development Mailing List (not for usage questions)
   Unsubscribe:
   openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
   
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
   



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev