Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Fei K Chen 

Hi all,

We agree this is an ingesting direction we can do exploration on OpenStack
and actually we already have such FPGA/GPU as a resource in our public
research cloud called SuperVessel (https://ptopenlab.com). With
modification of OpenStack from community, nearly all features mentioned by
Roman Dobosz are supported, such as FPGA boards detection, scheduling,
images management and user API layer. Here are some materials for you to
reference [1][2].

Unlike CPU and GPU, right now FPGA accelerator is developed without common
protocol above PCIe protocol. Along with huge kinds of FPGA chips, FPGA
boards, it is hard to be managed. Passing through the FPGA as a general
PCIe device to the application stack is a straight way to expose the FPGA
acceleration ability in cloud while it is far way to make it easy used. It
dose not help the accelerator detection, reload, reconfiguration,
deployment in cloud.

Good news is that, the FPGA community is actively promoting high level
programming language, such as OpenCL. They hide nearly all hardware detail
for the developers so it make the "accelerator" common in different
hardware. It is reasonable to see the FPGA become a popular acceleration
hardware. It also reasonable for OpenStack community begin the discussion
to support the FPGA resource management.


[1]
http://openpowerfoundation.org/blogs/fpga-acceleration-in-a-power8-cloud/
[2] http://dl.acm.org/citation.cfm?id=2597929



Best regards,
CHEN Fei





From:   Qiming Teng <teng...@linux.vnet.ibm.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Cc: Fei K Chen/China/IBM@IBMCN, Yong Hua Lin/China/IBM@IBMCN
Date:   2016/04/06 12:25
Subject:    Re: [openstack-dev] [Nova] FPGA as a resource



Emm... finally this is brought up. We from IBM have already done some
work on FPGA/GPU resource management [1]. Let me bring the SMEs into
this discussion and see if we together can work out a concrete roadmap
to land this upstream.

Fei and Yonghua, this is indeed very interesting a topic for us.


[1] SuperVessel Cloud: https://ptopenlab.com/

Regards,
  Qiming

On Tue, Apr 05, 2016 at 02:27:30PM +0200, Roman Dobosz wrote:
> Hey all,
>
> On yesterday's scheduler meeting I was raised the idea of bringing up
> the FPGA to the OpenStack as the resource, which than might be exposed
> to the VMs.
>
> The use cases for motivations, why one want do this, are pretty broad -
> having such chip ready on the computes might be beneficial either for
> consumers of the technology and data center administrators. The
> utilization of the hardware is very broad - the only limitations are
> human imagination and hardware capability - since it might be used for
> accelerating execution of algorithms from compression and cryptography,
> through pattern recognition, transcoding to voice/video analysis and
> processing and all the others in between. Using FPGA to perform data
> processing may significantly reduce CPU utilization, the time and power
> consumption, which is a benefit on its own.
>
> On OpenStack side, unlike utilizing the CPU or memory, for actually
> using specific algorithm with FPGAs, it has to be programmed first. So
> in simplified scenario, it might go like this:
>
> * User selects VM with image which supports acceleration,
> * Scheduler selects the appropriate compute host with FPGA available,
> * Compute gets request, program IP into FPGA and then boot up the
>   VM with accelerator attached.
> * If VM is removed, it may optionally erase the FPGA.
>
> As you can see, it seems not complicated at this point, however it
> become more complex due to following things we also have to take into
> consideration:
>
> * recent FPGA are divided into regions (or slots) and every of them
>   can be programmed separately
> * slots may or may not fit the same bitstream (the program which FPGA
>   is fed, the IP)
> * There is several products around (Altera, Xilinx, others), which make
>   bitstream incompatible. Even between the products of the same company
> * libraries which abstract the hardware layer like AAL[1] and their
>   versions
> * for some products, there is a need for tracking memory usage, which
>   is located on PCI boards
> * some of the FPGAs can be exposed using SR-IOV, while some other not,
>   which implies multiple usage abilities
>
> In other words, it may be necessary to incorporate another actions:
>
> * properly discover FPGA and its capabilities
> * schedule right bitstream with corresponding matching unoccupied FPGA
>   device/slot
> * actual program FPGA
> * provide libraries into VM, which are necessary for interacting between
>   user program and the exposed FPGA (or AAL) (this may be optional,
>   since user can upload complete ima

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Rayson Ho 
On Wed, Apr 6, 2016 at 6:37 AM, Daniel P. Berrange 
wrote:
> Most of the
> work to support FPGA will be internal to nova, to deal with modelling
> of assignable devices and their scheduling / allocation.

I think the EPA blueprint already covers some of the FPGA device discovery
& scheduling functionality needed.

https://wiki.openstack.org/wiki/Enhanced-platform-awareness-pcie


However, I was reading the Intel ONP whitepapers earlier, and found that
Intel has dropped coverage of the EPA functionality in 2.1:

 ->
https://download.01.org/packet-processing/ONPS2.1/Intel_ONP_Release_2.1_Reference_Architecture_Guide_Rev1.0.pdf

EPA is mentioned in section "6.3 Enhanced Platform Awareness" in 2.0:

 ->
https://download.01.org/packet-processing/ONPS2.0/Intel_ONP_Release_2.0_Reference_Architecture_Guide_Rev1.0-1.pdf

Rayson

==
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html




>
> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
:|
> |: http://libvirt.org  -o- http://virt-manager.org
:|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/
:|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc
:|
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Zhipeng Huang 
Yes the project is quite new, we are still developing requirements at the
OPNFV side. We will update with full description after the summit when we
got all the inputs. Look forward to have a chat with you at the summit :)

I think you are right that Nomad does not solve the problem per se in this
thread, however it would be the natural next step. After having Nova being
able to use FPGA as a resource, you will need to be able schedule FPGA
resources for the VM.

We also have a team based in Canada that specialized in FPGA related
management that will participate in the Nomad development, we could have
more deep dive chat when we got chance :)

On Wed, Apr 6, 2016 at 6:28 PM, Roman Dobosz  wrote:

> On Wed, 6 Apr 2016 16:12:16 +0800
> Zhipeng Huang  wrote:
>
> > You are actually touching on something we have been working on. There is
> a
> > team in OPNFV DPACC project has been working acceleration related topics,
> > including folks from CMCC, Intel, ARM, Freescale, Huawei. We found out
> that
> > in order to have acceleration working under NFV scenrios, other than Nova
> > and Neutron's support, we also need a standalone service that manage
> > accelerators itself.
> >
> > That means we want to treat accelerators, and FPGA being an important
> part
> > of it, as a first class resource citizen and we want to be able to do
> life
> > cycle management and scheduling on acceleration resources.
> >
> > Based upon that requirement we started a new project called Nomad [1] on
> > Jan this year, to serve as an OpenStack service for distributed
> > acceleration management.
> >
> > We've just started the project, and currently discussing the first BP
> [2].
> > We have a team working on IP-SEC based accelerator mgmt, and would love
> to
> > have more people to work on topics like FPGA.
> >
> > We also have a topic on introducing Nomad accepted in Austin Summit [3].
> >
> > You are more than welcomed to join the conversation : )
>
> Thanks! I'll try to attend.
>
> Nevertheless, I've briefly looked at page of project Nomad, and don't
> quite get it, how this might be related to cases described in this
> thread - i.e. providing attachable, non trivial devices as a new
> resources in Nova.
>
> --
> Cheers,
> Roman Dobosz
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Zhipeng (Howard) Huang

Standard Engineer
IT Standard & Patent/IT Prooduct Line
Huawei Technologies Co,. Ltd
Email: huangzhip...@huawei.com
Office: Huawei Industrial Base, Longgang, Shenzhen

(Previous)
Research Assistant
Mobile Ad-Hoc Network Lab, Calit2
University of California, Irvine
Email: zhipe...@uci.edu
Office: Calit2 Building Room 2402

OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Daniel P. Berrange 
On Wed, Apr 06, 2016 at 12:28:02PM +0200, Roman Dobosz wrote:
> On Wed, 6 Apr 2016 16:12:16 +0800
> Zhipeng Huang  wrote:
> 
> > You are actually touching on something we have been working on. There is a
> > team in OPNFV DPACC project has been working acceleration related topics,
> > including folks from CMCC, Intel, ARM, Freescale, Huawei. We found out that
> > in order to have acceleration working under NFV scenrios, other than Nova
> > and Neutron's support, we also need a standalone service that manage
> > accelerators itself.
> > 
> > That means we want to treat accelerators, and FPGA being an important part
> > of it, as a first class resource citizen and we want to be able to do life
> > cycle management and scheduling on acceleration resources.
> > 
> > Based upon that requirement we started a new project called Nomad [1] on
> > Jan this year, to serve as an OpenStack service for distributed
> > acceleration management.
> > 
> > We've just started the project, and currently discussing the first BP [2].
> > We have a team working on IP-SEC based accelerator mgmt, and would love to
> > have more people to work on topics like FPGA.
> > 
> > We also have a topic on introducing Nomad accepted in Austin Summit [3].
> > 
> > You are more than welcomed to join the conversation : )
> 
> Thanks! I'll try to attend.
> 
> Nevertheless, I've briefly looked at page of project Nomad, and don't
> quite get it, how this might be related to cases described in this
> thread - i.e. providing attachable, non trivial devices as a new
> resources in Nova.

I don't think it is really relevant in the immediate term. Most of the
work to support FPGA will be internal to nova, to deal with modelling
of assignable devices and their scheduling / allocation.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Roman Dobosz 
On Wed, 6 Apr 2016 16:12:16 +0800
Zhipeng Huang  wrote:

> You are actually touching on something we have been working on. There is a
> team in OPNFV DPACC project has been working acceleration related topics,
> including folks from CMCC, Intel, ARM, Freescale, Huawei. We found out that
> in order to have acceleration working under NFV scenrios, other than Nova
> and Neutron's support, we also need a standalone service that manage
> accelerators itself.
> 
> That means we want to treat accelerators, and FPGA being an important part
> of it, as a first class resource citizen and we want to be able to do life
> cycle management and scheduling on acceleration resources.
> 
> Based upon that requirement we started a new project called Nomad [1] on
> Jan this year, to serve as an OpenStack service for distributed
> acceleration management.
> 
> We've just started the project, and currently discussing the first BP [2].
> We have a team working on IP-SEC based accelerator mgmt, and would love to
> have more people to work on topics like FPGA.
> 
> We also have a topic on introducing Nomad accepted in Austin Summit [3].
> 
> You are more than welcomed to join the conversation : )

Thanks! I'll try to attend.

Nevertheless, I've briefly looked at page of project Nomad, and don't
quite get it, how this might be related to cases described in this
thread - i.e. providing attachable, non trivial devices as a new
resources in Nova.

-- 
Cheers,
Roman Dobosz

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Daniel P. Berrange 
On Wed, Apr 06, 2016 at 07:34:46AM +0200, Roman Dobosz wrote:
> On Tue, 5 Apr 2016 13:58:44 +0100
> "Daniel P. Berrange"  wrote:
> 
> > Along similar lines we have proposals to add vGPU support to Nova,
> > where the vGPUs may or may not be exposed using SR-IOV. We also want
> > to be able to on the fly decide whether any physical GPU is assigned
> > entirely to a guest as a full PCI device, or whether we only assign
> > individual "virtual functions" of the GPU. This means that even if
> > the GPU in question does *not* use SR-IOV, we still need to track
> > the GPU and vGPUs in the same way as we track PCI devices, so that
> > we can avoid assigning a vGPU to the guest, if the underlying physical
> > PCI device is already assigned to the guest.
> 
> That's correct. I'd like to mention, that FPGAs can be also exposed
> other way than PCI (like in Xeon+FPGA). Not sure if this also applies
> to GPU.
> 
> > I can see we will have much the same issue with FPGAs, where we may
> > either want to assign the entire physical PCI device to a guest, or
> > just assign a particular slot in the FPGA to the guest. So even if
> > the FPGA is not using SR-IOV, we need to tie this all into the PCI
> > device tracking code, as we are intending for vGPUs.
> > 
> > All in all, I think we probably ought to generalize the PCI device
> > assignment modelling so that we're actually modelling generic
> > hardware devices which may or may not be PCI based, so that we can
> > accurately track the relationships between the devices.
> > 
> > With NIC devices we're also seeing a need to expose capabilities
> > against the PCI devices, so that the schedular can be more selective
> > in deciding which particular devices to assign. eg so we can distinguish
> > between NICs which support RDMA and those which don't, or identify NIC
> > with hardware offload features, and so on. I can see this need to
> > associate capabilities with devices is something that will likely
> > be needed for the FPGA scenario, and vGPUs too. So again this points
> > towards more general purpose modelling of assignable hardware devices
> > beyond the limited PCI device modelling we've got today.
> > 
> > Looking to the future I think we'll see more usecases for device
> > assignment appearing for other types of device.
> > 
> > IOW, I think it would be a mistake to model FPGAs as a distinct
> > object type on their own. Generalization of assignable devices
> > is the way to go
> 
> That's why I've bring the topic here on the list, so we can think about
> similar devices which could be generalized into one common accelerator
> type or even think about modeling PCI as such.

I wouldn't specialize it to "accelerators" as we'll inevitably come
across a need for other types of device assignment. We should just
generalize it to track *any* type of host hardware device that is
potentially assignable.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-06 Thread Zhipeng Huang 
Hi Roman,

You are actually touching on something we have been working on. There is a
team in OPNFV DPACC project has been working acceleration related topics,
including folks from CMCC, Intel, ARM, Freescale, Huawei. We found out that
in order to have acceleration working under NFV scenrios, other than Nova
and Neutron's support, we also need a standalone service that manage
accelerators itself.

That means we want to treat accelerators, and FPGA being an important part
of it, as a first class resource citizen and we want to be able to do life
cycle management and scheduling on acceleration resources.

Based upon that requirement we started a new project called Nomad [1] on
Jan this year, to serve as an OpenStack service for distributed
acceleration management.

We've just started the project, and currently discussing the first BP [2].
We have a team working on IP-SEC based accelerator mgmt, and would love to
have more people to work on topics like FPGA.

We also have a topic on introducing Nomad accepted in Austin Summit [3].

You are more than welcomed to join the conversation : )

[1] https://wiki.openstack.org/wiki/Nomad
[2] https://review.openstack.org/#/c/284304/
[3]
https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=nomad



On Wed, Apr 6, 2016 at 1:34 PM, Roman Dobosz  wrote:

> On Tue, 5 Apr 2016 13:58:44 +0100
> "Daniel P. Berrange"  wrote:
>
> > Along similar lines we have proposals to add vGPU support to Nova,
> > where the vGPUs may or may not be exposed using SR-IOV. We also want
> > to be able to on the fly decide whether any physical GPU is assigned
> > entirely to a guest as a full PCI device, or whether we only assign
> > individual "virtual functions" of the GPU. This means that even if
> > the GPU in question does *not* use SR-IOV, we still need to track
> > the GPU and vGPUs in the same way as we track PCI devices, so that
> > we can avoid assigning a vGPU to the guest, if the underlying physical
> > PCI device is already assigned to the guest.
>
> That's correct. I'd like to mention, that FPGAs can be also exposed
> other way than PCI (like in Xeon+FPGA). Not sure if this also applies
> to GPU.
>
> > I can see we will have much the same issue with FPGAs, where we may
> > either want to assign the entire physical PCI device to a guest, or
> > just assign a particular slot in the FPGA to the guest. So even if
> > the FPGA is not using SR-IOV, we need to tie this all into the PCI
> > device tracking code, as we are intending for vGPUs.
> >
> > All in all, I think we probably ought to generalize the PCI device
> > assignment modelling so that we're actually modelling generic
> > hardware devices which may or may not be PCI based, so that we can
> > accurately track the relationships between the devices.
> >
> > With NIC devices we're also seeing a need to expose capabilities
> > against the PCI devices, so that the schedular can be more selective
> > in deciding which particular devices to assign. eg so we can distinguish
> > between NICs which support RDMA and those which don't, or identify NIC
> > with hardware offload features, and so on. I can see this need to
> > associate capabilities with devices is something that will likely
> > be needed for the FPGA scenario, and vGPUs too. So again this points
> > towards more general purpose modelling of assignable hardware devices
> > beyond the limited PCI device modelling we've got today.
> >
> > Looking to the future I think we'll see more usecases for device
> > assignment appearing for other types of device.
> >
> > IOW, I think it would be a mistake to model FPGAs as a distinct
> > object type on their own. Generalization of assignable devices
> > is the way to go
>
> That's why I've bring the topic here on the list, so we can think about
> similar devices which could be generalized into one common accelerator
> type or even think about modeling PCI as such.
>
> > > All of that makes modelling resource extremely complicated, contrary to
> > > CPU resource for example. I'd like to discuss how the goal of having
> > > reprogrammable accelerators in OpenStack can be achieved. Ideally I'd
> > > like to fit it into Jay and Chris work on resource-*.
> > I think you shouldn't look at the FPGAs as being like CPU resource, but
> > rather look at them as a generalization of PCI device asignment.
>
> CPU in this context was only an example of "easy" resource, which
> doesn't need any preparation before VM can use it :)
>
> --
> Cheers,
> Roman Dobosz
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Zhipeng (Howard) Huang

Standard Engineer
IT Standard & Patent/IT Prooduct Line
Huawei Technologies Co,. Ltd
Email:

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-05 Thread Roman Dobosz 
On Tue, 5 Apr 2016 13:58:44 +0100
"Daniel P. Berrange"  wrote:

> Along similar lines we have proposals to add vGPU support to Nova,
> where the vGPUs may or may not be exposed using SR-IOV. We also want
> to be able to on the fly decide whether any physical GPU is assigned
> entirely to a guest as a full PCI device, or whether we only assign
> individual "virtual functions" of the GPU. This means that even if
> the GPU in question does *not* use SR-IOV, we still need to track
> the GPU and vGPUs in the same way as we track PCI devices, so that
> we can avoid assigning a vGPU to the guest, if the underlying physical
> PCI device is already assigned to the guest.

That's correct. I'd like to mention, that FPGAs can be also exposed
other way than PCI (like in Xeon+FPGA). Not sure if this also applies
to GPU.

> I can see we will have much the same issue with FPGAs, where we may
> either want to assign the entire physical PCI device to a guest, or
> just assign a particular slot in the FPGA to the guest. So even if
> the FPGA is not using SR-IOV, we need to tie this all into the PCI
> device tracking code, as we are intending for vGPUs.
> 
> All in all, I think we probably ought to generalize the PCI device
> assignment modelling so that we're actually modelling generic
> hardware devices which may or may not be PCI based, so that we can
> accurately track the relationships between the devices.
> 
> With NIC devices we're also seeing a need to expose capabilities
> against the PCI devices, so that the schedular can be more selective
> in deciding which particular devices to assign. eg so we can distinguish
> between NICs which support RDMA and those which don't, or identify NIC
> with hardware offload features, and so on. I can see this need to
> associate capabilities with devices is something that will likely
> be needed for the FPGA scenario, and vGPUs too. So again this points
> towards more general purpose modelling of assignable hardware devices
> beyond the limited PCI device modelling we've got today.
> 
> Looking to the future I think we'll see more usecases for device
> assignment appearing for other types of device.
> 
> IOW, I think it would be a mistake to model FPGAs as a distinct
> object type on their own. Generalization of assignable devices
> is the way to go

That's why I've bring the topic here on the list, so we can think about
similar devices which could be generalized into one common accelerator
type or even think about modeling PCI as such.

> > All of that makes modelling resource extremely complicated, contrary to 
> > CPU resource for example. I'd like to discuss how the goal of having 
> > reprogrammable accelerators in OpenStack can be achieved. Ideally I'd 
> > like to fit it into Jay and Chris work on resource-*.
> I think you shouldn't look at the FPGAs as being like CPU resource, but
> rather look at them as a generalization of PCI device asignment.

CPU in this context was only an example of "easy" resource, which
doesn't need any preparation before VM can use it :)

-- 
Cheers,
Roman Dobosz

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-05 Thread Gerard Braad 
Hi,

On Tue, Apr 5, 2016 at 8:58 PM, Daniel P. Berrange  wrote:
>> As you can see, it seems not complicated at this point, however it
>> become more complex due to following things we also have to take into
>> consideration:

I agree with this. Many kinds of resources could be considered, not
just to augment processing, but also to provide input (or other forms
of output), such as video devices, etc.

regards,

Gerard

-- 
Gerard Braad β€” 吉拉德
   F/OSS & IT Consultant in Beijing
   http://gbraad.nl  gpg: 0x592CFE75

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-05 Thread Qiming Teng 
Emm... finally this is brought up. We from IBM have already done some
work on FPGA/GPU resource management [1]. Let me bring the SMEs into
this discussion and see if we together can work out a concrete roadmap
to land this upstream.

Fei and Yonghua, this is indeed very interesting a topic for us.


[1] SuperVessel Cloud: https://ptopenlab.com/

Regards,
  Qiming

On Tue, Apr 05, 2016 at 02:27:30PM +0200, Roman Dobosz wrote:
> Hey all,
> 
> On yesterday's scheduler meeting I was raised the idea of bringing up 
> the FPGA to the OpenStack as the resource, which than might be exposed 
> to the VMs.
> 
> The use cases for motivations, why one want do this, are pretty broad -
> having such chip ready on the computes might be beneficial either for
> consumers of the technology and data center administrators. The
> utilization of the hardware is very broad - the only limitations are
> human imagination and hardware capability - since it might be used for
> accelerating execution of algorithms from compression and cryptography,
> through pattern recognition, transcoding to voice/video analysis and
> processing and all the others in between. Using FPGA to perform data
> processing may significantly reduce CPU utilization, the time and power
> consumption, which is a benefit on its own.
> 
> On OpenStack side, unlike utilizing the CPU or memory, for actually 
> using specific algorithm with FPGAs, it has to be programmed first. So 
> in simplified scenario, it might go like this:
> 
> * User selects VM with image which supports acceleration,
> * Scheduler selects the appropriate compute host with FPGA available,
> * Compute gets request, program IP into FPGA and then boot up the 
>   VM with accelerator attached.
> * If VM is removed, it may optionally erase the FPGA.
> 
> As you can see, it seems not complicated at this point, however it 
> become more complex due to following things we also have to take into 
> consideration:
> 
> * recent FPGA are divided into regions (or slots) and every of them 
>   can be programmed separately
> * slots may or may not fit the same bitstream (the program which FPGA
>   is fed, the IP)
> * There is several products around (Altera, Xilinx, others), which make 
>   bitstream incompatible. Even between the products of the same company
> * libraries which abstract the hardware layer like AAL[1] and their 
>   versions
> * for some products, there is a need for tracking memory usage, which 
>   is located on PCI boards
> * some of the FPGAs can be exposed using SR-IOV, while some other not, 
>   which implies multiple usage abilities
> 
> In other words, it may be necessary to incorporate another actions:
> 
> * properly discover FPGA and its capabilities
> * schedule right bitstream with corresponding matching unoccupied FPGA
>   device/slot
> * actual program FPGA
> * provide libraries into VM, which are necessary for interacting between
>   user program and the exposed FPGA (or AAL) (this may be optional, 
>   since user can upload complete image with everything in place)
> * bitstream images have to be keep in some kind of service (Glance?) 
>   with some kind of way for identifying which image match what FPGA
> 
> All of that makes modelling resource extremely complicated, contrary to 
> CPU resource for example. I'd like to discuss how the goal of having 
> reprogrammable accelerators in OpenStack can be achieved. Ideally I'd 
> like to fit it into Jay and Chris work on resource-*.
> 
> Looking forward for any comments :)
> 
> [1] 
> http://www.intel.com/content/dam/doc/white-paper/quickassist-technology-aal-white-paper.pdf
> 
> -- 
> Cheers,
> Roman Dobosz
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FPGA as a resource

2016-04-05 Thread Daniel P. Berrange 
On Tue, Apr 05, 2016 at 02:27:30PM +0200, Roman Dobosz wrote:
> Hey all,
> 
> On yesterday's scheduler meeting I was raised the idea of bringing up 
> the FPGA to the OpenStack as the resource, which than might be exposed 
> to the VMs.
> 
> The use cases for motivations, why one want do this, are pretty broad -
> having such chip ready on the computes might be beneficial either for
> consumers of the technology and data center administrators. The
> utilization of the hardware is very broad - the only limitations are
> human imagination and hardware capability - since it might be used for
> accelerating execution of algorithms from compression and cryptography,
> through pattern recognition, transcoding to voice/video analysis and
> processing and all the others in between. Using FPGA to perform data
> processing may significantly reduce CPU utilization, the time and power
> consumption, which is a benefit on its own.
> 
> On OpenStack side, unlike utilizing the CPU or memory, for actually 
> using specific algorithm with FPGAs, it has to be programmed first. So 
> in simplified scenario, it might go like this:
> 
> * User selects VM with image which supports acceleration,
> * Scheduler selects the appropriate compute host with FPGA available,
> * Compute gets request, program IP into FPGA and then boot up the 
>   VM with accelerator attached.
> * If VM is removed, it may optionally erase the FPGA.
> 
> As you can see, it seems not complicated at this point, however it 
> become more complex due to following things we also have to take into 
> consideration:
> 
> * recent FPGA are divided into regions (or slots) and every of them 
>   can be programmed separately
> * slots may or may not fit the same bitstream (the program which FPGA
>   is fed, the IP)
> * There is several products around (Altera, Xilinx, others), which make 
>   bitstream incompatible. Even between the products of the same company
> * libraries which abstract the hardware layer like AAL[1] and their 
>   versions
> * for some products, there is a need for tracking memory usage, which 
>   is located on PCI boards
> * some of the FPGAs can be exposed using SR-IOV, while some other not, 
>   which implies multiple usage abilities

Along similar lines we have proposals to add vGPU support to Nova,
where the vGPUs may or may not be exposed using SR-IOV. We also want
to be able to on the fly decide whether any physical GPU is assigned
entirely to a guest as a full PCI device, or whether we only assign
individual "virtual functions" of the GPU. This means that even if
the GPU in question does *not* use SR-IOV, we still need to track
the GPU and vGPUs in the same way as we track PCI devices, so that
we can avoid assigning a vGPU to the guest, if the underlying physical
PCI device is already assigned to the guest.

I can see we will have much the same issue with FPGAs, where we may
either want to assign the entire physical PCI device to a guest, or
just assign a particular slot in the FPGA to the guest. So even if
the FPGA is not using SR-IOV, we need to tie this all into the PCI
device tracking code, as we are intending for vGPUs.

All in all, I think we probably ought to generalize the PCI device
assignment modelling so that we're actually modelling generic
hardware devices which may or may not be PCI based, so that we can
accurately track the relationships between the devices.

With NIC devices we're also seeing a need to expose capabilities
against the PCI devices, so that the schedular can be more selective
in deciding which particular devices to assign. eg so we can distinguish
between NICs which support RDMA and those which don't, or identify NIC
with hardware offload features, and so on. I can see this need to
associate capabilities with devices is something that will likely
be needed for the FPGA scenario, and vGPUs too. So again this points
towards more general purpose modelling of assignable hardware devices
beyond the limited PCI device modelling we've got today.

Looking to the future I think we'll see more usecases for device
assignment appearing for other types of device.

IOW, I think it would be a mistake to model FPGAs as a distinct
object type on their own. Generalization of assignable devices
is the way to go

> In other words, it may be necessary to incorporate another actions:
> 
> * properly discover FPGA and its capabilities
> * schedule right bitstream with corresponding matching unoccupied FPGA
>   device/slot
> * actual program FPGA
> * provide libraries into VM, which are necessary for interacting between
>   user program and the exposed FPGA (or AAL) (this may be optional, 
>   since user can upload complete image with everything in place)
> * bitstream images have to be keep in some kind of service (Glance?) 
>   with some kind of way for identifying which image match what FPGA
> 
> All of that makes modelling resource extremely complicated, contrary to 
> CPU resource for example. I'd like to discuss how the goal

[openstack-dev] [Nova] FPGA as a resource

2016-04-05 Thread Roman Dobosz 
Hey all,

On yesterday's scheduler meeting I was raised the idea of bringing up 
the FPGA to the OpenStack as the resource, which than might be exposed 
to the VMs.

The use cases for motivations, why one want do this, are pretty broad -
having such chip ready on the computes might be beneficial either for
consumers of the technology and data center administrators. The
utilization of the hardware is very broad - the only limitations are
human imagination and hardware capability - since it might be used for
accelerating execution of algorithms from compression and cryptography,
through pattern recognition, transcoding to voice/video analysis and
processing and all the others in between. Using FPGA to perform data
processing may significantly reduce CPU utilization, the time and power
consumption, which is a benefit on its own.

On OpenStack side, unlike utilizing the CPU or memory, for actually 
using specific algorithm with FPGAs, it has to be programmed first. So 
in simplified scenario, it might go like this:

* User selects VM with image which supports acceleration,
* Scheduler selects the appropriate compute host with FPGA available,
* Compute gets request, program IP into FPGA and then boot up the 
  VM with accelerator attached.
* If VM is removed, it may optionally erase the FPGA.

As you can see, it seems not complicated at this point, however it 
become more complex due to following things we also have to take into 
consideration:

* recent FPGA are divided into regions (or slots) and every of them 
  can be programmed separately
* slots may or may not fit the same bitstream (the program which FPGA
  is fed, the IP)
* There is several products around (Altera, Xilinx, others), which make 
  bitstream incompatible. Even between the products of the same company
* libraries which abstract the hardware layer like AAL[1] and their 
  versions
* for some products, there is a need for tracking memory usage, which 
  is located on PCI boards
* some of the FPGAs can be exposed using SR-IOV, while some other not, 
  which implies multiple usage abilities

In other words, it may be necessary to incorporate another actions:

* properly discover FPGA and its capabilities
* schedule right bitstream with corresponding matching unoccupied FPGA
  device/slot
* actual program FPGA
* provide libraries into VM, which are necessary for interacting between
  user program and the exposed FPGA (or AAL) (this may be optional, 
  since user can upload complete image with everything in place)
* bitstream images have to be keep in some kind of service (Glance?) 
  with some kind of way for identifying which image match what FPGA

All of that makes modelling resource extremely complicated, contrary to 
CPU resource for example. I'd like to discuss how the goal of having 
reprogrammable accelerators in OpenStack can be achieved. Ideally I'd 
like to fit it into Jay and Chris work on resource-*.

Looking forward for any comments :)

[1] 
http://www.intel.com/content/dam/doc/white-paper/quickassist-technology-aal-white-paper.pdf

-- 
Cheers,
Roman Dobosz

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev