Re: [openstack-dev] FPGA as a dynamic nested resources

[Fei K Chen]

Roman Dobosz  wrote on 2016/07/20 15:25:28:

> From: Roman Dobosz 
> To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Cc: Ed Leafe 
> Date: 2016/07/20 15:30
> Subject: Re: [openstack-dev] FPGA as a dynamic nested resources
>
>
> > > 4. It might be also necessary to track every VF individually,
although
> > >   I didn't assumed it will be needed, nevertheless with nested
> > >   resources it should be easy to handle it.
> >
> > I’m still not seeing the need for nesting. If you have a single FPGA
> > with 8 slots, when you program the slots with accelerators, you now
> > have 8 consumable resources. The fact that they came from a
> > particular FPGA unit doesn’t seem relevant from a scheduling
> > perspective.
>
> Unless you have one FPGA with 8 slots, which can become FPGA with 4
> slots. From scheduling perspective you have to know, which FPGA
> resources can be reconfigured, and which not, isn't it? Also, AFAIRC
> to provide VM with VF, there is a need for providing libvirt with
> address of such VF, right? That's why I've putted this last point.
>
> The whole idea of getting FPGA as resource is its ability to swap
> resources on demand. So it can be thought of as several available
> hardware (means - accelerators, consumable by VMs) which most of the
> time are not programmed in certain moment.
>
Let's have more thought about the resource swapping. The number of run-time
accelerators is not limited by the number of region/slot. Inside FPGA,
there
can be some self-scheduling logic to schedule accelerators on regions by
using
the fast partial reconfiguration. It is not new, there are lots of such
design in FPGA academic.


-- Fei Chen
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] FPGA as a dynamic nested resources

[Fei K Chen]

Roman Dobosz  wrote on 2016/07/20 02:03:28:

> From: Roman Dobosz 
> To: openstack-dev 
> Date: 2016/07/20 02:07
> Subject: [openstack-dev] FPGA as a dynamic nested resources
>
> Hi all,
>
> Some time ago Jay Pipes published etherpad[1] with ideas around
> modelling nested resources, taking NUMA as an example. I was also
> encouraged ;) to start this thread, on last Nova scheduler meeting.
>
> I was read mentioned etherpad and what hits me was that described
> scenario with NUMA cells resembles the way how FPGA can be managed. In
> some extent.
>
> NUMA cell can be treated as a vessel for memory cells, and it is
> expressed as number of MB. So it is possible to extract the
> information from existing data and add another level of aggregation
> using only clever prepared SQL query.
>
> I think, that problem might be broader, than using existing, tweaked a
> bit model. If we take a look into resources, which FPGA may expose,
> than it can be couple of levels, and each of them can be treated as
> resource.
>
> It can identified 3 levels of FPGA resources, which can be nested one
> on the others:
>
> 1. Whole FPGA. If used discrete FPGA, than even today it might be pass
>through to the VM.
>
> 2. Region in FPGA. Some of the FPGA models can be divided into regions
>or slots. Also, for some model it is possible to (re)program such
>region individually - in this case there is a possibility to pass
>entire slot to the VM, so that it might be possible to reprogram
>such slot, and utilize the algorithm within the VM.
>
> 3. Accelerator in region/FPGA. If there is an accelerator programmed
>in the slot, it is possible, that such accelerator provides us with
>Virtual Functions (similar to the SR-IOV), than every available VF
>can be treated as a resource.
>
> 4. It might be also necessary to track every VF individually, although
>I didn't assumed it will be needed, nevertheless with nested
>resources it should be easy to handle it.
You need. For example you have 4 region and 8 VF. Some region is configured
with an accelerator so it can be shared to multi-VM (each consume a VF).
But
some other region is configured with private exclusive accelerator so it
can
only be bind to one VF. That's why we need to track both region and VF.

>
> Correlation between such resources are a bit different from NUMA -
> while in NUMA case there is a possibility to either schedule a VM with
> some memory specified, or request memory within NUMA cell, in FPGA if
> there is slot taken, or accelerator already programmed and used, there
> is no way to offer FPGA as a whole to the tenant, until all
> accelerators and slots are free.
>
> I've followed Jay idea about nested resources and having in mind
> blueprint[2] regarding dynamic resources I've prepared how it fit in.
>
> Tables are unchanged - it is a copy-paste from the etherpad[1]:
>
>
> CREATE TABLE resource_providers (
> id INT NOT NULL AUTOINCREMENT PRIMARY KEY,
> uuid CHAR(36) NOT NULL,
> name VARCHAR(100) NULL,
> root_provider_id INT NULL,
> parent_provider_id INT NULL
> );
>
> CREATE TABLE inventories (
> id INT NOT NULL AUTOINCREMENT PRIMARY KEY,
> resource_provider_id INT NOT NULL,
> resource_class_id INT NOT NULL,
> total INT NOT NULL,
> reserved INT NOT NULL,
> min_unit INT NOT NULL,
> max_unit INT NOT NULL,
> step_size INT NOT NULL,
> allocation_ratio INT NOT NULL
> );
>
> CREATE TABLE allocations (
> id INT NOT NULL AUTOINCREMENT PRIMARY KEY,
> resource_provider_id INT NOT NULL,
> consumer_uuid CHAR(36) NOT NULL,
> resource_class_id INT NOT NULL,
> used INT NOT NULL
> );
>
>
> Than lets fill the tables with data of following structure:
>
> -- FPGA-1
> --   +- FPGA-1 slot1 (taken), resource_provider_id:
> --   +- FPGA-1 slot2
> --   +- FPGA-1 slot2 acceleratorX
> --   +- FPGA-1 slot2 acceleratorX VF1 (taken)
> --   +- FPGA-1 slot2 acceleratorX VF2 (taken)
> --   +- FPGA-1 slot2 acceleratorX VF3 (taken)
> --   +- FPGA-1 slot2 acceleratorX VF4 (taken)
> --   +- FPGA-1 slot2 acceleratorX VF5
> --   +- ..
> --   +- FPGA-1 slot2 acceleratorX VF32
> --   +- FPGA-1 slot3
> -- FPGA-2
> --   +- FPGA-2 slot1
>
> where FPGA-1 and FPGA-2 are hosts with FPGA on board. There is also
> assumed, that new dynamic resources are created: id 1666 means 'FPGA'
> (although it might be simply standard class, which will be hardcoded
> ENUM), 1667 means 'FPGA slot' and 1668 'FPGA accelerator'.
>
>
> INSERT INTO resource_providers VALUES
> (1, '', 'FPGA-1', 1, NULL),
> (2, '', 'FPGA-1 slot 1', 1, 1),
> (3, '', 'FPGA-1 slot 2', 1, 1),
> (4, '', 'FPGA-1 slot 3', 1, 1),
> (5, '', 'FPGA-1 slot 2 acceleratorX', 1, 3),
> (6, '', 'FPGA-2', 6, NULL),
> (7, '', 'FPGA-2 slot', 6, 6);
>
>
> INSERT INTO inventories VALUES
> (1, 1, 1666, 1, 0, 1, 1, 1, 1.0),

Re: [openstack-dev] [Nova] FPGA as a resource

[Fei K Chen]

Hi all,

We agree this is an ingesting direction we can do exploration on OpenStack
and actually we already have such FPGA/GPU as a resource in our public
research cloud called SuperVessel (https://ptopenlab.com). With
modification of OpenStack from community, nearly all features mentioned by
Roman Dobosz are supported, such as FPGA boards detection, scheduling,
images management and user API layer. Here are some materials for you to
reference [1][2].

Unlike CPU and GPU, right now FPGA accelerator is developed without common
protocol above PCIe protocol. Along with huge kinds of FPGA chips, FPGA
boards, it is hard to be managed. Passing through the FPGA as a general
PCIe device to the application stack is a straight way to expose the FPGA
acceleration ability in cloud while it is far way to make it easy used. It
dose not help the accelerator detection, reload, reconfiguration,
deployment in cloud.

Good news is that, the FPGA community is actively promoting high level
programming language, such as OpenCL. They hide nearly all hardware detail
for the developers so it make the "accelerator" common in different
hardware. It is reasonable to see the FPGA become a popular acceleration
hardware. It also reasonable for OpenStack community begin the discussion
to support the FPGA resource management.


[1]
http://openpowerfoundation.org/blogs/fpga-acceleration-in-a-power8-cloud/
[2] http://dl.acm.org/citation.cfm?id=2597929



Best regards,
CHEN Fei





From:   Qiming Teng <teng...@linux.vnet.ibm.com>
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Cc: Fei K Chen/China/IBM@IBMCN, Yong Hua Lin/China/IBM@IBMCN
Date:   2016/04/06 12:25
Subject:Re: [openstack-dev] [Nova] FPGA as a resource



Emm... finally this is brought up. We from IBM have already done some
work on FPGA/GPU resource management [1]. Let me bring the SMEs into
this discussion and see if we together can work out a concrete roadmap
to land this upstream.

Fei and Yonghua, this is indeed very interesting a topic for us.


[1] SuperVessel Cloud: https://ptopenlab.com/

Regards,
  Qiming

On Tue, Apr 05, 2016 at 02:27:30PM +0200, Roman Dobosz wrote:
> Hey all,
>
> On yesterday's scheduler meeting I was raised the idea of bringing up
> the FPGA to the OpenStack as the resource, which than might be exposed
> to the VMs.
>
> The use cases for motivations, why one want do this, are pretty broad -
> having such chip ready on the computes might be beneficial either for
> consumers of the technology and data center administrators. The
> utilization of the hardware is very broad - the only limitations are
> human imagination and hardware capability - since it might be used for
> accelerating execution of algorithms from compression and cryptography,
> through pattern recognition, transcoding to voice/video analysis and
> processing and all the others in between. Using FPGA to perform data
> processing may significantly reduce CPU utilization, the time and power
> consumption, which is a benefit on its own.
>
> On OpenStack side, unlike utilizing the CPU or memory, for actually
> using specific algorithm with FPGAs, it has to be programmed first. So
> in simplified scenario, it might go like this:
>
> * User selects VM with image which supports acceleration,
> * Scheduler selects the appropriate compute host with FPGA available,
> * Compute gets request, program IP into FPGA and then boot up the
>   VM with accelerator attached.
> * If VM is removed, it may optionally erase the FPGA.
>
> As you can see, it seems not complicated at this point, however it
> become more complex due to following things we also have to take into
> consideration:
>
> * recent FPGA are divided into regions (or slots) and every of them
>   can be programmed separately
> * slots may or may not fit the same bitstream (the program which FPGA
>   is fed, the IP)
> * There is several products around (Altera, Xilinx, others), which make
>   bitstream incompatible. Even between the products of the same company
> * libraries which abstract the hardware layer like AAL[1] and their
>   versions
> * for some products, there is a need for tracking memory usage, which
>   is located on PCI boards
> * some of the FPGAs can be exposed using SR-IOV, while some other not,
>   which implies multiple usage abilities
>
> In other words, it may be necessary to incorporate another actions:
>
> * properly discover FPGA and its capabilities
> * schedule right bitstream with corresponding matching unoccupied FPGA
>   device/slot
> * actual program FPGA
> * provide libraries into VM, which are necessary for interacting between
>   user program and the exposed FPGA (or AAL) (this may be optional,
>   since user can upload complete ima