[
https://issues.apache.org/jira/browse/YARN-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988445#comment-15988445
]
Zhankun Tang commented on YARN-5983:
------------------------------------
[~devaraj.k], Thanks a lot for the review.
1.
{code:xml}
The scheduler only considers non-exclusive resource. The exclusive resources may
have extra attributes needs to be matched when scheduling. Not just simply add
or
reduce a number. For instance, in our PoC, a FPGA slot in one node may already
have one IP flashed so that the scheduler should try to match this IP attribute
to
reuse it.
{code}
{quote}
If you are passing all the attributes of the FPGA resources to RM scheduler,
why do you want to have the NM side resource management? Can you give some
details about the attributes passing to the RM and details maintain by the NM
side resource management in abstract terms?
{quote}
Zhankun --> The RM maintains all attributes of FPGA resources is an ideal
solution which needs to extend YARN-3926 scheduler implementation. But in this
design proposal, we'd prefer to keep YARN-3926 RM scheduler unchanged(which
only consider the count of resource) and add NM a local FPGA resource handler
which will mainly:
a). Schedule the FPGA resource request(2 MCP for instance) into real FPGA device
b). Handle FPGA IP download and re-configure of specific slot
The key part of this proposal is similar to GPU design(YARN-6223) but add the
vendor framework part.
2.
{quote}
Does this need to be done for each container, Can it be done one time during
the cluster installation?
{quote}
Zhankun: --> Good question. Actually the admin can do the IP download and
configuration of FPGA one time to the cluster manually. But if an application
wants to use a new FPGA IP that needs re-configuration, a good way of doing
these may be in YARN itself. This automation in YARN can potentially increase
the FPGA utilization(not good as the ideal RM scheduler) if we have large
amount of FPGA devices.
And these work should be done for each container, although we can skip it if an
FPGA slot already matches IP requirements.
3.
{quote}
Can FPGA slots share my multiple containers? How do we prevent if any
container(Non FPGA allocated container)/application try to use the FPGA
resources which are not allocated to them?
{quote}
Zhankun: --> We can utilize cgroups/Docker to do FPGA slot isolation to avoid
invalid access. But I guess there may be requirements that don't want isolation
in order to maximize device utilization( the user may have a daemon managing
requests from multiple containers to FPGA IP), we can provide option for user
to disable isolation?
4.
{quote}
Any changes to ContainerExecutor, how does the application code running in the
container come to know about the allocated FPGA resource to access/use the FPFA?
{quote}
Zhankun:--> It should be no changes to both DefaultContainerExecutor and
LinuxContainerExecutor. Knowing about which FPGA resource to use depends on the
vendor specific SDK I think. For instance, container application code may
utilize Intel AAL SDK and the latter will know which FPGA slot to use (maybe
based on some environment setting).
5.
{quote}
What are the configurations user to need to configure for the application to
use FPGA resources?
{quote}
Zhankun: --> For YARN configurations before cluster start, the admin need to
define different type of FPGA resources and its count. When a user wants to
submit the application, it needs to provide two inputs:
a). the required FPGA resource for container request (AM will use this to
request FPGA container from RM)
b). the ID/name of the required FPGA IP (can be set in container's environment
variable for container run-time to use)
And great thanks again for the questions and remind me some points uncertain.
I'll provide a patch to make it more clear later.
> [Umbrella] Support for FPGA as a Resource in YARN
> -------------------------------------------------
>
> Key: YARN-5983
> URL: https://issues.apache.org/jira/browse/YARN-5983
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn
> Reporter: Zhankun Tang
> Assignee: Zhankun Tang
> Attachments: YARN-5983-Support-FPGA-resource-on-NM-side_v1.pdf
>
>
> As various big data workload running on YARN, CPU will no longer scale
> eventually and heterogeneous systems will become more important. ML/DL is a
> rising star in recent years, applications focused on these areas have to
> utilize GPU or FPGA to boost performance. Also, hardware vendors such as
> Intel also invest in such hardware. It is most likely that FPGA will become
> popular in data centers like CPU in the near future.
> So YARN as a resource managing and scheduling system, would be great to
> evolve to support this. This JIRA proposes FPGA to be a first-class citizen.
> The changes roughly includes:
> 1. FPGA resource detection and heartbeat
> 2. Scheduler changes
> 3. FPGA related preparation and isolation before launch container
> We know that YARN-3926 is trying to extend current resource model. But still
> we can leave some FPGA related discussion here
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
