[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624736#comment-16624736
]
Weiwei Yang commented on YARN-8811:
-----------------------------------
Hi [~eyang], [~leftnoteasy]
Thanks for the feedback.
{quote}Mount propagation flags, file system type, source and destination mount
points.
{quote}
[~eyang], thanks for pointing these out. Mount propagation is not in the scope
of this feature, in another word we will only support \{{private}} mount here.
File system type can be specified in the CSI spec, see more in VolumeCapability
- MountVolume - fs_type, so this will depend on the storage system's
capability. CSI has API to validate VolumeCapability to avoid invalid calls.
Source/destination mount points are roughly introduced in section 5.2. I can
add more info for these. For the comment about object store user API key
information, I am not sure about this point, could you please elaborate.
{quote}The name "IGNORABLE" is ambiguous,
{quote}
[~leftnoteasy], I am totally fine with other names, maybe
*{color:#333333}UNMANAGED{color}* {color:#333333}is a better one{color}?
{quote}user will be configured about is it ignorable by main scheduler / nm or
app, etc. And from the name it looks like "no one cares about the resource type
{quote}
For built-in resources, we'll disallow user to modify the resource type. User
will only be allowed to configure type for new user defined resource, and they
take the responsibility to make it right.
{quote}I prefer to have a String[] tags added to each ResourceInformation
{quote}
I thought about this before, but gave up after a PoC. Pros of tags is we
provide the flexibility to support resource tagging/filtering, AKA filter
resources by user given tags. But if we want to support this, then we run into
scenario like following,
For example, for a resource {{yarn.io/gpu}}, a cluster has 2 GPUs from 2
vendors, then user want to mark one device with tag {{vendor_1}}, and the other
tag with {{vendor_2}}.
So when NMs report resource,
{code}
// NM1
yarn.io/gpu {
tags : ["default", "vendor_1"]
...
}
// NM2
yarn.io/gpu {
tags : ["default", "vendor_2"]
...
}
{code}
Note, they have to be two different {{ResoueceInformation}} instances, so when
user ask for {{vendor_1}}, we could allocate {{vendor_1}} gpu correctly on NM1
to the request. If user defines more tags, we will have to deal with more
instances, this is going to be complex and hurt the perf. (This is because our
resource model is flat, multi-dimensional resource is not supported yet).
If you are suggesting just to use static tags, which means a resource can only
associate with a certain set of tags. Then I don't see how it is different with
resource URI. Am I misunderstanding your point? Please let me know.
Thanks
> Support Container Storage Interface (CSI) in YARN
> -------------------------------------------------
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
