[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331858#comment-17331858 ] Akira Ajisaka commented on YARN-8811: - Is this project still ongoing? I didn't see any documents in the source code. > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Labels: CSI > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf, Support Container Storage Interface(CSI) in > YARN_design_doc_v3.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660248#comment-16660248 ] Weiwei Yang commented on YARN-8811: --- Thanks [~sunilg] for the note (y) > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660231#comment-16660231 ] Sunil Govindan commented on YARN-8811: -- I am not very sure why this notification came here. -YARN-8873- is committed. This is a subtask only. I have added parent Jira in the commit message, this may be due to that. I ll remove such messages later on for other patches. > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660148#comment-16660148 ] Hudson commented on YARN-8811: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15290 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15290/]) YARN-8873. [YARN-8811] Add CSI java-based client library. Contributed by (sunilg: rev 0efddd85f02e5942056338746b7fa0d32c255319) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/utils/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/client/CsiGrpcClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/client/CsiClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/client/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/utils/GrpcHelper.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/client/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/dev-support/findbugs-exclude.xml * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/client/FakeCsiIdentityService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/proto/csi.proto * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/client/FakeCsiDriver.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/pom.xml * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/main/java/org/apache/hadoop/yarn/csi/client/CsiClientImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/src/test/java/org/apache/hadoop/yarn/csi/client/TestCsiClient.java > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642697#comment-16642697
]
Weiwei Yang commented on YARN-8811:
---
Thanks [~eyang] for the feedback. The idea behind this is to create an abstract
layer standing between YARN and various of storage systems, and that layer is
built upon CSI spec. This allows us to support various of storage systems
without changing a lot of YARN internals. Given there is already eco-system
around CSI, I think this is a good direction for us too. And technically, the
CSI driver should be CO independent, this was listed as the first goal to
address per CSI spec
{quote}Enable SP authors to write one CSI compliant Plugin that “just works”
across all COs that implement CSI.
{quote}
Hope that clarifies the motivation.
I agree with your suggestion about making the user interface more easier to
use, e.g use unified source path as URI form to make this more consistent for
user to consume. Lets get the infra stuff built first and see if that can be
done on top of that. Hope that makes sense.
Thanks
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design
> doc_20180928.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633235#comment-16633235 ] Eric Yang commented on YARN-8811: - [~cheersyang] This sounds like a risky choice of design that everyone must write a driver to integrate the existing storage system to the new CSI API. Instead of focus on identifying the common denominator of existing storage design principals to draft out CSI spec API. This design will only creates poor performance adapters that tries to convert from one format to another and to another rather than support as many available storage system as possible. K8s seems to make fair share of mistake in this area that they have parallel sets of cli for managing NFS csi driver and other csi drivers. It might end up to be painful to pass in dozen of parameters at different level to just mount one directory. > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633188#comment-16633188 ] Weiwei Yang commented on YARN-8811: --- Hi [~eyang] This scope limits us only to support storage volumes that compatible with CSI, that means we can support NFS via a NFS CSI driver, something similar to [k8s nfs-csi driver|https://github.com/kubernetes-csi/drivers/tree/master/pkg/nfs]. The source mount will be managed by YARN, more specifically its lifecycle is managed by calling CSI driver's node stage/unstage/publish/unpublish APIs. This means we are unable to support NFS gateway if already mounts HDFS to a local filesystem, unless there is CSI driver for the gateway. Thanks > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633059#comment-16633059 ] Eric Yang commented on YARN-8811: - [~cheersyang] What happen when the source mount is location outside of YARN local dir? i.e. NFS proxy to HDFS? It seems the encapsulation is trying to cover too much that limits the choice of source storage options. > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632927#comment-16632927
]
Weiwei Yang commented on YARN-8811:
---
Hi [~eyang]
For volume mounts, user will only need to provide the destination mount, see
the "mountPoint" attribute in following example,
{code:java}
yarn.io/nfs-volume {
size : 100,
unit: Gi,
ResourceType: countable,
tags : [“externalVolume”],
attributes : {
driver: “nfs-csi-driver”,
mountPoint: “/mnt/amt”,
accessMode: “readonly”,
volumeId: "xxx"
}
}
{code}
this mount point will be created in the docker container. YARN will generate an
unique source mount on local disk, for example
{{/hadoop-data/yarn/local/usercache/wwei/filecache/container_1536560680767_0001_01_05_volume-xyz_mount}}.
So in this example, it looks like following
{code}
Destination: /mnt/amt
Source:
/hadoop-data/yarn/local/usercache/wwei/filecache/container_1536560680767_0001_01_05_volume-xyz_mount
{code}
bq. Can we define a way to pass storage specific options such as noatime,
nosuid flags similar to 4th column in /etc/fstab
We are not going to expose those mount options in volume resource, what can be
specified in volume attributes are limited to the options supported by CSI
drivers, per CSI spec. This is because CSI driver covers the complexity of
creating/publishing/mounting volumes for YARN.
Hope that makes sense. Thanks
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design
> doc_20180928.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632050#comment-16632050
]
Eric Yang commented on YARN-8811:
-
[~cheersyang] Thanks for the revision.
{quote}
In upon diagram, “/mnt/data” is the destination mount point of the docker
container, per
container “MountPath” is the source mount point of the docker container. And
“MountPath” is
where a volume is mounted to via CSI driver.{quote}
Both source and destination is specified using the same keyword? This seems
like a typo. Source path might be different from destination path.
Attribute > Source is capitalized, but other attributes are not capitalized.
Can we define a way to pass storage specific options such as noatime, nosuid
flags similar to 4th column in /etc/fstab? It might help to make the spec more
resilient to change later.
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design
> doc_20180928.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631672#comment-16631672 ] Weiwei Yang commented on YARN-8811: --- Had some offline discussion with [~leftnoteasy] and [~eyang], just uploaded v2 design doc. Thanks. > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design > doc_20180928.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626064#comment-16626064
]
Eric Yang commented on YARN-8811:
-
[~cheersyang] {quote}For the comment about object store user API key
information, I am not sure about this point, could you please elaborate.{quote}
In [Hadoop AWS Integration
document|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html],
we need to specify fs.s3a.access.key and fs.s3a.secret.key to connect to aws
account. The same principal applies to swift fs, and other object store.
Therefore, the CSI specification should include ability to pass the key
information to connect to object store. In the current specification, it is
also missing source storage information.
Propagation options are required to make sure multiple mount of the same source
storage system can be shared or exclusive mount. Without this defined, it
might be troublesome for source storage system to decide the locking mechanism.
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624736#comment-16624736
]
Weiwei Yang commented on YARN-8811:
---
Hi [~eyang], [~leftnoteasy]
Thanks for the feedback.
{quote}Mount propagation flags, file system type, source and destination mount
points.
{quote}
[~eyang], thanks for pointing these out. Mount propagation is not in the scope
of this feature, in another word we will only support \{{private}} mount here.
File system type can be specified in the CSI spec, see more in VolumeCapability
- MountVolume - fs_type, so this will depend on the storage system's
capability. CSI has API to validate VolumeCapability to avoid invalid calls.
Source/destination mount points are roughly introduced in section 5.2. I can
add more info for these. For the comment about object store user API key
information, I am not sure about this point, could you please elaborate.
{quote}The name "IGNORABLE" is ambiguous,
{quote}
[~leftnoteasy], I am totally fine with other names, maybe
*{color:#33}UNMANAGED{color}* {color:#33}is a better one{color}?
{quote}user will be configured about is it ignorable by main scheduler / nm or
app, etc. And from the name it looks like "no one cares about the resource type
{quote}
For built-in resources, we'll disallow user to modify the resource type. User
will only be allowed to configure type for new user defined resource, and they
take the responsibility to make it right.
{quote}I prefer to have a String[] tags added to each ResourceInformation
{quote}
I thought about this before, but gave up after a PoC. Pros of tags is we
provide the flexibility to support resource tagging/filtering, AKA filter
resources by user given tags. But if we want to support this, then we run into
scenario like following,
For example, for a resource {{yarn.io/gpu}}, a cluster has 2 GPUs from 2
vendors, then user want to mark one device with tag {{vendor_1}}, and the other
tag with {{vendor_2}}.
So when NMs report resource,
{code}
// NM1
yarn.io/gpu {
tags : ["default", "vendor_1"]
...
}
// NM2
yarn.io/gpu {
tags : ["default", "vendor_2"]
...
}
{code}
Note, they have to be two different {{ResoueceInformation}} instances, so when
user ask for {{vendor_1}}, we could allocate {{vendor_1}} gpu correctly on NM1
to the request. If user defines more tags, we will have to deal with more
instances, this is going to be complex and hurt the perf. (This is because our
resource model is flat, multi-dimensional resource is not supported yet).
If you are suggesting just to use static tags, which means a resource can only
associate with a certain set of tags. Then I don't see how it is different with
resource URI. Am I misunderstanding your point? Please let me know.
Thanks
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623901#comment-16623901
]
Wangda Tan commented on YARN-8811:
--
Thanks [~cheersyang], [~sunilg] for the design doc.
In general the plan looks good, I have some suggestions regarding to resource
type related changes:
The name "IGNORABLE" is ambiguous, user will be configured about is it
ignorable by main scheduler / nm or app, etc. And from the name it looks like
"no one cares about the resource type".
Instead, I prefer to have a String[] tags added to each ResourceInformation:
* All resource types configured inside resource-types.xml have a tag =
"default".
* All existing methods inside Resource work as-is. And returns "default" tag
only.
* A new method will be added to Resource to get/set ResourceInformation by tags.
* For resource types like volume, admin doesn't need, and shouldn't configure
inside resource-types.xml
* Volume still uses the same COUNTABLE resource type.
* If we want to support per-node disk information report to RM, NM should have
a plugin extends {{...resourceplugin.ResourcePlugin}}.
Above suggestion should require minimum changes to scheduler, minimum overhead
for scheduler to make decisions. (Except we may need to copy the extra volume
information). And in the future if new plugins need to be added to scheduler /
node manager to handle different resource tags, developers can easily do that.
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[ https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623889#comment-16623889 ] Eric Yang commented on YARN-8811: - [~cheersyang] Thank you for the proposal. There are several common features in docker and kubernetes for mounting volumes. Mount propagation flags, file system type, source and destination mount points. For object store, there is user API key information that need to be processed. It would be nice if those features can be specified as part of the spec. > Support Container Storage Interface (CSI) in YARN > - > > Key: YARN-8811 > URL: https://issues.apache.org/jira/browse/YARN-8811 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: Support Container Storage Interface(CSI) in YARN_design > doc_20180921.pdf > > > The Container Storage Interface (CSI) is a vendor neutral interface to bridge > Container Orchestrators and Storage Providers. With the adoption of CSI in > YARN, it will be easier to integrate 3rd party storage systems, and provide > the ability to attach persistent volumes for stateful applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
