[jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204612#comment-16204612
]
Shane Kumpf edited comment on YARN-7066 at 10/14/17 12:51 PM:
--
[~eyang] thanks for the updated patch. I'm still confused as to what this is
providing over YARN-5534. Both support the user supplying mounts via an
environment variable, which then gets added to the {{docker run}} command. The
only difference for that piece is the format of the user supplied environment
variable, but YARN-5534 goes a step further to attempt to validate the mount is
also in the white list. I reread your comment about why you feel the approach
used by this patch is superior, but I'm not following what you mean.
was (Author: shaneku...@gmail.com):
[~eyang] thanks for the updated patch. I'm still confused as to what this is
providing over YARN-5534. Both support the user supplying mounts via an
environment variable, which then gets added to the {{docker run}} command. The
only difference for that piece is the format of the user supplied environment
variable, but YARN-5534 goes a step further to attempt to validate the mount is
also in the white list. I reread your comment about why you feel this approach
is superior, but I'm not following what you mean.
> Add ability to specify volumes to mount for DockerContainerRuntime
> --
>
> Key: YARN-7066
> URL: https://issues.apache.org/jira/browse/YARN-7066
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Affects Versions: 3.0.0-beta1
>Reporter: Eric Yang
> Attachments: YARN-7066.001.patch, YARN-7066.002.patch
>
>
> Yarnfile describes environment, docker image, and configuration template for
> launching docker containers in YARN. It would be nice to have ability to
> specify the volumes to mount. This can be used in combination to
> AMBARI-21748 to mount HDFS as data directories to docker containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204612#comment-16204612
]
Shane Kumpf edited comment on YARN-7066 at 10/14/17 12:50 PM:
--
[~eyang] thanks for the updated patch. I'm still confused as to what this is
providing over YARN-5534. Both support the user supplying mounts via an
environment variable, which then gets added to the {{docker run}} command. The
only difference for that piece is the format of the user supplied environment
variable, but YARN-5534 goes a step further to attempt to validate the mount is
also in the white list. I reread your comment about why you feel this approach
is superior, but I'm not following what you mean.
was (Author: shaneku...@gmail.com):
[~eyang] thanks for the updated patch. I'm still confused as to what this is
providing over YARN-5534. Both support the user supplying mounts via an
environment variable, which then gets added to the {{docker run}} command. The
only difference for that piece is the format of the user supplied environment
variable, but YARN-5534 goes a step further to attempt to validate the mount is
also in the white list. I reread your comment about why you feel this approach
is similar, but I'm not following what you mean.
> Add ability to specify volumes to mount for DockerContainerRuntime
> --
>
> Key: YARN-7066
> URL: https://issues.apache.org/jira/browse/YARN-7066
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Affects Versions: 3.0.0-beta1
>Reporter: Eric Yang
> Attachments: YARN-7066.001.patch, YARN-7066.002.patch
>
>
> Yarnfile describes environment, docker image, and configuration template for
> launching docker containers in YARN. It would be nice to have ability to
> specify the volumes to mount. This can be used in combination to
> AMBARI-21748 to mount HDFS as data directories to docker containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204321#comment-16204321 ] Eric Yang edited comment on YARN-7066 at 10/13/17 11:08 PM: [~ebadger] Security restriction will be enforced by: # Check for sudo privileges for launching privileged container (YARN-7221) # Enforced effective uid:gid (YARN-4266) # Black listed volume (YARN-7197) # Allowed white list volume (YARN-5534) For privileged users, there is minimum restrictions. For unprivileged users, they can express path to mount, but they will be blocked to unauthorized area or by their own uid:gid privileges to file system ACL. When the listed security defects are solved, this feature will be as good as accessing local file system ACL. was (Author: eyang): [~ebadger] Security restriction will be enforced by: # Check for sudo privileges for launching privileged container (YARN-7221) # Enforced effective uid:gid (YARN-4266) # Black listed volume (YARN-7197) # Allowed white list volume (YARN-5534) For privileged users, there is minimum restrictions. For unprivileged user, they can express path to mount, but they will be blocked to unauthorized area or by their own uid:gid privileges to file system ACL. When the listed security defects are solved, this feature will be as good as accessing local file system ACL. > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > Attachments: YARN-7066.001.patch, YARN-7066.002.patch > > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138862#comment-16138862
]
Eric Yang edited comment on YARN-7066 at 8/23/17 6:43 PM:
--
The current proposed syntax looks like this:
{code}
{
"name": "hbase-app-1",
"components": [
{
"name": "hbasemaster",
...
"configuration": {
"env": {
"HBASE_LOG_DIR": "",
"MOUNTS": "[{ \"source\":\"/home/${USER}\",
\"target\":\"/mnt/hdfs/user/${USER}\", \"option\":\"ro\" },{
\"source\":\"/tmp/${USER}/data\", \"target\":\"/mnt/hdfs/tmp/${USER}/data\" }]"
},
},
{
...
}
],
...
}
{code}
Where "MOUNTS" is a string of JSON that specifies list of mount point source,
target, and option.
{code}
{
"source": "/home/${USER}",
"target": "/mnt/hdfs/${USER}",
"option": "ro"
}
{code}
The nicer design looks like this in Yarnfile:
{code}
{
"name": "serving",
...
"configuration": {
"volumes": [
{
"source": "/mnt/hdfs/user/${USER}",
"target": "/home/${USER}",
"option": "ro"
}
]
}
}
{code}
The nice design will break a couple Yarn container interface because the
original design doesn't contain volumes. Hence, I will go with environment
variable implementation. It might be possible to expose the volumes keyword
for Yarnfile, then pass the information through interface using the environment
variables to avoid changes to container interface.
was (Author: eyang):
The current proposed syntax looks like this:
{code}
{
"name": "hbase-app-1",
"components": [
{
"name": "hbasemaster",
...
"configuration": {
"env": {
"HBASE_LOG_DIR": "",
"MOUNTS": "[{ \"source\":\"/home/${USER}\",
\"target\":\"/mnt/hdfs/user/${USER}\", \"option\":\"ro\" },{
\"source\":\"/tmp/${USER}/data\", \"target\":\"/mnt/hdfs/tmp/${USER}/data\" }]"
},
},
{
...
}
],
...
}
{code}
Where "MOUNTS" is a string of JSON that specifies list of mount point source,
target, and option.
{code}
{
"source": "/home/${USER}",
"target": "/mnt/hdfs/${USER}",
"option": "ro"
}
{code}
The nicer design looks like this in Yarnfile:
{code}
{
"name": "serving",
...
"configuration": {
"volumes": [
{
"source": "/mnt/hdfs/user/${user}",
"target": "/home/${user}",
"option": "ro"
}
]
}
}
{code}
The nice design will break a couple Yarn container interface because the
original design doesn't contain volumes. Hence, I will go with environment
variable implementation. It might be possible to expose the volumes keyword
for Yarnfile, then pass the information through interface using the environment
variables to avoid changes to container interface.
> Add ability to specify volumes to mount for DockerContainerRuntime
> --
>
> Key: YARN-7066
> URL: https://issues.apache.org/jira/browse/YARN-7066
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Affects Versions: 3.0.0-beta1
>Reporter: Eric Yang
>
> Yarnfile describes environment, docker image, and configuration template for
> launching docker containers in YARN. It would be nice to have ability to
> specify the volumes to mount. This can be used in combination to
> AMBARI-21748 to mount HDFS as data directories to docker containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136185#comment-16136185 ] Eric Yang edited comment on YARN-7066 at 8/22/17 2:47 AM: -- [~miklos.szeg...@cloudera.com] Correct. Updated title accordingly. Thanks was (Author: eyang): [~miklos.szeg...@cloudera.com] Correct. Updated title accordingly. > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
