[jira] [Created] (YARN-3252) YARN LinuxContainerExecutor runs as nobody in Simple Security mode for all applications
Eric Yang created YARN-3252: --- Summary: YARN LinuxContainerExecutor runs as nobody in Simple Security mode for all applications Key: YARN-3252 URL: https://issues.apache.org/jira/browse/YARN-3252 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.5.2, 2.5.1, 2.6.0, 2.4.0, 2.3.0 Environment: Linux Reporter: Eric Yang Priority: Critical When using YARN + Slider + LinuxContainerExecutor, all slider application are running as nobody. This is because the modification in YARN-1253 to restrict all containers to run as a single user. This becomes a exploite to any application that runs inside YARN + Slider + LCE. The original behavior is more correct. The original statement indicated that users can impersonate any other users. This supposed to be only valid for proxy users, who can proxy as other users. It is designed as intended that the service user needs to be trusted by the framework to impersonate end users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3252) YARN LinuxContainerExecutor runs as nobody in Simple Security mode for all applications
[ https://issues.apache.org/jira/browse/YARN-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335677#comment-14335677 ] Eric Yang commented on YARN-3252: - Thank you Allen. By using the fix in YARN-2424, it is possible to run application as submitted user. Proxy user concept could be an enhancement to ensure only a small set of users have privileges to run core services. Such as Spark on YARN or HBase on YARN, those users are supposely trusted services. Some proxy user rules in YARN can help to reduce the security risk that YARN-1253 was originally concerned about. YARN LinuxContainerExecutor runs as nobody in Simple Security mode for all applications --- Key: YARN-3252 URL: https://issues.apache.org/jira/browse/YARN-3252 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.3.0, 2.4.0, 2.6.0, 2.5.1, 2.5.2 Environment: Linux Reporter: Eric Yang Priority: Critical When using YARN + Slider + LinuxContainerExecutor, all slider application are running as nobody. This is because the modification in YARN-1253 to restrict all containers to run as a single user. This becomes a exploite to any application that runs inside YARN + Slider + LCE. The original behavior is more correct. The original statement indicated that users can impersonate any other users. This supposed to be only valid for proxy users, who can proxy as other users. It is designed as intended that the service user needs to be trusted by the framework to impersonate end users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4266) Allow users to enter containers as UID:GID pair instead of by username
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137455#comment-16137455 ] Eric Yang commented on YARN-4266: - +1 works on my local build and pass test-patch test. > Allow users to enter containers as UID:GID pair instead of by username > -- > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266.002.patch, YARN-4266.003.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7066: Issue Type: Sub-task (was: New Feature) Parent: YARN-3611 > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167345#comment-16167345 ] Eric Yang edited comment on YARN-7193 at 9/15/17 5:26 AM: -- Fixed firebug warnings, author tag, javadoc and check styles warnings. The unit test failures are not related to code changes in this JIRA. was (Author: eyang): Fixed firebug warnings, author tag, and check styles warnings. The unit test failures are not related to code changes in this JIRA. > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch, > YARN-7193.yarn-native-services.002.patch, > YARN-7193.yarn-native-services.003.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7193: Attachment: YARN-7193.yarn-native-services.003.patch Fixed firebug warnings, author tag, and check styles warnings. The unit test failures are not related to code changes in this JIRA. > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch, > YARN-7193.yarn-native-services.002.patch, > YARN-7193.yarn-native-services.003.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7193: Attachment: YARN-7193.yarn-native-services.004.patch Fixed checkstyle, white space and firebugs issues. The unit test failures are test case race conditions with network timeout. The test failure exists independent of this patch. > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch, > YARN-7193.yarn-native-services.002.patch, > YARN-7193.yarn-native-services.003.patch, > YARN-7193.yarn-native-services.004.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6626: Attachment: (was: YARN-6626.yarn-native-services.001.patch) > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6626: Attachment: YARN-6626.yarn-native-services.001.patch > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170274#comment-16170274 ] Eric Yang commented on YARN-7193: - This modifies dev-support/docker/Dockerfile to include a more recent version of nodejs, and apidoc for documenting REST API. However, JDK 1.7 seems to have disappeared from Oracle website. This broke the pre-commit test for patch 005. Do we need to keep dev-support/docker/Dockerfile and patchprocess/yetus-0.4.0-lib/precommit/test-patch-docker/Dockerfile in sync? Nodejs required dependencies are not described in yetus version of Dockerfile. > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch, > YARN-7193.yarn-native-services.002.patch, > YARN-7193.yarn-native-services.003.patch, > YARN-7193.yarn-native-services.004.patch, > YARN-7193.yarn-native-services.005.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7193: Attachment: YARN-7193.yarn-native-services.005.patch > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch, > YARN-7193.yarn-native-services.002.patch, > YARN-7193.yarn-native-services.003.patch, > YARN-7193.yarn-native-services.004.patch, > YARN-7193.yarn-native-services.005.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Description: We can show case the following capabilities in the YARN service examples: # Description of the service # Component dependencies # How to mount HDFS volume via NFS Gateway # Enable privileged container # Quick link to web application # Queue to submit was: We can show case the following capabilities in the YARN service examples: # Description of the service # Component dependencies # How to mount HDFS volume via NFS Gateway # Enforce non-privileged container # Enable privileged container # Quick link to web application # Queue to submit > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Description: We can show case the following capabilities in the YARN service examples: # Description of the service # Component dependencies # How to mount HDFS volume via NFS Gateway # Enable privileged container # Quick link to web application # Queue to submit # Use private docker registry was: We can show case the following capabilities in the YARN service examples: # Description of the service # Component dependencies # How to mount HDFS volume via NFS Gateway # Enable privileged container # Quick link to web application # Queue to submit > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Description: We can show case the following capabilities in the YARN service examples: # Description of the service # Component dependencies # How to mount HDFS volume via NFS Gateway # Enforce non-privileged container # Enable privileged container # Quick link to web application # Queue to submit > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enforce non-privileged container > # Enable privileged container > # Quick link to web application > # Queue to submit -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170741#comment-16170741 ] Eric Yang commented on YARN-6626: - Base on new finding that ui2 webapp is highly coupled with RMWebServices.java. However /ui2 and RMWebServices are initialized are two separate web applications. Java Sessions are not synchronized between both web applications. At this time, ui2 is pure javascript based webapp. Java session is not yet a real problem. However, it might be best to manage application oriented REST end points independent from ui2 webapplication to avoid introducing out of sync Java session problems. > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170803#comment-16170803 ] Eric Yang commented on YARN-5534: - [~miklos.szeg...@cloudera.com] I think core-site.xml make most sense to ensure both hdfs and yarn can agree on same security setting even though hdfs service doesn't require knowledge of this today. The idea of global white list and job specific white lists, have their own attractiveness. However, I think having global white list in container-executor.cfg might be risky. If the information is leaked and admin did not secure white list mount point properly, then the system could be vulnerable to attack. For white list, more eye balls can examine the configuration, would make the system more secure. On the other hand, if a black list is to be implemented, then it might have advantage to be in container-executor.cfg and readable by root only. Basic security through obscurity can be performed with some level of effectiveness. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.001.patch Example for deploying Apache, MySQL, PHPMyAdmin docker containers. > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172617#comment-16172617
]
Eric Yang commented on YARN-7215:
-
Data stored in ZooKeeper can not exceed 1MB per node. It is possible for large
scale application to exceed that limit when the hostnames and config key/value
pairs are stored in the state or spec file. Application state maybe fine, but
I can't recommend to use ZooKeeper as low latency storage for application
configuration. Ambari version 0.0 (HMS) had implemented similar use case, and
it quickly hits z-node size limitation.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7215) REST API to list all deployed services by the same user
[ https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7215: Issue Type: Sub-task (was: Bug) Parent: YARN-7054 > REST API to list all deployed services by the same user > --- > > Key: YARN-7215 > URL: https://issues.apache.org/jira/browse/YARN-7215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, applications >Reporter: Eric Yang >Assignee: Eric Yang > > In Slider, it is possible to list deployed applications from the same user by > using: > > slider list > > This API can help UI to display application and services deployed by the same > user. > Apiserver does not have ability to list all applications/services at this > time. This API requires fast response to list all applications because it is > a common UI operation. ApiServer deployed applications persist configuration > in HDFS similar to slider, but using directory listing to display deployed > application might cost too much overhead to namenode. We may want to use > alternative storage mechanism to cache deployed application configuration to > accelerate the response time of list deployed applications. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7216) Missing ability to list configuration vs status
[
https://issues.apache.org/jira/browse/YARN-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7216:
Description:
API Server has /ws/v1/services/{service_name}. This REST end point returns
Services object which contains both configuration and status. When status or
macro based parameters changed in Services object, it can confuse UI code to
making configuration changes. The suggestion is to preserve a copy of
configuration object independent of status object. This gives UI ability to
change services configuration and update configuration.
Similar to Ambari, it might provide better information if we have the following
separated REST end points:
{code}
/ws/v1/services/[service_name]/config
/ws/v1/services/[service_name]/status
{code}
was:
API Server has /ws/v1/services/{service_name}. This REST end point returns
Services object which contains both configuration and status. When status or
macro based parameters changed in Services object, it can confuse UI code to
making configuration changes. The suggestion is to preserve a copy of
configuration object independent of status object. This gives UI ability to
change services configuration and update configuration.
Similar to Ambari, it might provide better information if we have the following
separated REST end points:
{code}
/ws/v1/services/{service_name}/config
/ws/v1/services/{service_name}/status
{code}
> Missing ability to list configuration vs status
> ---
>
> Key: YARN-7216
> URL: https://issues.apache.org/jira/browse/YARN-7216
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> API Server has /ws/v1/services/{service_name}. This REST end point returns
> Services object which contains both configuration and status. When status or
> macro based parameters changed in Services object, it can confuse UI code to
> making configuration changes. The suggestion is to preserve a copy of
> configuration object independent of status object. This gives UI ability to
> change services configuration and update configuration.
> Similar to Ambari, it might provide better information if we have the
> following separated REST end points:
> {code}
> /ws/v1/services/[service_name]/config
> /ws/v1/services/[service_name]/status
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7220) Use apidoc for REST API documentation
Eric Yang created YARN-7220: --- Summary: Use apidoc for REST API documentation Key: YARN-7220 URL: https://issues.apache.org/jira/browse/YARN-7220 Project: Hadoop YARN Issue Type: Improvement Components: documentation Reporter: Eric Yang Assignee: Eric Yang There are more REST API being developed in Hadoop, and it would be great to standardize on the method of generate REST API document. There are several method done today: Swagger YAML Javadoc Wiki pages JIRA comments The most frequently used method is JIRA comments and Wiki pages. Both methods are prone to data loss through passage of time. We will need a more effortless approach to maintain REST API documentation. Swagger YAML can also be out of sync with reality, if new methods are added to java code directly. Javadoc annotation seems like a good approach to maintain REST API document. Both Jersey and Atlassian community has maven plugin to help generating REST API document, but those maven plugins have ceased to function. After searching online for REST API documentation for a bit, [apidoc|http://apidocjs.com/] is one library that stand out. This could be the ideal approach to manage Hadoop REST API document. It supports javadoc like annotations, and generate beautiful schema changes documentation. If this is accepted, I will add apidoc installation to dev-support Dockerfile, and pom.xml changes for javadoc plugin to ignore the custom tags. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172359#comment-16172359
]
Eric Yang commented on YARN-7217:
-
[~jianhe] Agree, spec will make this less confusing.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172434#comment-16172434
]
Eric Yang commented on YARN-7215:
-
[~jianhe] How does RM handle a service that is in stopped state? Stopped
slider application does not have any record in resource manager. Same slider
application can have multiple Application ID when the application has been
restarted. Slider uses HDFS file to persist the paused application, but having
resource manager to crawl through lists of HDFS directories to find stopped
service seems like potential load attack to namenode. It would be better to
have the operational record index, and cached by well known mechanism like a
SOLR collection. This also reduces having to brew another random read/write,
low latency, index, cache mechanism in YARN. Both HBase and SOLR have solved
random read/write on top of HDFS with some success. It would be better to we
use existing libraries that have been baked for several years than inventing
something new for specialized purpose.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7221) Add security check for privileged docker container
Eric Yang created YARN-7221: --- Summary: Add security check for privileged docker container Key: YARN-7221 URL: https://issues.apache.org/jira/browse/YARN-7221 Project: Hadoop YARN Issue Type: Sub-task Reporter: Eric Yang When a docker is running with privileges, majority of the use case is to have some program running with root then drop privileges to another user. i.e. httpd to start with privileged and bind to port 80, then drop privileges to www user. # We should add security check for submitting users, to verify they have "sudo" access to run privileged container. # We should remove --user=uid:gid for privileged containers. Docker can be launched with --privileged=true, and --user=uid:gid flag. With this parameter combinations, user will not have access to become root user. All docker exec command will be drop to uid:gid user to run instead of granting privileges. User can gain root privileges if container file system contains files that give user extra power, but this type of image is considered as dangerous. Non-privileged user can launch container with special bits to acquire same level of root power. Hence, we lose control of which image should be run with --privileges, and who have sudo rights to use privileged container images. As the result, we should check for sudo access then decide to parameterize --privileged=true OR --user=uid:gid. This will avoid leading developer down the wrong path. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7216) Missing ability to list configuration vs status
[
https://issues.apache.org/jira/browse/YARN-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7216:
Issue Type: Sub-task (was: Bug)
Parent: YARN-7054
> Missing ability to list configuration vs status
> ---
>
> Key: YARN-7216
> URL: https://issues.apache.org/jira/browse/YARN-7216
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> API Server has /ws/v1/services/{service_name}. This REST end point returns
> Services object which contains both configuration and status. When status or
> macro based parameters changed in Services object, it can confuse UI code to
> making configuration changes. The suggestion is to preserve a copy of
> configuration object independent of status object. This gives UI ability to
> change services configuration and update configuration.
> Similar to Ambari, it might provide better information if we have the
> following separated REST end points:
> {code}
> /ws/v1/services/[service_name]/config
> /ws/v1/services/[service_name]/status
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7215:
Description:
In Slider, it is possible to list deployed applications from the same user by
using:
{code}
slider list
{code}
This API can help UI to display application and services deployed by the same
user.
Apiserver does not have ability to list all applications/services at this time.
This API requires fast response to list all applications because it is a
common UI operation. ApiServer deployed applications persist configuration in
HDFS similar to slider, but using directory listing to display deployed
application might cost too much overhead to namenode. We may want to use
alternative storage mechanism to cache deployed application configuration to
accelerate the response time of list deployed applications.
was:
In Slider, it is possible to list deployed applications from the same user by
using:
slider list
This API can help UI to display application and services deployed by the same
user.
Apiserver does not have ability to list all applications/services at this time.
This API requires fast response to list all applications because it is a
common UI operation. ApiServer deployed applications persist configuration in
HDFS similar to slider, but using directory listing to display deployed
application might cost too much overhead to namenode. We may want to use
alternative storage mechanism to cache deployed application configuration to
accelerate the response time of list deployed applications.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7218) ApiServer REST API naming convention /ws/v1 is already used in Hadoop v2
Eric Yang created YARN-7218: --- Summary: ApiServer REST API naming convention /ws/v1 is already used in Hadoop v2 Key: YARN-7218 URL: https://issues.apache.org/jira/browse/YARN-7218 Project: Hadoop YARN Issue Type: Sub-task Components: api, applications Reporter: Eric Yang Assignee: Eric Yang In YARN-6626, there is a desire to have ability to run ApiServer REST API in Resource Manager, this can eliminate the requirement to deploy another daemon service for submitting docker applications. In YARN-5698, a new UI has been implemented as a separate web application. There are some problems in the arrangement that can cause conflicts of how Java session are being managed. The root context of Resource Manager web application is /ws. This is hard coded in startWebapp method in ResourceManager.java. This means all the session management is applied to Web URL of /ws prefix. /ui2 is independent of /ws context, therefore session management code doesn't apply to /ui2. This could be a session management problem, if servlet based code is going to be introduced into /ui2 web application. ApiServer code base is designed as a separate web application. There is no easy way to inject a separate web application into the same /ws context because ResourceManager is already setup to bind to RMWebServices. Unless ApiServer code is moved into RMWebServices, otherwise, they will not share the same session management. The alternate solution is to keep ApiServer prefix URL independent of /ws context. However, this will be a departure from YARN web services naming convention. This can be loaded as a separate web application in Resource Manager jetty server. One possible proposal is /app/v1/services. This can keep ApiServer code modular and independent from Resource Manager. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7217) PUT method for update service for Service API doesn't function correctly
Eric Yang created YARN-7217:
---
Summary: PUT method for update service for Service API doesn't
function correctly
Key: YARN-7217
URL: https://issues.apache.org/jira/browse/YARN-7217
Project: Hadoop YARN
Issue Type: Task
Components: api, applications
Reporter: Eric Yang
The PUT method for updateService API provides multiple functions:
# Stopping a service.
# Start a service.
# Increase or decrease number of containers.
The overloading is buggy depending on how the configuration should be applied.
Scenario 1
A user retrieves Service object from getService call, and the Service object
contains state: STARTED. The user would like to increase number of containers
for the deployed service. The JSON has been updated to increase container
count. The PUT method does not actually increase container count.
Scenario 2
A user retrieves Service object from getService call, and the Service object
contains state: STOPPED. The user would like to make a environment
configuration change. The configuration does not get updated after PUT method.
This is possible to address by rearranging the logic of START/STOP after
configuration update. However, there are other potential combinations that can
break PUT method. For example, user like to make configuration changes, but
not yet restart the service until a later time.
The alternative is to separate the PUT method into PUT method for configuration
vs status. This increase the number of action that can be performed. New API
could look like:
{code}
@PUT
/ws/v1/services/[service_name]/config
Request Data:
{
"name":"[service_name]",
"number_of_containers": 5
}
{code}
{code}
@PUT
/ws/v1/services/[service_name]/state
Request data:
{
"name": "[service_name]",
"state": "STOPPED|STARTED"
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172434#comment-16172434
]
Eric Yang edited comment on YARN-7215 at 9/19/17 10:19 PM:
---
[~jianhe] How does RM handle a service that is in stopped state? Stopped
slider application does not have any record in resource manager. Same slider
application can have multiple Application ID when the application has been
restarted. Slider uses HDFS file to persist the paused application, but having
resource manager to crawl through lists of HDFS directories to find stopped
service seems like potential load attack to namenode. It would be better to
have the operational record index, and cached by well known mechanism like a
SOLR collection. This also reduces having to brew another random read/write,
low latency, index, cache mechanism in YARN. Both HBase and SOLR have solved
random read/write on top of HDFS with some success. It would be better to use
existing libraries that have been baked for several years than inventing
something new for specialized purpose.
was (Author: eyang):
[~jianhe] How does RM handle a service that is in stopped state? Stopped
slider application does not have any record in resource manager. Same slider
application can have multiple Application ID when the application has been
restarted. Slider uses HDFS file to persist the paused application, but having
resource manager to crawl through lists of HDFS directories to find stopped
service seems like potential load attack to namenode. It would be better to
have the operational record index, and cached by well known mechanism like a
SOLR collection. This also reduces having to brew another random read/write,
low latency, index, cache mechanism in YARN. Both HBase and SOLR have solved
random read/write on top of HDFS with some success. It would be better to we
use existing libraries that have been baked for several years than inventing
something new for specialized purpose.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172816#comment-16172816
]
Eric Yang commented on YARN-7215:
-
Slider list was not robust. It take multiple seconds to respond to the query.
If there are several hundred users using a UI to manage user's own application.
The retrieval of information should have low range of millisecond response
time. When application data are persisted in the same metastore with index and
search capability, it will be easier to use the same storage mechanism to build
application catalog. Although it is easy to build a view of YARN deployed
applications base on computing metadata stored on HDFS and ZooKeeper. However,
those services are not optimized for serving web application REST API. Let's
take one step further on reducing too many small file problem on HDFS and too
big z-node on ZooKeeper in consideration of the design. This will help to
steer developers toward good design pattern. I am open to suggestion to list
yarn applications which can survive ResourceManager restart.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6626: Attachment: YARN-6626.yarn-native-services.002.patch Added ability to load ApiServer to RM as a web filter using reflection. This is done to avoid Maven cyclic dependency on hadoop-yarn-services-api module. > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch, > YARN-6626.yarn-native-services.002.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7201) Add an apache httpd example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175438#comment-16175438 ] Eric Yang commented on YARN-7201: - +1 for patch 007 to reflect the current state of the code base. > Add an apache httpd example YARN service > > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch, > YARN-7201.yarn-native-services.004.patch, > YARN-7201.yarn-native-services.005.patch, > YARN-7201.yarn-native-services.006.patch, > YARN-7201.yarn-native-services.007.patch > > > Add an apache httpd example service -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7201) Add an apache httpd example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175597#comment-16175597 ] Eric Yang commented on YARN-7201: - +1 for patch 008. > Add an apache httpd example YARN service > > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch, > YARN-7201.yarn-native-services.004.patch, > YARN-7201.yarn-native-services.005.patch, > YARN-7201.yarn-native-services.006.patch, > YARN-7201.yarn-native-services.007.patch, > YARN-7201.yarn-native-services.008.patch > > > Add an apache httpd example service -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6626: Attachment: YARN-6626.yarn-native-services.004.patch Added @XmlType annotation to make data structure serializable when Jersey validates schema. > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch, > YARN-6626.yarn-native-services.002.patch, > YARN-6626.yarn-native-services.003.patch, > YARN-6626.yarn-native-services.004.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7221) Add security check for privileged docker container
[ https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173881#comment-16173881 ] Eric Yang commented on YARN-7221: - [~chris.douglas] This is not a duplicate of YARN-6623. This is extension to permit privileged containers, if the launching user has sudo rights to run docker or being part of docker group. > Add security check for privileged docker container > -- > > Key: YARN-7221 > URL: https://issues.apache.org/jira/browse/YARN-7221 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang > > When a docker is running with privileges, majority of the use case is to have > some program running with root then drop privileges to another user. i.e. > httpd to start with privileged and bind to port 80, then drop privileges to > www user. > # We should add security check for submitting users, to verify they have > "sudo" access to run privileged container. > # We should remove --user=uid:gid for privileged containers. > > Docker can be launched with --privileged=true, and --user=uid:gid flag. With > this parameter combinations, user will not have access to become root user. > All docker exec command will be drop to uid:gid user to run instead of > granting privileges. User can gain root privileges if container file system > contains files that give user extra power, but this type of image is > considered as dangerous. Non-privileged user can launch container with > special bits to acquire same level of root power. Hence, we lose control of > which image should be run with --privileges, and who have sudo rights to use > privileged container images. As the result, we should check for sudo access > then decide to parameterize --privileged=true OR --user=uid:gid. This will > avoid leading developer down the wrong path. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168584#comment-16168584 ] Eric Yang commented on YARN-5534: - Yarn-site.xml and core-site.xml are trusted configuration from Hdoop server point of view. Hadoop Kerberos, and proxy users configuration are defined in the *.xml files. WIthout trusting those configurations, Hadoop security fall apart. There is keyword like final to lock configuration in place therefore an overlay of Hadoop configuration can not alter the configuration values. Volume white list in core-site.xml or yarn-site.xml are secured. There should be very little configuration in container-executor.cfg file to govern uid and banned user. The rest of the logic in core-site.xml is preferred to ensure the logic is preprocessed in yarn user before handing off to root for execution. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168584#comment-16168584 ] Eric Yang edited comment on YARN-5534 at 9/15/17 9:58 PM: -- Yarn-site.xml and core-site.xml are trusted configuration from Hdoop server point of view. Hadoop Kerberos, and proxy users configuration are defined in the *.xml files. WIthout trusting those configurations, Hadoop security fall apart. There is keyword like final to lock configuration in place therefore an overlay of Hadoop configuration can not alter the configuration values. Volume white list in core-site.xml or yarn-site.xml are secured. There should be very little configuration in container-executor.cfg file to govern uid and banned user. The rest of the logic in core-site.xml is preferred to ensure the logic is preprocessed in yarn user before handing off to root for execution. YARN can act as a shielding user in pre-processing to make exploits more difficult. was (Author: eyang): Yarn-site.xml and core-site.xml are trusted configuration from Hdoop server point of view. Hadoop Kerberos, and proxy users configuration are defined in the *.xml files. WIthout trusting those configurations, Hadoop security fall apart. There is keyword like final to lock configuration in place therefore an overlay of Hadoop configuration can not alter the configuration values. Volume white list in core-site.xml or yarn-site.xml are secured. There should be very little configuration in container-executor.cfg file to govern uid and banned user. The rest of the logic in core-site.xml is preferred to ensure the logic is preprocessed in yarn user before handing off to root for execution. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169188#comment-16169188 ] Eric Yang commented on YARN-5534: - [~miklos.szeg...@cloudera.com] It's a cute perspective, but there might be usability issues. Today, it is possible to keep container-executor.cfg read only to root and yarn user. Authorized and banned users are only known to root user and yarn user. This is similar to sudoers file that managers who has sudoers rights. Other the other hand, file system mount points needs to be known by all users who would like to use mount points. It would be more convenient to give everyone read access to file system mount point file, like /etc/fstab. If volume white list is mixed with user privileges control, then we lose some flexibility to keep banned users a secret or we lose ability to know what mount points can be used. With this reason, I prefer to keep white list volume separated from container-executor.cfg for separation of duty from security point of view. However, black list volume maintained in container-executor.cfg, can make attack more difficult. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6626: Attachment: YARN-6626.yarn-native-services.001.patch First draft for bundle api service REST API in RM. This JIRA depends on pom.xml file modification in YARN-7193. The current approach is to bundle the REST API code as a jar file, and allow hadoop-yarn-ui project to bundle the REST API jar file, and configure web.xml to load REST API classes. The drawback of this approach is the URI is prefixed with /ui2 instead of /ws/v1/. /ws/v1/ is already taken by resource manager's own REST API. We probably want to label this as /ws/v2/ or /app/v1/ as prefix of REST API to avoid confusion between resource manager API vs application REST API. The alternative method is to move the code from ApiServer.java into RMWebApp.java, which makes that file bloated, and can't be separated as another standalone server. > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166916#comment-16166916 ] Eric Yang commented on YARN-7197: - Consider the following scenarios: # Docker container with --privileged=true flag enabled and run systemd in docker container with: /sys/fs/cgroup:/sys/fs/cgroup:ro. This allows systemd to run in docker container, while protecting cgroup control enforced from host layer. # Docker container attempt to mount /run should be forbidden, always initialized with tmpfs. # Allow docker container to access /var/run/docker.socket on the host layer for privileged container to interact with host layer docker daemon. The black list needs to have ability to list all the forbidden mount points, such as /sys, and /run. There are some exception that needs to have ability to mount as read only. The last but not the least, the ability to create special mount points for privileged container to access host layer docker daemon. This feature requires to maintain 3 control lists for accuracy. # A general backlisted mount points # A read-only mount points # A exception list that is allowed to mount, if running with privileged mode. Last one can be covered by white list feature in YARN-5534. This JIRA only needs to cover case 1, and 2. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7193) Implement REST API for register Yarnfile in application catalog
Eric Yang created YARN-7193: --- Summary: Implement REST API for register Yarnfile in application catalog Key: YARN-7193 URL: https://issues.apache.org/jira/browse/YARN-7193 Project: Hadoop YARN Issue Type: Sub-task Components: applications Affects Versions: 3.1.0 Reporter: Eric Yang Assignee: Eric Yang For support ability to register and index Yarnfile, we need a set of REST API to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7193: Attachment: YARN-7193.yarn-native-services.002.patch Missed two license files, update license again. > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch, > YARN-7193.yarn-native-services.002.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7193) Implement REST API for register Yarnfile in application catalog
[ https://issues.apache.org/jira/browse/YARN-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7193: Attachment: YARN-7193.yarn-native-services.001.patch Implement REST API for application catalog. > Implement REST API for register Yarnfile in application catalog > --- > > Key: YARN-7193 > URL: https://issues.apache.org/jira/browse/YARN-7193 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Affects Versions: 3.1.0 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7193.yarn-native-services.001.patch > > > For support ability to register and index Yarnfile, we need a set of REST API > to register, search, and recommend applications from application catalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6626) Embed REST API service into RM
[
https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167071#comment-16167071
]
Eric Yang commented on YARN-6626:
-
This can be done by specifying hadoop-yarn-service-api as dependency to
hadoop-yarn-ui project, this will automatically place
hadoop-yarn-service-api*.jar and dependencies into WEB-INF/lib of
hadoop-yarn-ui war file. In web.xml file, specify:
{code}
YARN UI
REST_API
com.sun.jersey.spi.container.servlet.ServletContainer
com.sun.jersey.config.property.packages
org.apache.hadoop.yarn.appcatalog.controller;org.apache.hadoop.yarn.service.webapp
com.sun.jersey.api.json.POJOMappingFeature
true
1
REST_API
/ws/v2/*
{code}
> Embed REST API service into RM
> --
>
> Key: YARN-6626
> URL: https://issues.apache.org/jira/browse/YARN-6626
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gour Saha
> Fix For: yarn-native-services
>
>
> As of now the deployment model of the Native Services REST API service is
> standalone. There are several cross-cutting solutions that can be inherited
> for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API
> service if it is embedded into the RM process. In fact we can expose the REST
> API via the same port as RM UI (8088 default). The URI path
> /services/v1/applications will distinguish the REST API calls from other RM
> APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6626) Embed REST API service into RM
[
https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167071#comment-16167071
]
Eric Yang edited comment on YARN-6626 at 9/14/17 10:47 PM:
---
This can be done by specifying hadoop-yarn-service-api as dependency to
hadoop-yarn-ui project, this will automatically place
hadoop-yarn-service-api*.jar and dependencies into WEB-INF/lib of
hadoop-yarn-ui war file. In web.xml file, specify:
{code}
YARN UI
REST_API
com.sun.jersey.spi.container.servlet.ServletContainer
com.sun.jersey.config.property.packages
org.apache.hadoop.yarn.appcatalog.controller;org.apache.hadoop.yarn.service.webapp
com.sun.jersey.api.json.POJOMappingFeature
true
1
REST_API
/ws/v2/*
{code}
Note that the prefix URL is hard coded in RestApiConstants. The better
approach is to ensure the prefix directory is described by configuration like
in web.xml, or in ApiServerWebApp where addJerseyResourcePackage is the prefix
path. Hence the URL have the same prefix between apiserver and resource
manager.
was (Author: eyang):
This can be done by specifying hadoop-yarn-service-api as dependency to
hadoop-yarn-ui project, this will automatically place
hadoop-yarn-service-api*.jar and dependencies into WEB-INF/lib of
hadoop-yarn-ui war file. In web.xml file, specify:
{code}
YARN UI
REST_API
com.sun.jersey.spi.container.servlet.ServletContainer
com.sun.jersey.config.property.packages
org.apache.hadoop.yarn.appcatalog.controller;org.apache.hadoop.yarn.service.webapp
com.sun.jersey.api.json.POJOMappingFeature
true
1
REST_API
/ws/v2/*
{code}
> Embed REST API service into RM
> --
>
> Key: YARN-6626
> URL: https://issues.apache.org/jira/browse/YARN-6626
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gour Saha
> Fix For: yarn-native-services
>
>
> As of now the deployment model of the Native Services REST API service is
> standalone. There are several cross-cutting solutions that can be inherited
> for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API
> service if it is embedded into the RM process. In fact we can expose the REST
> API via the same port as RM UI (8088 default). The URI path
> /services/v1/applications will distinguish the REST API calls from other RM
> APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167123#comment-16167123
]
Eric Yang commented on YARN-7197:
-
[~ebadger] This is similar to firewall rules. We need a way to define ACCEPTED
and REJECTED rules. White list could be cumbersome to express an exhaustive
list of mount points sometimes. If there are a lot of paths intermixed by a lot
of allow list, and few of black list path.
For example, if a system admin would like to allow mounting of
/mnt/hdfs/user/${USER} but not /mnt/hdfs/user/yarn using nfs gateway. White
list can not express the proper constraints alone. If white list has some
smarts to define if condition, black list might not be necessary.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176621#comment-16176621 ] Eric Yang edited comment on YARN-6626 at 9/22/17 3:59 PM: -- Fixed findbugs, unit test, and checkstyle issues. was (Author: eyang): Fixed findbugs issue. > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch, > YARN-6626.yarn-native-services.002.patch, > YARN-6626.yarn-native-services.003.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6626) Embed REST API service into RM
[ https://issues.apache.org/jira/browse/YARN-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6626: Attachment: YARN-6626.yarn-native-services.003.patch Fixed findbugs issue. > Embed REST API service into RM > -- > > Key: YARN-6626 > URL: https://issues.apache.org/jira/browse/YARN-6626 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > Attachments: YARN-6626.yarn-native-services.001.patch, > YARN-6626.yarn-native-services.002.patch, > YARN-6626.yarn-native-services.003.patch > > > As of now the deployment model of the Native Services REST API service is > standalone. There are several cross-cutting solutions that can be inherited > for free (kerberos, HA, ACLs, trusted proxy support, etc.) by the REST API > service if it is embedded into the RM process. In fact we can expose the REST > API via the same port as RM UI (8088 default). The URI path > /services/v1/applications will distinguish the REST API calls from other RM > APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.006.patch Correction to artifact image name. > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch, > YARN-7201.yarn-native-services.004.patch, > YARN-7201.yarn-native-services.005.patch, > YARN-7201.yarn-native-services.006.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.002.patch Remove organization, description, icon metadata field to match the current state of REST API. > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.003.patch Fix typo. > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.004.patch Remove mounts because YARN-4266 and YARN-7066 are not committed. > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch, > YARN-7201.yarn-native-services.004.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: (was: YARN-7201.yarn-native-services.004.patch) > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.004.patch > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch, > YARN-7201.yarn-native-services.004.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7201) Add more sophisticated example YARN service
[ https://issues.apache.org/jira/browse/YARN-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7201: Attachment: YARN-7201.yarn-native-services.005.patch Added example document. > Add more sophisticated example YARN service > --- > > Key: YARN-7201 > URL: https://issues.apache.org/jira/browse/YARN-7201 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7201.yarn-native-services.001.patch, > YARN-7201.yarn-native-services.002.patch, > YARN-7201.yarn-native-services.003.patch, > YARN-7201.yarn-native-services.004.patch, > YARN-7201.yarn-native-services.005.patch > > > We can show case the following capabilities in the YARN service examples: > # Description of the service > # Component dependencies > # How to mount HDFS volume via NFS Gateway > # Enable privileged container > # Quick link to web application > # Queue to submit > # Use private docker registry -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7215) REST API to list all deployed services by the same user
Eric Yang created YARN-7215: --- Summary: REST API to list all deployed services by the same user Key: YARN-7215 URL: https://issues.apache.org/jira/browse/YARN-7215 Project: Hadoop YARN Issue Type: Bug Components: api, applications Reporter: Eric Yang Assignee: Eric Yang In Slider, it is possible to list deployed applications from the same user by using: slider list This API can help UI to display application and services deployed by the same user. Apiserver does not have ability to list all applications/services at this time. This API requires fast response to list all applications because it is a common UI operation. ApiServer deployed applications persist configuration in HDFS similar to slider, but using directory listing to display deployed application might cost too much overhead to namenode. We may want to use alternative storage mechanism to cache deployed application configuration to accelerate the response time of list deployed applications. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7216) Missing ability to list configuration vs status
Eric Yang created YARN-7216:
---
Summary: Missing ability to list configuration vs status
Key: YARN-7216
URL: https://issues.apache.org/jira/browse/YARN-7216
Project: Hadoop YARN
Issue Type: Bug
Components: api, applications
Reporter: Eric Yang
Assignee: Eric Yang
API Server has /ws/v1/services/{service_name}. This REST end point returns
Services object which contains both configuration and status. When status or
macro based parameters changed in Services object, it can confuse UI code to
making configuration changes. The suggestion is to preserve a copy of
configuration object independent of status object. This gives UI ability to
change services configuration and update configuration.
Similar to Ambari, it might provide better information if we have the following
separated REST end points:
{code}
/ws/v1/services/{service_name}/config
/ws/v1/services/{service_name}/status
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: YARN-7202.yarn-native-services.008.patch Rebase patch by the changes Jian He made in YARN-6626. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7215:
Attachment: YARN-7215.yarn-native-services.002.patch
Rebase the patch based on Jian's last minute changes to YARN-6626 patch.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7215.yarn-native-services.001.patch,
> YARN-7215.yarn-native-services.002.patch
>
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7215:
Attachment: YARN-7215.yarn-native-services.003.patch
- Ensure solrj client is part of Hadoop distro tarball.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7215.yarn-native-services.001.patch,
> YARN-7215.yarn-native-services.002.patch,
> YARN-7215.yarn-native-services.003.patch
>
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188942#comment-16188942 ] Eric Yang commented on YARN-7202: - TestYarnNativeServices and TestApiServer combination provides end-to-end unit tests coverages. After reviewing HADOOP-9122, it doesn't appear that powermock can help in Hadoop unit test. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: YARN-7202.yarn-native-services.003.patch Revised negative test to work without powermock. Unfortunately, the way the ServiceClient is initialized, it requires an extra method to override ServiceClient for negative tests. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: YARN-7202.yarn-native-services.006.patch Fixed checkstyle errors. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7265) Hadoop Server Log Correlation
[ https://issues.apache.org/jira/browse/YARN-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190285#comment-16190285 ] Eric Yang commented on YARN-7265: - Chukwa has some degree of success in doing this. Chukwa 0.8.0 utilize SOLR to provide indexing feature. This allows user to search for unique text (i.e. Application ID), and all logs contains unique text are categorized and display as results. This includes user's container logs, hdfs (nn/dn), yarn (rm/nm) , and hbase (master/rs) log files. This exists as proof of concept. Chukwa docker image is available on Docker hub for testing. There is a lot work to productize it, but feel free to experiment with it. > Hadoop Server Log Correlation > --- > > Key: YARN-7265 > URL: https://issues.apache.org/jira/browse/YARN-7265 > Project: Hadoop YARN > Issue Type: Wish > Components: log-aggregation >Reporter: Tanping Wang > > Hadoop has many server logs, yarn tasks logs, node manager logs, HDFS logs.. > There are also a lot of different ways can be used to expose the logs, build > relationship horizontally to correlate the logs or search the logs by > keyword. There is a need for a default and yet convenient logging analytics > mechanism in Hadoop itself that at least covers all the server logs of > Hadoop. This log analytics system can correlate the Hadoop server logs by > grouping them by various dimensions including application ID, task ID, job > ID or node ID etc. The raw logs with correlation can be easily accessed by > the application developer or cluster administrator via web page for managing > and debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned YARN-7202: --- Assignee: Eric Yang Retrigger precommit test. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: YARN-7202.yarn-native-services.005.patch - Added ApiServer constructor without parameters. This version of constructor is used by Jersey to initialize REST API. It does not work without the base constructor. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: YARN-7202.yarn-native-services.004.patch - Added positive test case for REST API, ServiceClient gets exercised with create and flex. > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198976#comment-16198976
]
Eric Yang commented on YARN-7202:
-
[~jianhe] wrote:
{code}
Confused. I'm seeing the opposite.
Before the patch, it directly returns the status code based on whatever failure
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST. e.g. the
NOT_FOUND status code for stopService method is instead converted to the
BAD_REQUEST status code. Why is this converstion needed?
{code}
In ServiceClient, YarnException is used for invalid request, i.e. Service Name
not found, Application already exist, lost contact to AM, or Service definition
is not found in HDFS. There are many other exceptions overloaded using
YarnException and all of them are general error handling. We end up with web
response that are inconsistent between Web response code and the root cause,
such as 404 NOT_FOUND, and application already exist. This is the reason the
response code is generalized into BAD_REQUEST to avoid confusion.
> End-to-end UT for api-server
>
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch,
> YARN-7202.yarn-native-services.002.patch,
> YARN-7202.yarn-native-services.003.patch,
> YARN-7202.yarn-native-services.004.patch,
> YARN-7202.yarn-native-services.005.patch,
> YARN-7202.yarn-native-services.006.patch,
> YARN-7202.yarn-native-services.007.patch,
> YARN-7202.yarn-native-services.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7202) End-to-end UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198976#comment-16198976
]
Eric Yang edited comment on YARN-7202 at 10/10/17 5:20 PM:
---
[~jianhe] wrote:
{quote}
Confused. I'm seeing the opposite.
Before the patch, it directly returns the status code based on whatever failure
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST. e.g. the
NOT_FOUND status code for stopService method is instead converted to the
BAD_REQUEST status code. Why is this converstion needed?
{quote}
In ServiceClient, YarnException is used for invalid request, i.e. Service Name
not found, Application already exist, lost contact to AM, or Service definition
is not found in HDFS. There are many other exceptions overloaded using
YarnException and all of them are general error handling. We end up with web
response that are inconsistent between Web response code and the root cause,
such as 404 NOT_FOUND, and application already exist. This is the reason the
response code is generalized into BAD_REQUEST to avoid confusion.
was (Author: eyang):
[~jianhe] wrote:
{code}
Confused. I'm seeing the opposite.
Before the patch, it directly returns the status code based on whatever failure
conditions it is inside the method which is accurate.
After the patch, it only returns INTERNAL_SERVER_ERROR or BAD_REQUEST. e.g. the
NOT_FOUND status code for stopService method is instead converted to the
BAD_REQUEST status code. Why is this converstion needed?
{code}
In ServiceClient, YarnException is used for invalid request, i.e. Service Name
not found, Application already exist, lost contact to AM, or Service definition
is not found in HDFS. There are many other exceptions overloaded using
YarnException and all of them are general error handling. We end up with web
response that are inconsistent between Web response code and the root cause,
such as 404 NOT_FOUND, and application already exist. This is the reason the
response code is generalized into BAD_REQUEST to avoid confusion.
> End-to-end UT for api-server
>
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch,
> YARN-7202.yarn-native-services.002.patch,
> YARN-7202.yarn-native-services.003.patch,
> YARN-7202.yarn-native-services.004.patch,
> YARN-7202.yarn-native-services.005.patch,
> YARN-7202.yarn-native-services.006.patch,
> YARN-7202.yarn-native-services.007.patch,
> YARN-7202.yarn-native-services.008.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7202) End-to-end UT for api-server
[
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198881#comment-16198881
]
Eric Yang commented on YARN-7202:
-
[~jianhe] I took another look at the code from patch 08, and I have answers for
this question:
{quote}
confused, isn't this code not part of this patch ? I did clean before build.
Can you try compile only this patch with the dependency removed?
{quote}
I refactored TestYarnNativeServices to move the configuration initializer code
to ServiceTestUtils java class and move ServiceTestUtils java class to
org/apache/hadoop/yarn/service/client/ServiceTestUtils. This was done to
enhance the ability to reuse the utility class for YARN service setup for test
cases. TestApiService extends ServiceTestUtils for mini yarn cluster setup.
This is the reason that MiniHDFSCluster and MiniYARNCluster are both required
dependencies for the test to run. The scope of the dependencies is set to
"test". This is the reason that you were able to compile without the
dependencies expressed, they are used during test phase. Hence, both exception
throwing, and dependency expression are non-issues. Could you take another
look at patch 08? I deleted patch 09 and 10 because the decision that we made
are not correct in yesterday's meeting.
> End-to-end UT for api-server
>
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Attachments: YARN-7202.yarn-native-services.001.patch,
> YARN-7202.yarn-native-services.002.patch,
> YARN-7202.yarn-native-services.003.patch,
> YARN-7202.yarn-native-services.004.patch,
> YARN-7202.yarn-native-services.005.patch,
> YARN-7202.yarn-native-services.006.patch,
> YARN-7202.yarn-native-services.007.patch,
> YARN-7202.yarn-native-services.008.patch,
> YARN-7202.yarn-native-services.009.patch,
> YARN-7202.yarn-native-services.010.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: (was: YARN-7202.yarn-native-services.009.patch) > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7202) End-to-end UT for api-server
[ https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7202: Attachment: (was: YARN-7202.yarn-native-services.010.patch) > End-to-end UT for api-server > > > Key: YARN-7202 > URL: https://issues.apache.org/jira/browse/YARN-7202 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Eric Yang > Attachments: YARN-7202.yarn-native-services.001.patch, > YARN-7202.yarn-native-services.002.patch, > YARN-7202.yarn-native-services.003.patch, > YARN-7202.yarn-native-services.004.patch, > YARN-7202.yarn-native-services.005.patch, > YARN-7202.yarn-native-services.006.patch, > YARN-7202.yarn-native-services.007.patch, > YARN-7202.yarn-native-services.008.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204985#comment-16204985 ] Eric Yang commented on YARN-7066: - [~shaneku...@gmail.com] Sorry, I did not know YARN-5534 already includes user defined mount. I probably should have read the patch before this was opened. We can close this as a dupe. Thanks > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > Attachments: YARN-7066.001.patch, YARN-7066.002.patch > > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned YARN-7066: --- Resolution: Duplicate Assignee: Eric Yang > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7066.001.patch, YARN-7066.002.patch > > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7216) Missing ability to list configuration vs status
[
https://issues.apache.org/jira/browse/YARN-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7216:
Attachment: (was: YARN-7216.yarn-native-services.001.patch)
> Missing ability to list configuration vs status
> ---
>
> Key: YARN-7216
> URL: https://issues.apache.org/jira/browse/YARN-7216
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7216.yarn-native-services.001.patch
>
>
> API Server has /ws/v1/services/{service_name}. This REST end point returns
> Services object which contains both configuration and status. When status or
> macro based parameters changed in Services object, it can confuse UI code to
> making configuration changes. The suggestion is to preserve a copy of
> configuration object independent of status object. This gives UI ability to
> change services configuration and update configuration.
> Similar to Ambari, it might provide better information if we have the
> following separated REST end points:
> {code}
> /ws/v1/services/[service_name]/spec
> /ws/v1/services/[service_name]/status
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193974#comment-16193974 ] Eric Yang commented on YARN-6623: - [~ebadger] In some secure environment, I have seen that container-executor.cfg is set to non-world readable because the administrator doesn't want people to know about the allowed and bannded users on the cluster. Another possibility is the default umask are set to 027, when admin generates container-executor.cfg via Ambari. it is often set to world non-readable for secure environment. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch, > YARN-6623.006.patch, YARN-6623.007.patch, YARN-6623.008.patch, > YARN-6623.009.patch, YARN-6623.010.patch, YARN-6623.011.patch, > YARN-6623.012.patch, YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7216) Missing ability to list configuration vs status
[
https://issues.apache.org/jira/browse/YARN-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7216:
Attachment: YARN-7216.yarn-native-services.001.patch
> Missing ability to list configuration vs status
> ---
>
> Key: YARN-7216
> URL: https://issues.apache.org/jira/browse/YARN-7216
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7216.yarn-native-services.001.patch
>
>
> API Server has /ws/v1/services/{service_name}. This REST end point returns
> Services object which contains both configuration and status. When status or
> macro based parameters changed in Services object, it can confuse UI code to
> making configuration changes. The suggestion is to preserve a copy of
> configuration object independent of status object. This gives UI ability to
> change services configuration and update configuration.
> Similar to Ambari, it might provide better information if we have the
> following separated REST end points:
> {code}
> /ws/v1/services/[service_name]/spec
> /ws/v1/services/[service_name]/status
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7215) REST API to list all deployed services by the same user
[
https://issues.apache.org/jira/browse/YARN-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7215:
Attachment: YARN-7215.yarn-native-services.001.patch
- Implemented ability to list services deployed by the same user.
- This patch depends on patch from YARN-7216.
> REST API to list all deployed services by the same user
> ---
>
> Key: YARN-7215
> URL: https://issues.apache.org/jira/browse/YARN-7215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7215.yarn-native-services.001.patch
>
>
> In Slider, it is possible to list deployed applications from the same user by
> using:
> {code}
> slider list
> {code}
> This API can help UI to display application and services deployed by the same
> user.
> Apiserver does not have ability to list all applications/services at this
> time. This API requires fast response to list all applications because it is
> a common UI operation. ApiServer deployed applications persist configuration
> in HDFS similar to slider, but using directory listing to display deployed
> application might cost too much overhead to namenode. We may want to use
> alternative storage mechanism to cache deployed application configuration to
> accelerate the response time of list deployed applications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206108#comment-16206108
]
Eric Yang commented on YARN-7217:
-
The unit tests failure are not related to this patch. It looks like the recent
commit or rebase yarn-native-services branch broked yarn-native-services branch.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206108#comment-16206108
]
Eric Yang edited comment on YARN-7217 at 10/16/17 4:59 PM:
---
The recent yarn-native-services branch rebased to trunk, and this patch needs
to rebase to the current code on yarn-native-services branch.
was (Author: eyang):
The unit tests failure are not related to this patch. It looks like the recent
commit or rebase yarn-native-services branch broked yarn-native-services branch.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.
[ https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207080#comment-16207080 ] Eric Yang commented on YARN-7338: - Hi [~sunilg], If the new UI has ability to view container execution log, then it is possible that javascript output in the log cause browser to execute unauthorized cross site scripts. There are two requirements to secure javascript. # Set document.domain property in javascript # Set Access-Control-Allow-Origin: http://[hostname]/ws/v1/ The first javascript property is to make sure the javascript only evaluates ajax call to the same origin. The second header is respond by server to ensure that access control can only be coming from the same origin. It is likely that Access-Control-Allow-Origin needs to be set to a narrow list of /ws/v1/ prefix to be secured. > Support same origin policy for cross site scripting prevention. > --- > > Key: YARN-7338 > URL: https://issues.apache.org/jira/browse/YARN-7338 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Vrushali C > > Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new > web UI) to branch2 > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E > -- > Ui2 does not seem to support same origin policy for cross site scripting > prevention. > The following parameters has no effect for /ui2: > hadoop.http.cross-origin.enabled = true > yarn.resourcemanager.webapp.cross-origin.enabled = true > This is because ui2 is designed as a separate web application. WebFilters > setup for existing resource manager doesn’t apply to the new web application. > Please open JIRA to track the security issue and resolve the problem prior to > backporting this to branch-2. > This would minimize the risk to open up security hole in branch-2. > -- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7344) Unit test for white list mount fails on CentOS 7
Eric Yang created YARN-7344:
---
Summary: Unit test for white list mount fails on CentOS 7
Key: YARN-7344
URL: https://issues.apache.org/jira/browse/YARN-7344
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0-beta1
Environment: CentOS Linux release 7.4.1708 (Core)
cmake3-3.6.3-1.el7.x86_64
openjdk version "1.8.0_144"
OpenJDK Runtime Environment (build 1.8.0_144-b01)
OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode)
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
2017-04-03T15:39:06-04:00)
libprotoc 2.5.0
Reporter: Eric Yang
YARN-6623 introduced ability to turn off docker support for container executor.
When running C++ unit tests, the newly introduced tests failed.
{code}
[ FAILED ] 3 tests, listed below:
[ FAILED ] TestDockerUtil.test_check_mount_permitted
[ FAILED ] TestDockerUtil.test_normalize_mounts
[ FAILED ] TestDockerUtil.test_add_rw_mounts
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7323) Data structure update in service REST API
[ https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7323: Description: Unix terminology implies daemon process, such as httpd as a service. In Hadoop terminology, hdfs is a file system service, and hdfs has various components such as namenode, journal nodes, and datanodes. YARN service data model was more closely describing daemon process as a service. This JIRA propose to remap the YARN service data model to align with Hadoop terminology. Service is composed of a collection of components. Component is either a unix process or docker container. Data structure for Yarnfile organized launched_command and state at Service level, but the grouping definition was following Hadoop terminology. For clarity, we will change data structure to match Hadoop terminology. State and launch commands are moved to component level. was:Data structure for Yarnfile organized state at service level. > Data structure update in service REST API > - > > Key: YARN-7323 > URL: https://issues.apache.org/jira/browse/YARN-7323 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7323.yarn-native-services.01.patch, > YARN-7323.yarn-native-services.02.patch > > > Unix terminology implies daemon process, such as httpd as a service. In > Hadoop terminology, hdfs is a file system service, and hdfs has various > components such as namenode, journal nodes, and datanodes. YARN service data > model was more closely describing daemon process as a service. This JIRA > propose to remap the YARN service data model to align with Hadoop > terminology. Service is composed of a collection of components. Component > is either a unix process or docker container. > > Data structure for Yarnfile organized launched_command and state at Service > level, but the grouping definition was following Hadoop terminology. For > clarity, we will change data structure to match Hadoop terminology. State > and launch commands are moved to component level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7326) Some issues in RegistryDNS
[
https://issues.apache.org/jira/browse/YARN-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204181#comment-16204181
]
Eric Yang commented on YARN-7326:
-
Sounds like dns java is misconfigured to use itself as the upstream dns server.
Hence, query runs into a infinite loop. When setting up DNS, the code
probably can look at /etc/resolv.conf and filter out its own IP address, then
use the rest of the servers as upstream dns servers.
> Some issues in RegistryDNS
> --
>
> Key: YARN-7326
> URL: https://issues.apache.org/jira/browse/YARN-7326
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
>
> [~aw] helped to identify these issues:
> Now some general bad news, not related to this patch:
> Ran a few queries, but this one is a bit concerning:
> {code}
> root@ubuntu:/hadoop/logs# dig @localhost -p 54 .
> ;; Warning: query response not set
> ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @localhost -p 54 .
> ; (2 servers found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOTAUTH, id: 47794
> ;; flags: rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
> ;; WARNING: recursion requested but not available
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#54(127.0.0.1)
> ;; WHEN: Thu Oct 12 16:04:54 PDT 2017
> ;; MSG SIZE rcvd: 12
> root@ubuntu:/hadoop/logs# dig @localhost -p 54 axfr .
> ;; Connection to ::1#54(::1) for . failed: connection refused.
> ;; communications error to 127.0.0.1#54: end of file
> root@ubuntu:/hadoop/logs#
> {code}
> It looks like it effectively fails when asked about a root zone, which is bad.
> It's also kind of interesting in what it does and doesn't log. Probably
> should be configured to rotate logs based on size not date.
> The real showstopper though: RegistryDNS basically eats a core. It is running
> with 100% cpu utilization with and without jsvc. On my laptop, this is
> triggering my fan.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7217:
Attachment: YARN-7217.yarn-native-services.001.patch
This patch modifies the PUT method to have /ws/v1/services/[service_name]/spec
and /ws/v1/services/[service_name]/status.
This patch contains work for YARN-7215 and YARN-7216. For changing number of
containers with in a service, the api call must invoke:
# Update service configuration
# Stop existing service
# Start service
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned YARN-7217:
---
Assignee: Eric Yang
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204321#comment-16204321 ] Eric Yang commented on YARN-7066: - [~ebadger] Security restriction will be enforced by: # Check for sudo privileges for launching privileged container (YARN-7221) # Enforced effective uid:gid (YARN-4266) # Black listed volume (YARN-7197) # Allowed white list volume (YARN-5534) For privileged users, there is minimum restrictions. For unprivileged user, they can express path to mount, but they will be blocked to unauthorized area or by their own uid:gid privileges to file system ACL. When the listed security defects are solved, this feature will be as good as accessing local file system ACL. > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > Attachments: YARN-7066.001.patch, YARN-7066.002.patch > > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7327) CapacityScheduler: Allocate containers asynchronously by default
[ https://issues.apache.org/jira/browse/YARN-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204359#comment-16204359 ] Eric Yang commented on YARN-7327: - The purpose of the artificial synchronous delay was to ensure all nodes have the same opportunity to talk to Jobtracker or Application Master. This ensures there is somewhat evenly distribution of workload on all nodes instead of faster node end up with majority of data. With faster network, and lower latency, it might be reasonable to shorten the heartbeat frequency to improve container allocation response time. Asynchronous allocation will guarantee that fastest node end up with most data. I don't have preference in the default, but large scale cluster is likely to use synchronous delay to prevent large skew of data in cluster. > CapacityScheduler: Allocate containers asynchronously by default > > > Key: YARN-7327 > URL: https://issues.apache.org/jira/browse/YARN-7327 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Craig Ingram >Priority: Trivial > Attachments: yarn-async-scheduling.png > > > I was recently doing some research into Spark on YARN's startup time and > observed slow, synchronous allocation of containers/executors. I am testing > on a 4 node bare metal cluster w/48 cores and 128GB memory per node. YARN was > only allocating about 3 containers per second. Moreover when starting 3 Spark > applications at the same time with each requesting 44 containers, the first > application would get all 44 requested containers and then the next > application would start getting containers and so on. > > From looking at the code, it appears this is by design. There is an > undocumented configuration variable that will enable asynchronous allocation > of containers. I'm sure I'm missing something, but why is this not the > default? Is there a bug or race condition in this code path? I've done some > testing with it and it's been working and is significantly faster. > > Here's the config: > `yarn.scheduler.capacity.schedule-asynchronously.enable` > > Any help understanding this would be appreciated. > > Thanks, > Craig > > If you're curious about the performance difference with this setting, here > are the results: > > The following tool was used for the benchmarks: > https://github.com/SparkTC/spark-bench > h2. async scheduler research > The goal of this test is to determine if running Spark on YARN with async > scheduling of containers reduces the amount of time required for an > application to receive all of its requested resources. This setting should > also reduce the overall runtime of short-lived applications/stages or > notebook paragraphs. This setting could prove crucial to achieving optimal > performance when sharing resources on a cluster with dynalloc enabled. > h3. Test Setup > Must update /etc/hadoop/conf/capacity-scheduler.xml (or through Ambari) > between runs. > `yarn.scheduler.capacity.schedule-asynchronously.enable=true|false` > conf files request executors counts of: > * 2 > * 20 > * 50 > * 100 > The apps are being submitted to the default queue on each cluster which caps > at 48 cores on dynalloc and 72 cores on baremetal. The default queue was > expanded for the last two tests on baremetal so it could potentially take > advantage of all 144 cores. > h3. Test Environments > h4. dynalloc > 4 VMs in Fyre (1 master, 3 workers) > 8 CPUs/16 GB per node > model name: QEMU Virtual CPU version 2.5+ > h4. baremetal > 4 baremetal instances in Fyre (1 master, 3 workers) > 48 CPUs/128GB per node > model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz > h3. Using spark-bench with timedsleep workload sync > h4. dynalloc > || requested containers | avg | stdev|| > |2 | 23.814900 | 1.110725| > |20 | 29.770250 | 0.830528| > |50 | 44.486600 | 0.593516| > |100 | 44.337700 | 0.490139| > h4. baremetal - 2 queues splitting cluster 72 cores each > || requested containers | avg | stdev|| > |2 | 14.827000 | 0.292290| > |20 | 19.613150 | 0.155421| > |50 | 30.768400 | 0.083400| > |100 | 40.931850 | 0.092160| > h4. baremetal - 1 queue to rule them all - 144 cores > || requested containers | avg | stdev|| > |2 | 14.833050 | 0.334061| > |20 | 19.575000 | 0.212836| > |50 | 30.765350 | 0.111035| > |100 | 41.763300 | 0.182700| > h3. Using spark-bench with timedsleep workload async > h4. dynalloc > || requested containers | avg | stdev|| > |2 | 22.575150 | 0.574296| > |20 | 26.904150 | 1.244602| > |50 | 44.721800 | 0.655388| > |100 | 44.57 | 0.514540| > h5. 2nd run > || requested containers | avg | stdev|| > |2 | 22.441200 | 0.715875| > |20 | 26.683400 | 0.583762| > |50 | 44.227250 | 0.512568| > |100 | 44.238750 | 0.329712| > h4. baremetal - 2 queues splitting cluster 72
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204375#comment-16204375
]
Eric Yang commented on YARN-7217:
-
[~gsaha]
{quote}
What do you mean by it does not increase the count? Are you saying there is a
bug in the code, or are you saying that it should not increase container count?
{quote}
There is a bug in the code.
{quote}
Modifying service config for a service in STOPPED state is not supported via
the REST API. It is supported in classic Slider though. Are you saying you want
to support this in REST API?
{quote}
Yes, pause application should be supported in the REST API. It is a valid use
case to allow pause of the application by running slider stop, and resume the
application later with a different YARN application ID,andt brand new set of
containers.
{quote}
Do you intend to support GET for /ws/v1/services/[service_name]/spec as well?
If yes, then for a STARTED app, what will GET return - the spec corresponding
to the current snapshot of the running app OR the saved spec which might be
more recent and has not been reflected to the running state yet? For STOPPED
app this is not a problem.
{quote}
Yes, GET is also implemented in the current patch. I incorporated patch for
YARN-7216 in this patch to reduce logistic work.
{quote}
Also, the patch contains solr specific changes and cannot be reviewed in its
current state.
{quote}
The patch allows the configuration to be stored in Solr for faster search and
retrieval time. There is a on/off flag to use Slor as backend. When solr
storage backend is disabled (default), it will look for spec in HDFS.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7323) Data structure update in service REST API
[ https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7323: Description: Data structure for Yarnfile organized state at service level. > Data structure update in service REST API > - > > Key: YARN-7323 > URL: https://issues.apache.org/jira/browse/YARN-7323 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7323.yarn-native-services.01.patch, > YARN-7323.yarn-native-services.02.patch > > > Data structure for Yarnfile organized state at service level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7323) Data structure update in service REST API
[ https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7323: Summary: Data structure update in service REST API (was: Some changes in service REST API) > Data structure update in service REST API > - > > Key: YARN-7323 > URL: https://issues.apache.org/jira/browse/YARN-7323 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7323.yarn-native-services.01.patch, > YARN-7323.yarn-native-services.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204321#comment-16204321 ] Eric Yang edited comment on YARN-7066 at 10/13/17 11:08 PM: [~ebadger] Security restriction will be enforced by: # Check for sudo privileges for launching privileged container (YARN-7221) # Enforced effective uid:gid (YARN-4266) # Black listed volume (YARN-7197) # Allowed white list volume (YARN-5534) For privileged users, there is minimum restrictions. For unprivileged users, they can express path to mount, but they will be blocked to unauthorized area or by their own uid:gid privileges to file system ACL. When the listed security defects are solved, this feature will be as good as accessing local file system ACL. was (Author: eyang): [~ebadger] Security restriction will be enforced by: # Check for sudo privileges for launching privileged container (YARN-7221) # Enforced effective uid:gid (YARN-4266) # Black listed volume (YARN-7197) # Allowed white list volume (YARN-5534) For privileged users, there is minimum restrictions. For unprivileged user, they can express path to mount, but they will be blocked to unauthorized area or by their own uid:gid privileges to file system ACL. When the listed security defects are solved, this feature will be as good as accessing local file system ACL. > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > Attachments: YARN-7066.001.patch, YARN-7066.002.patch > > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7066) Add ability to specify volumes to mount for DockerContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7066: Attachment: YARN-7066.002.patch Rebase patch on trunk code. > Add ability to specify volumes to mount for DockerContainerRuntime > -- > > Key: YARN-7066 > URL: https://issues.apache.org/jira/browse/YARN-7066 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Affects Versions: 3.0.0-beta1 >Reporter: Eric Yang > Attachments: YARN-7066.001.patch, YARN-7066.002.patch > > > Yarnfile describes environment, docker image, and configuration template for > launching docker containers in YARN. It would be nice to have ability to > specify the volumes to mount. This can be used in combination to > AMBARI-21748 to mount HDFS as data directories to docker containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206736#comment-16206736 ] Eric Yang commented on YARN-6623: - [~vvasudev] How is the C++ test case getting triggered? mvn clean test -Pnative doesn't trigger test_docker_util.cc to run. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-6623: Attachment: cetest.stderr cetest.stdout [~vvasudev] Here are the output of the failed test. Any idea why the tests fail? > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Fix For: 3.0.0 > > Attachments: YARN-6623-branch-2.013.patch, > YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, > YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, > YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, > YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, > YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, > YARN-6623.013.patch, cetest.stderr, cetest.stdout > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206761#comment-16206761
]
Eric Yang commented on YARN-6623:
-
Talked to [~wangda] and he said the test can be triggered using:
{code}
-Dtest=cetest,test-container-executor -Pnative
{code}
To the maven command. It would be nice if it is covered by default test
without passing in test case name.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch,
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch,
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch,
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch,
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch,
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch,
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206764#comment-16206764
]
Eric Yang commented on YARN-6623:
-
The current trunk seems to fail on these test cases:
{code}
[ FAILED ] 3 tests, listed below:
[ FAILED ] TestDockerUtil.test_check_mount_permitted
[ FAILED ] TestDockerUtil.test_normalize_mounts
[ FAILED ] TestDockerUtil.test_add_rw_mounts
{code}
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch,
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch,
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch,
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch,
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch,
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch,
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7217:
Attachment: YARN-7217.yarn-native-services.002.patch
- Rebase patch on current HEAD of yarn-native-services branch.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7197: Attachment: YARN-7197.001.patch Basic check for black list requires exact match of path to ban. The evaluation order is white list first, then black list. This is assuming majority of the volumes can be mounted, and there are only a few system protected area, which can not be mounted. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf > Attachments: YARN-7197.001.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned YARN-7197: --- Assignee: Eric Yang > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
