[
https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250312#comment-16250312
]
Eric Yang edited comment on YARN-7430 at 11/13/17 9:59 PM:
-----------------------------------------------------------
[~ebadger] {quote}
I could be wrong, but I don't think Shane Kumpf's question is whether this
happens in docker, but whether this is possible to happen in hadoop. i.e. can
the above docker run (or similar) ever actually be created by the NM starting
up a docker container.
{quote}
Yes, it's possible in Hadoop. Here is the container log for application master:
{code}
2017-11-13 21:24:57,440 [pool-5-thread-4] INFO
registry.YarnRegistryViewForProviders - [COMPINSTANCE kafkabroker-0 :
container_1510599241403_0006_01_000002]: Deleting registry path
/users/1234/services/yarn-service/amp/components/ctr-1510599241403-0006-01-000002
2017-11-13 21:24:57,441 [CompInstance dispatcher] INFO
instance.ComponentInstance - [COMPINSTANCE kafkabroker-0] Transitioned from
STARTED to INIT on STOP event
2017-11-13 21:24:59,449 [AMRM Callback Handler Thread] INFO
service.ServiceScheduler - 1 containers allocated.
2017-11-13 21:24:59,450 [AMRM Callback Handler Thread] INFO
service.ServiceScheduler - [COMPONENT kafkabroker]: 1 outstanding container
requests.
2017-11-13 21:24:59,450 [AMRM Callback Handler Thread] INFO
service.ServiceScheduler - [COMPONENT kafkabroker]: removing one container
request.
2017-11-13 21:24:59,450 [Component dispatcher] INFO component.Component -
[COMPONENT kafkabroker]: container_1510599241403_0006_01_000003 allocated, num
pending component instances reduced to 0
2017-11-13 21:24:59,450 [Component dispatcher] INFO component.Component -
[COMPONENT kafkabroker]: Assigned container_1510599241403_0006_01_000003 to
component instance kafkabroker-0 and launch on host
eyang-5.openstacklocal:34611
2017-11-13 21:24:59,454 [pool-6-thread-1] INFO provider.ProviderUtils -
Component instance conf dir already exists:
hdfs://eyang-1:9000/user/1234/.yarn/services/amp/components/kafkabroker/kafkabroker-0
2017-11-13 21:24:59,466 [pool-6-thread-1] INFO provider.ProviderUtils - Add
config file for localization: conf/server.properties ->
/user/1234/.yarn/services/amp/components/kafkabroker/kafkabroker-0/server.properties,
dest mount path: /etc/kafka/conf/server.properties
2017-11-13 21:24:59,467 [pool-6-thread-1] INFO
containerlaunch.AbstractLauncher - yarn docker env var has been set
{LANGUAGE=en_US.UTF-8, HADOOP_USER_NAME=1234,
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_HOSTNAME=kafkabroker-0.amp.1234,
WORK_DIR=$PWD, LC_ALL=en_US.UTF-8,
YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS=conf/server.properties:/etc/kafka/conf/server.properties,
YARN_CONTAINER_RUNTIME_TYPE=docker,
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=registry.eng.hortonworks.com/hwx-assemblies/kafka:0.10.1,
LANG=en_US.UTF-8, YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=bridge,
LOG_DIR=<LOG_DIR>, YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=false}
2017-11-13 21:24:59,468
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] INFO
impl.NMClientAsyncImpl - Processing Event EventType: START_CONTAINER for
Container container_1510599241403_0006_01_000003
2017-11-13 21:24:59,487 [CompInstance dispatcher] INFO
instance.ComponentInstance - [COMPINSTANCE kafkabroker-0 :
container_1510599241403_0006_01_000003] Transitioned from INIT to STARTED on
START event
2017-11-13 21:25:00,458 [Component dispatcher] INFO component.Component -
[COMPONENT kafkabroker]: container_1510599241403_0006_01_000003 completed, num
pending comp instances increased to 1.
{code}
And screenshot attached in the attachment section.
was (Author: eyang):
[~ebadger] {quote}
I could be wrong, but I don't think Shane Kumpf's question is whether this
happens in docker, but whether this is possible to happen in hadoop. i.e. can
the above docker run (or similar) ever actually be created by the NM starting
up a docker container.
{quote}
Yes, it's possible in Hadoop. Here is the container log for application master:
{code}
2017-11-13 21:24:57,440 [pool-5-thread-4] INFO
registry.YarnRegistryViewForProviders - [COMPINSTANCE kafkabroker-0 :
container_1510599241403_0006_01_000002]: Deleting registry path
/users/1234/services/yarn-service/amp/components/ctr-1510599241403-0006-01-000002
2017-11-13 21:24:57,441 [CompInstance dispatcher] INFO
instance.ComponentInstance - [COMPINSTANCE kafkabroker-0] Transitioned from
STARTED to INIT on STOP event
2017-11-13 21:24:59,449 [AMRM Callback Handler Thread] INFO
service.ServiceScheduler - 1 containers allocated.
2017-11-13 21:24:59,450 [AMRM Callback Handler Thread] INFO
service.ServiceScheduler - [COMPONENT kafkabroker]: 1 outstanding container
requests.
2017-11-13 21:24:59,450 [AMRM Callback Handler Thread] INFO
service.ServiceScheduler - [COMPONENT kafkabroker]: removing one container
request.
2017-11-13 21:24:59,450 [Component dispatcher] INFO component.Component -
[COMPONENT kafkabroker]: container_1510599241403_0006_01_000003 allocated, num
pending component instances reduced to 0
2017-11-13 21:24:59,450 [Component dispatcher] INFO component.Component -
[COMPONENT kafkabroker]: Assigned container_1510599241403_0006_01_000003 to
component instance kafkabroker-0 and launch on host
eyang-5.openstacklocal:34611
2017-11-13 21:24:59,454 [pool-6-thread-1] INFO provider.ProviderUtils -
Component instance conf dir already exists:
hdfs://eyang-1:9000/user/1234/.yarn/services/amp/components/kafkabroker/kafkabroker-0
2017-11-13 21:24:59,466 [pool-6-thread-1] INFO provider.ProviderUtils - Add
config file for localization: conf/server.properties ->
/user/1234/.yarn/services/amp/components/kafkabroker/kafkabroker-0/server.properties,
dest mount path: /etc/kafka/conf/server.properties
2017-11-13 21:24:59,467 [pool-6-thread-1] INFO
containerlaunch.AbstractLauncher - yarn docker env var has been set
{LANGUAGE=en_US.UTF-8, HADOOP_USER_NAME=1234,
YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_HOSTNAME=kafkabroker-0.amp.1234,
WORK_DIR=$PWD, LC_ALL=en_US.UTF-8,
YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS=conf/server.properties:/etc/kafka/conf/server.properties,
YARN_CONTAINER_RUNTIME_TYPE=docker,
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=registry.eng.hortonworks.com/hwx-assemblies/kafka:0.10.1,
LANG=en_US.UTF-8, YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=bridge,
LOG_DIR=<LOG_DIR>, YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=false}
2017-11-13 21:24:59,468
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1] INFO
impl.NMClientAsyncImpl - Processing Event EventType: START_CONTAINER for
Container container_1510599241403_0006_01_000003
2017-11-13 21:24:59,487 [CompInstance dispatcher] INFO
instance.ComponentInstance - [COMPINSTANCE kafkabroker-0 :
container_1510599241403_0006_01_000003] Transitioned from INIT to STARTED on
START event
2017-11-13 21:25:00,458 [Component dispatcher] INFO component.Component -
[COMPONENT kafkabroker]: container_1510599241403_0006_01_000003 completed, num
pending comp instances increased to 1.
{code}
And screenshot attached in the attachement section.
> User and Group mapping are incorrect in docker container
> --------------------------------------------------------
>
> Key: YARN-7430
> URL: https://issues.apache.org/jira/browse/YARN-7430
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: security, yarn
> Affects Versions: 2.9.0, 3.0.0
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Blocker
> Attachments: YARN-7430.001.patch, YARN-7430.png
>
>
> In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to
> enforce user and group for the running user. In YARN-6623, this translated
> to --user=test --group-add=group1. The code no longer enforce group
> correctly for launched process.
> In addition, the implementation in YARN-6623 requires the user and group
> information to exist in container to translate username and group to uid/gid.
> For users on LDAP, there is no good way to populate container with user and
> group information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
