[
https://issues.apache.org/jira/browse/YARN-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580751#comment-16580751
]
Rohith Sharma K S commented on YARN-8667:
-----------------------------------------
Container Relaunch shares same working directory. As a result, launch container
script create a symlink which is already created leading this issue. In order
to debug the issue, follow the below steps
# launch a sleeper service with below spec. Note that I am providing TAR BALL
artifacts
{code}
curl --negotiate -u: -H "Content-Type: application/json" -X POST
http://localhost:8088/app/v1/services?user.name=yarn-ats -d '{
"name": "sleeper",
"version": "1.0.0",
"queue": "default",
"artifact": {
"id": "/mapreduce/mapreduce.tar.gz",
"type": "TARBALL"
},
"components" :
[
{
"name": "sleeper1",
"number_of_containers": 1,
"launch_command": "sleep infinity",
"resource": {
"cpus": 1,
"memory": "2048"
}
}
]
}'
{code}
# After sleeper service is launched, goto working directory of container. There
you see below files
{noformat}
[root@ctr-e138-1518143905142-431547-01-000004
container_e04_1534244457405_0004_01_000002]# ll
total 24
-rw-r--r-- 1 yarn hadoop 7 Aug 15 05:47 container_tokens
-rwx------ 1 yarn hadoop 656 Aug 15 05:47 default_container_executor_session.sh
-rwx------ 1 yarn hadoop 711 Aug 15 05:47 default_container_executor.sh
-rwx------ 1 yarn hadoop 3817 Aug 15 05:47 launch_container.sh
lrwxrwxrwx 1 yarn hadoop 107 Aug 15 05:47 lib ->
/hadoop/yarn/local/usercache/yarn-ats/appcache/application_1534244457405_0004/filecache/10/mapreduce.tar.gz
drwx--x--- 2 yarn hadoop 4096 Aug 15 05:47 tmp
{noformat}
# You can try to execute launch_container.sh again manually which fails with
below error.
{code:}
find: File system loop detected; ‘./lib/mapreduce.tar.gz’ is part of the same
file system loop as ‘./lib’.{code}
In container relaunch also the same working directory is shared which executes
launch_container.sh again. This is causing an error which terminates
launch_container.sh with error code 1
> Container Relaunch fails with "find: File system loop detected;" for tar ball
> artifacts
> ---------------------------------------------------------------------------------------
>
> Key: YARN-8667
> URL: https://issues.apache.org/jira/browse/YARN-8667
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Rohith Sharma K S
> Priority: Major
>
> Service is launched with TAR BALL artifacts. If a container is exited due to
> any reasons, container relaunch policy try to relaunch the container on same
> node with same container work space. As a result, container relaunch is keep
> on failing.
> If container relaunch max-retry policy is disabled, then container never
> launched in any other nodes also rather it keep on retrying on same node
> manager which never succeeds.
> {code}
> Relaunching Container container_e05_1533635581781_0001_01_000002. Remaining
> retry attempts(after relaunch) : -4816.
> {code}
> There are two issues
> # Container relaunch is keep on failing
> # Log message is misleading
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
