[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537672#comment-16537672
]
Jason Lowe commented on HADOOP-15528:
-
It may be useful to have a config disable for this given it could cause
difficulties, but it'd also be nice if we could avoid users shooting themselves
in the foot with this config. If we know using this config with some container
executors makes no sense then it'd be nice to either fail fast on NM startup,
warn it's being ignored, or otherwise do something smarter than just failing
every container execution in a difficult to debug manner.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537498#comment-16537498
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
[~jlowe] thanks for the comment.
You have a good point. I am going to add a boolean configuration to enable the
execution of symlink from NM or fallback to the batch script.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535409#comment-16535409
]
Jason Lowe commented on HADOOP-15528:
-
Sorry for the delay in replying, as I recently got back from an extended
vacation and am catching up on things.
bq. However, the new behavior is the symlink operation is executed by NM
itself, which is executed as a child process under NM itself, it shares the
same execution environment as NM.
This cannot work in a secure environment. Well at least the one we have today
on Linux with the native container executor. In that secure environment the
container is running as the user and therefore has access to things that the NM
user does not. The container working directory is one of those things.
Normally the NM user has no need or reason to be able to see the contents of
the container working directory nor be able to modify it.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535317#comment-16535317
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
Thanks [~yqwang] for the comments.
Totally agree with you overall.
# We need to add security checks. We can implement the security check in a way
that we are way more secure than the old behavior. As [~ste...@apache.org] said
we should improve security to get more approval from OSS community.
{noformat}
The old behavior is the symlink operation is executed in the batch script,
which is executed as a child process under some limited privileged and resource
isolation environment, such as windows job object (with windows secure
container) or linux cgroups, etc.
However, the new behavior is the symlink operation is executed by NM itself,
which is executed as a child process under NM itself, it shares the same
execution environment as NM.{noformat}
In the old behavior, even if we run symlink with limited privileges we still do
not check the content of CLC.
In the new implementation, I am planning to try to restrict the privileges and
add checks on the content of CLC.
# As I said before, in case of error we should avoid starting the container. I
would like to still keep the old behavior, aka start the container and exit
with a better error log. Let me try to improve the exit message.
# The old behavior does not allow to retry. We can add a retry logic in a
future Jira as an improvement.
I don't know if we should work for the security aspects in this Jira or in next
jira(s).
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534409#comment-16534409
]
Yuqi Wang commented on HADOOP-15528:
[~giovanni.fumarola], [~elgoiri],
Some concerns from my side (please correct me if I am wrong), please take a
look:
# *Maybe Security and Resource Isolation Leak:***
The *old behavior* is the symlink operation is *executed in the batch script*,
which is executed as a child process under some limited privileged and resource
isolation environment, such as windows job object (with windows secure
container) or linux cgroups, etc.
However, the *new behavior* is the symlink operation is executed by NM itself,
which is executed as a child process under NM itself, it shares the same
execution environment as NM.
So, I worry about there may be some leak for Security, Resource Isolation, etc.
# *Exit procedure is not straightforward and exit info is too less to debug.*
For the PATCH implementation:
It execute the symlink operation before container starts. If fails, it just
record a "exit XXX" in batch script instead of throw the failure to its caller.
So, even if you execute symlink before container starts, but the fail will not
be propagated outside until the container starts.
So, if I try to debug a container failure, I will see there is a sudden "exit
XXX" in the batch script without any other info for why NM add this line there.
I hope we can make the execution and propagate exit status in the same
execution environment, instead of split them into different. The old behavior
is all in batch script. But the new behavior split them into NM and batch
script.
# *Better to have Retry:*
For the PATCH implementation:
A symlink error from container launch caller should be a transient error, so
you will also need to add the corresponding symlink failure exitcode into
shouldCountTowardsMaxAttemptRetry. So RM will always retry the AM container in
face of symlink error.
Overall, at least for the PATCH implementation, I did not see any benefits.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532933#comment-16532933
]
Steve Loughran commented on HADOOP-15528:
-
Talk to the YARN people; security matters as much as perf here. If you can show
this is more secure too, expect enthusiasm
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531835#comment-16531835
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
{noformat}
To make it clear, this comes from switching from winutils symlink to
FileUtil#symLink in ContainerLaunch. Right?{noformat}
Yes.
{noformat}
The semantic change is a different discussion; that one comes from running the
code directly before instead of adding it to the cmd/sh that launches the
container.{noformat}
Yes. I added a bunch of folks, let's see what is their feedback.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531818#comment-16531818
]
Íñigo Goiri commented on HADOOP-15528:
--
bq. I run this patch in Windows, and I saw a good 35% reduction in latency for
symlink and 7000 IO ops less than before.
To make it clear, this comes from switching from {{winutils symlink}} to
{{FileUtil#symLink}} in {{ContainerLaunch}}. Right?
The semantic change is a different discussion; that one comes from running the
code directly before instead of adding it to the cmd/sh that launches the
container.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531802#comment-16531802
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
This Jira is changing the way a container starts.
In the old semantic, we create the directories and all the symlink *AFTER* the
container starts.
In this new semantic, we create everything *BEFORE* the container starts. We
will have more control over failures due to Symlink and better performance.
I run this patch in Windows, and I saw a good 35% reduction in latency for
symlink and 7000 IO ops less than before.
In future, we can add more retry logic over failures and possible avoiding to
start a container when we are not able to recover from a failure.
cc. [~subru] , [~curino] , [~leftnoteasy] , [~sunilg]
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531784#comment-16531784
]
Íñigo Goiri commented on HADOOP-15528:
--
{quote}
2 possible future improvements that we can do within this Jira or in a future
jira:
* For each resource to create the symlink we try to create the directory. We
should move the creation of the directory in ContainerExecutor.
* If the symlink failed we still start the container and we crash afterward. We
have to avoid to start the container.
{quote}
[^HADOOP-15528-HADOOP-15461.v3.patch] is changing the way containers are
launched.
I'm not sure this is OK.
[~jlowe], [~ste...@apache.org], what are your thoughts on this?
I would be inclined towards keeping the old semantic.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530606#comment-16530606
]
genericqa commented on HADOOP-15528:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15461 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
18s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} HADOOP-15461 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930019/HADOOP-15528-HADOOP-15461.v3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2f3dfd46913e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-15461 / ad353e3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14846/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14846/testReport/ |
| Max. process+thread count | 335 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530551#comment-16530551
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
Attached [^HADOOP-15528-HADOOP-15461.v3.patch].
2 possible future improvements that we can do within this Jira or in a future
jira:
* For each resource to create the symlink we try to create the directory. We
should move the creation of the directory in ContainerExecutor.
* If the symlink failed we still start the container and we crash afterward. We
have to avoid to start the container.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch, HADOOP-15528-HADOOP-15461.v3.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512876#comment-16512876
]
Íñigo Goiri commented on HADOOP-15528:
--
HADOOP-15537 will take care of the cleanup for the launcher.
Once we have that in, we can rebase [^HADOOP-15528-HADOOP-15461.v2.patch].
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511512#comment-16511512
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
Thanks [~elgoiri]. I created HADOOP-15536 to support the creation of the
directories within FileUtils and avoid to rely on mkdir commands.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510433#comment-16510433
]
genericqa commented on HADOOP-15528:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15528 does not apply to HADOOP-15461. Rebase required?
Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927557/HADOOP-15528-HADOOP-15461.v2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14768/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510432#comment-16510432
]
Íñigo Goiri commented on HADOOP-15528:
--
Thanks [~giovanni.fumarola] for [^HADOOP-15528-HADOOP-15461.v2.patch].
Let's focus to the scope of the scope and do the following:
* Keep a writeLaunchEnv with the old parameters and passing null. This will
reduce the number of calls where we have to add null manually.
* Don't remove the supresswarnings, do a separate JIRA for that; we can commit
that to trunk.
* Don't remove link just mark it as deprecate.
* Remove putEnvIfNotNull and putEnvIfNotNull in the new JIRA for trunk.
* Avoid the stetic fixes to parts that are not changed (e.g.,
TestContainerLaunch #200)
* Avoid fixes to close and supresswarning and jarFile.close().
* Don't remove the testWindowsShellScriptBuilderLink.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510407#comment-16510407
]
Giovanni Matteo Fumarola commented on HADOOP-15528:
---
After an offline chat with [~elgoiri] we discussed on how to avoid to break the
compatibility. Proper design notes will follow up.
Attached [^HADOOP-15528-HADOOP-15461.v2.patch].
Let's see what Yetus will say and then we can continue work on the patch.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch,
> HADOOP-15528-HADOOP-15461.v2.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508912#comment-16508912
]
genericqa commented on HADOOP-15528:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15461 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
1s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} HADOOP-15461 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-15461 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 17s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.nodemanager.TestNodeManagerShutdown |
| |
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch |
| | hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
| |
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor |
| | hadoop.yarn.server.nodemanager.TestNodeManagerResync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15528 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927384/HADOOP-15528-HADOOP-15461.v1.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7746332dd627 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-15461 / ae9d83a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HADOOP-15528) Deprecate ContainerLaunch#link by using FileUtil#SymLink
[
https://issues.apache.org/jira/browse/HADOOP-15528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508786#comment-16508786
]
Íñigo Goiri commented on HADOOP-15528:
--
Thanks [~giovanni.fumarola] for the patch.
I'm not sure if {{ContainerLaunch}} can be considered as public or not.
I would mark the link method as deprecated instead of removing it.
For the @SuppressWarnings("unchecked"), I think we should do that in a separate
JIRA; probably in trunk
We need to look for the source of this supresswarning and why is not needed
anymore.
> Deprecate ContainerLaunch#link by using FileUtil#SymLink
>
>
> Key: HADOOP-15528
> URL: https://issues.apache.org/jira/browse/HADOOP-15528
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
>Priority: Major
> Attachments: HADOOP-15528-HADOOP-15461.v1.patch
>
>
> {{ContainerLaunch}} currently uses its own utility to create links (including
> winutils).
> This should be deprecated and rely on {{FileUtil#SymLink}} which is already
> multi-platform and pure Java.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
