[jira] [Commented] (YARN-5701) Fix issues in yarn native services apps-of-apps
[ https://issues.apache.org/jira/browse/YARN-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241029#comment-16241029 ] Hudson commented on YARN-5701: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13193 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13193/]) YARN-5701. Fix issues in yarn native services apps-of-apps. Contributed (jianhe: rev f206a43954b460308c09c51647208f5e41ba2d9d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/core/buildutils/InstanceBuilder.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/providers/docker/DockerProviderService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/providers/ProviderUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/providers/docker/DockerClientProvider.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/java/org/apache/slider/client/SliderClient.java > Fix issues in yarn native services apps-of-apps > --- > > Key: YARN-5701 > URL: https://issues.apache.org/jira/browse/YARN-5701 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-5701-yarn-native-services-001.patch, > YARN-5701-yarn-native-services.001.patch > > > In testing out complex apps-of-apps with the docker provider, I found a few > issues primarily related to exports, as well as a couple of issues in config > generation and CredentialProvider handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5701) Fix issues in yarn native services apps-of-apps
[
https://issues.apache.org/jira/browse/YARN-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576040#comment-15576040
]
Billie Rinaldi commented on YARN-5701:
--
Here are details on the bugs I am attempting to fix in this patch.
The changes in SliderClient are to use System.console() instead of a
BufferedReader to prompt the user for passwords when the app requests passwords
in the appConfig. When using the old slider python script that launched the
SliderClient, System.console did not work, so we used BufferedReader and masked
the passwords in the slider script. When running with the yarn script instead
of slider, we need to use System.console() (as is done in CredentialShell)
because the yarn script is not masking the passwords.
Another issue that came up when testing an assembly app that requests passwords
is that the assembly is not inheriting the credentials request of the child /
external apps. This is fixed in the InstanceBuilder.java changes.
In ProviderUtils.filterSiteOptions, the return map is being used as
substitution tokens, so the keys should be in the format $\{key\}. This is
fixing a problem where tokens are not being substituted properly.
In ProviderUtils.createConfigFile and ProviderUtils.localizeConfigFile, there
is a concurrency problem in the creation of config files that occurs when
multiple containers of the same type are being launched at once. This part of
the patch fixes those.
The new methods in ProviderUtils, getHostNamesList and getIPsList, are used to
populate substitution tokens COMPONENTNAME_HOSTNAME and COMPONENTNAME_IP. These
are needed in addition to COMPONENTNAME_HOST token which provides the name of
the host where a container is running. In the docker container case, the
HOSTNAME substitution will be the docker hostname of the container (we may want
to change this later to be the DNS hostname in the case when RegistryDNS is
enabled). This patch also adds handling for the nested assembly case when
role.prefix is populated.
DockerProviderService line 374 fixes a typo; it should have been using
replaceAll instead of replace.
DockerProviderService.buildMonitorDetails and
DockerProviderService.buildRoleHostDetails are needed to populate information
on the AM UI page. These methods are copied from the original
AgentProviderService.
The .keep file needs to be added to make the Slider AM web service work. It is
hard to add this file. You have to run "git add -f
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core/src/main/resources/webapps/slideram/.keep"
before doing the git commit.
Lastly, clusterName has been added as a parameter to a couple of methods where
it was missing. CLUSTER_NAME was not being substituted properly in exports
because the parameter was not being passed.
> Fix issues in yarn native services apps-of-apps
> ---
>
> Key: YARN-5701
> URL: https://issues.apache.org/jira/browse/YARN-5701
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5701-yarn-native-services-001.patch,
> YARN-5701-yarn-native-services.001.patch
>
>
> In testing out complex apps-of-apps with the docker provider, I found a few
> issues primarily related to exports, as well as a couple of issues in config
> generation and CredentialProvider handling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5701) Fix issues in yarn native services apps-of-apps
[
https://issues.apache.org/jira/browse/YARN-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563018#comment-15563018
]
Hadoop QA commented on YARN-5701:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
56s {color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 55s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
in yarn-native-services has 317 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s
{color} | {color:red} hadoop-yarn-slider-core in yarn-native-services failed.
{color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core:
The patch generated 6 new + 418 unchanged - 4 fixed = 424 total (was 422)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
generated 0 new + 314 unchanged - 3 fixed = 314 total (was 317) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-yarn-slider-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 20s
{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 18s
{color} | {color:red} The patch generated 11 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 39s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832506/YARN-5701-yarn-native-services.001.patch
|
| JIRA Issue | YARN-5701 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ba286cf43b89 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 625b92c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/13338/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core-warnings.html
|
| javadoc |
https://builds.apache.org/job/PreCommi
[jira] [Commented] (YARN-5701) Fix issues in yarn native services apps-of-apps
[
https://issues.apache.org/jira/browse/YARN-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562863#comment-15562863
]
Hadoop QA commented on YARN-5701:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} YARN-5701 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831387/YARN-5701-yarn-native-services-001.patch
|
| JIRA Issue | YARN-5701 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13337/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix issues in yarn native services apps-of-apps
> ---
>
> Key: YARN-5701
> URL: https://issues.apache.org/jira/browse/YARN-5701
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-5701-yarn-native-services-001.patch
>
>
> In testing out complex apps-of-apps with the docker provider, I found a few
> issues primarily related to exports, as well as a couple of issues in config
> generation and CredentialProvider handling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
