[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[
https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618519#comment-16618519
]
Bibin A Chundatt commented on YARN-7599:
Thank you [~botong] for updated patch
{quote}
I should have excluded all apps that are still in YarnRM memory. That should
eliminate the race condition you mentioned. What do you think?
{quote}
Yes , I aggree with the same.
> [GPG] ApplicationCleaner in Global Policy Generator
> ---
>
> Key: YARN-7599
> URL: https://issues.apache.org/jira/browse/YARN-7599
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Labels: federation, gpg
> Attachments: YARN-7599-YARN-7402.v1.patch,
> YARN-7599-YARN-7402.v2.patch, YARN-7599-YARN-7402.v3.patch,
> YARN-7599-YARN-7402.v4.patch
>
>
> In Federation, we need a cleanup service for StateStore as well as Yarn
> Registry. For the former, we need to remove old application records. For the
> latter, failed and killed applications might leave records in the Yarn
> Registry (see YARN-6128). We plan to do both cleanup work in
> ApplicationCleaner in GPG
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8787) Fix broken list items in PlacementConstraints.md.vm
[ https://issues.apache.org/jira/browse/YARN-8787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618517#comment-16618517 ] Weiwei Yang commented on YARN-8787: --- Yes, it should be list items, lets get this fixed in 3.2.0 ... it's close to cut out. [~masatana], do you want to provide a patch? > Fix broken list items in PlacementConstraints.md.vm > --- > > Key: YARN-8787 > URL: https://issues.apache.org/jira/browse/YARN-8787 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.1.1 >Reporter: Masahiro Tanaka >Assignee: Masahiro Tanaka >Priority: Minor > > It looks like some parts of the document below should be list items. > https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html > It might be because of missing newlines before listing. > https://github.com/apache/hadoop/blob/ee051ef9fec1fddb612aa1feae9fd3df7091354f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PlacementConstraints.md.vm#L89-L92 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8788) mvn package -Pyarn-ui fails on JDK9
[
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8788:
Environment: Java 9.0.4, CentOS 7.5
> mvn package -Pyarn-ui fails on JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Java 9.0.4, CentOS 7.5
>Reporter: Akira Ajisaka
>Priority: Major
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
> (default) on project hadoop-yarn-ui: Execution default of goal
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] =
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8788) mvn package -Pyarn-ui fails on JDK9
[
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8788:
Description:
{{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk.
{noformat}
[ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
(default) on project hadoop-yarn-ui: Execution default of goal
ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
java.lang.ExceptionInInitializerError: null
[ERROR] -
[ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] =
file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
[ERROR] urls[1] =
file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
[ERROR] urls[2] =
file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
(snip)
{noformat}
was:
{{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk with
Java 9.0.4.
{noformat}
[ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
(default) on project hadoop-yarn-ui: Execution default of goal
ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
java.lang.ExceptionInInitializerError: null
[ERROR] -
[ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] =
file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
[ERROR] urls[1] =
file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
[ERROR] urls[2] =
file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
(snip)
{noformat}
> mvn package -Pyarn-ui fails on JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Major
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
> (default) on project hadoop-yarn-ui: Execution default of goal
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] =
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8788) mvn package -Pyarn-ui fails in JDK9
[
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618509#comment-16618509
]
Akira Ajisaka commented on YARN-8788:
-
The latest version of wro4j-maven-plugin does not support JDK 9.
https://github.com/wro4j/wro4j/issues/1039
> mvn package -Pyarn-ui fails in JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Major
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk with
> Java 9.0.4.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
> (default) on project hadoop-yarn-ui: Execution default of goal
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] =
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8788) mvn package -Pyarn-ui fails on JDK9
[
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-8788:
Summary: mvn package -Pyarn-ui fails on JDK9 (was: mvn package -Pyarn-ui
fails in JDK9)
> mvn package -Pyarn-ui fails on JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Major
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk with
> Java 9.0.4.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
> (default) on project hadoop-yarn-ui: Execution default of goal
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] =
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-8788) mvn package -Pyarn-ui fails in JDK9
[
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka moved HADOOP-15766 to YARN-8788:
--
Key: YARN-8788 (was: HADOOP-15766)
Project: Hadoop YARN (was: Hadoop Common)
> mvn package -Pyarn-ui fails in JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Major
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk with
> Java 9.0.4.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
> (default) on project hadoop-yarn-ui: Execution default of goal
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] =
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8635) Container Resource localization fails if umask is 077
[
https://issues.apache.org/jira/browse/YARN-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618504#comment-16618504
]
Bilwa S T commented on YARN-8635:
-
[~jlowe] Please review
> Container Resource localization fails if umask is 077
> -
>
> Key: YARN-8635
> URL: https://issues.apache.org/jira/browse/YARN-8635
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Bibin A Chundatt
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-8635-001.patch
>
>
> {code}
> java.io.IOException: Application application_1533652359071_0001
> initialization failed (exitCode=255) with output: main : command provided 0
> main : run as user is mapred
> main : requested yarn user is mapred
> Path
> /opt/HA/OSBR310/nmlocal/usercache/mapred/appcache/application_1533652359071_0001
> has permission 700 but needs permission 750.
> Did not create any app directories
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:411)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1229)
> Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=255:
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:402)
> ... 1 more
> Caused by: ExitCodeException exitCode=255:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
> at org.apache.hadoop.util.Shell.run(Shell.java:902)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
> ... 2 more
> 2018-08-08 17:43:26,918 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_e04_1533652359071_0001_01_27 transitioned from
> LOCALIZING to LOCALIZATION_FAILED
> 2018-08-08 17:43:26,916 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_e04_1533652359071_0001_01_31 startLocalizer is :
> 255
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=255:
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:402)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1229)
> Caused by: ExitCodeException exitCode=255:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
> at org.apache.hadoop.util.Shell.run(Shell.java:902)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
> ... 2 more
> 2018-08-08 17:43:26,923 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Localizer failed for containe
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8787) Fix broken list items in PlacementConstraints.md.vm
[ https://issues.apache.org/jira/browse/YARN-8787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated YARN-8787: Priority: Minor (was: Trivial) > Fix broken list items in PlacementConstraints.md.vm > --- > > Key: YARN-8787 > URL: https://issues.apache.org/jira/browse/YARN-8787 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.1.1 >Reporter: Masahiro Tanaka >Assignee: Masahiro Tanaka >Priority: Minor > > It looks like some parts of the document below should be list items. > https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html > It might be because of missing newlines before listing. > https://github.com/apache/hadoop/blob/ee051ef9fec1fddb612aa1feae9fd3df7091354f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PlacementConstraints.md.vm#L89-L92 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8787) Fix broken list items in PlacementConstraints.md.vm
[ https://issues.apache.org/jira/browse/YARN-8787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masahiro Tanaka updated YARN-8787: -- Description: It looks like some parts of the document below should be list items. https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html It might be because of missing newlines before listing. https://github.com/apache/hadoop/blob/ee051ef9fec1fddb612aa1feae9fd3df7091354f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PlacementConstraints.md.vm#L89-L92 was: It looks like some parts of the document below should be list items. https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html It might be because of missing newlines. https://github.com/apache/hadoop/blob/ee051ef9fec1fddb612aa1feae9fd3df7091354f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PlacementConstraints.md.vm#L89-L92 > Fix broken list items in PlacementConstraints.md.vm > --- > > Key: YARN-8787 > URL: https://issues.apache.org/jira/browse/YARN-8787 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.1.1 >Reporter: Masahiro Tanaka >Assignee: Masahiro Tanaka >Priority: Trivial > > It looks like some parts of the document below should be list items. > https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html > It might be because of missing newlines before listing. > https://github.com/apache/hadoop/blob/ee051ef9fec1fddb612aa1feae9fd3df7091354f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PlacementConstraints.md.vm#L89-L92 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8787) Fix broken list items in PlacementConstraints.md.vm
Masahiro Tanaka created YARN-8787: - Summary: Fix broken list items in PlacementConstraints.md.vm Key: YARN-8787 URL: https://issues.apache.org/jira/browse/YARN-8787 Project: Hadoop YARN Issue Type: Bug Components: documentation Affects Versions: 3.1.1 Reporter: Masahiro Tanaka Assignee: Masahiro Tanaka It looks like some parts of the document below should be list items. https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html It might be because of missing newlines. https://github.com/apache/hadoop/blob/ee051ef9fec1fddb612aa1feae9fd3df7091354f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PlacementConstraints.md.vm#L89-L92 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8665) Yarn Service Upgrade: Support cancelling upgrade
[
https://issues.apache.org/jira/browse/YARN-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618405#comment-16618405
]
Hadoop QA commented on YARN-8665:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 29s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 18 new + 460 unchanged - 4 fixed = 478 total (was 464) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
15s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
50s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m
37s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
6s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
3s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8665 |
| JIRA Patch URL |
[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[
https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618401#comment-16618401
]
Hadoop QA commented on YARN-7599:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
50s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
59s{color} | {color:green} YARN-7402 passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} The patch fails to run checkstyle in hadoop-yarn
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
33s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange} The patch fails to run checkstyle in hadoop-yarn
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
13s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
16s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-7599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940104/YARN-7599-YARN-7402.v4.patch
|
| Optional Tests | dupname asflicense compile javac
[jira] [Created] (YARN-8786) LinuxContainerExecutor fails sporadically in create_local_dirs
Jon Bender created YARN-8786:
Summary: LinuxContainerExecutor fails sporadically in
create_local_dirs
Key: YARN-8786
URL: https://issues.apache.org/jira/browse/YARN-8786
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jon Bender
We started using CGroups with LinuxContainerExecutor recently, running Apache
Hadoop 3.0.0. Occasionally (once out of many millions of tasks) a yarn
container will fail with a message like the following:
{code:java}
[2018-09-02 23:48:02.458691] 18/09/02 23:48:02 INFO container.ContainerImpl:
Container container_1530684675517_516620_01_020846 transitioned from SCHEDULED
to RUNNING
[2018-09-02 23:48:02.458874] 18/09/02 23:48:02 INFO
monitor.ContainersMonitorImpl: Starting resource-monitoring for
container_1530684675517_516620_01_020846
[2018-09-02 23:48:02.506114] 18/09/02 23:48:02 WARN
privileged.PrivilegedOperationExecutor: Shell execution returned exit code: 35.
Privileged Execution Operation Stderr:
[2018-09-02 23:48:02.506159] Could not create container dirsCould not create
local files and directories
[2018-09-02 23:48:02.506220]
[2018-09-02 23:48:02.506238] Stdout: main : command provided 1
[2018-09-02 23:48:02.506258] main : run as user is nobody
[2018-09-02 23:48:02.506282] main : requested yarn user is root
[2018-09-02 23:48:02.506294] Getting exit code file...
[2018-09-02 23:48:02.506307] Creating script paths...
[2018-09-02 23:48:02.506330] Writing pid file...
[2018-09-02 23:48:02.506366] Writing to tmp file
/path/to/hadoop/yarn/local/nmPrivate/application_1530684675517_516620/container_1530684675517_516620_01_020846/container_1530684675517_516620_01_020846.pid.tmp
[2018-09-02 23:48:02.506389] Writing to cgroup task files...
[2018-09-02 23:48:02.506402] Creating local dirs...
[2018-09-02 23:48:02.506414] Getting exit code file...
[2018-09-02 23:48:02.506435] Creating script paths...
{code}
Looking at the container executor source it's traceable to errors here:
[https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L1604]
And ultimately to
[https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L672]
The root failure seems to be in the underlying mkdir call, but that exit code /
errno is swallowed so we don't have more details. We tend to see this when many
containers start at the same time for the same application on a host, and
suspect it may be related to some race conditions around those shared
directories between containers for the same application.
For example, this is a typical pattern in the audit logs:
{code:java}
[2018-09-07 17:16:38.447654] 18/09/07 17:16:38 INFO nodemanager.NMAuditLogger:
USER=rootIP=<> Container Request TARGET=ContainerManageImpl
RESULT=SUCCESS APPID=application_1530684675517_559126
CONTAINERID=container_1530684675517_559126_01_012871
[2018-09-07 17:16:38.492298] 18/09/07 17:16:38 INFO nodemanager.NMAuditLogger:
USER=rootIP=<> Container Request TARGET=ContainerManageImpl
RESULT=SUCCESS APPID=application_1530684675517_559126
CONTAINERID=container_1530684675517_559126_01_012870
[2018-09-07 17:16:38.614044] 18/09/07 17:16:38 WARN nodemanager.NMAuditLogger:
USER=rootOPERATION=Container Finished - Failed TARGET=ContainerImpl
RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE
APPID=application_1530684675517_559126
CONTAINERID=container_1530684675517_559126_01_012871
{code}
Two containers for the same application starting in quick succession followed
by the EXITED_WITH_FAILURE step (exit code 35).
We plan to upgrade to 3.1.x soon but I don't expect this to be fixed by this,
the only major JIRAs that affected the executor since 3.0.0 seem unrelated
([https://github.com/apache/hadoop/commit/bc285da107bb84a3c60c5224369d7398a41db2d8]
and
[https://github.com/apache/hadoop/commit/a82be7754d74f4d16b206427b91e700bb5f44d56])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[ https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-7599: --- Attachment: YARN-7599-YARN-7402.v4.patch > [GPG] ApplicationCleaner in Global Policy Generator > --- > > Key: YARN-7599 > URL: https://issues.apache.org/jira/browse/YARN-7599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Labels: federation, gpg > Attachments: YARN-7599-YARN-7402.v1.patch, > YARN-7599-YARN-7402.v2.patch, YARN-7599-YARN-7402.v3.patch, > YARN-7599-YARN-7402.v4.patch > > > In Federation, we need a cleanup service for StateStore as well as Yarn > Registry. For the former, we need to remove old application records. For the > latter, failed and killed applications might leave records in the Yarn > Registry (see YARN-6128). We plan to do both cleanup work in > ApplicationCleaner in GPG -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8665) Yarn Service Upgrade: Support cancelling upgrade
[
https://issues.apache.org/jira/browse/YARN-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618286#comment-16618286
]
Chandni Singh edited comment on YARN-8665 at 9/17/18 11:52 PM:
---
Patch 1 includes support for cancel upgrade.
To test the patch :
1. yarn app -launch test1 sleeper
2. yarn app -upgrade test1 -initiate sleeper_v2.json
3. yarn app -upgrade test1 -instances sleeper-0
4. yarn app -upgrade test1 -cancel
[~eyang] [~gsaha] Could you please review?
Please note that with my changes, the ServiceAM will only send a second reinit
request for the same container until it has received a response from the NM for
previous reinit request.
I see an intermittent issue with NM when there are subsequent reinitialization
requests made for the same container. NM sends premature response back to
ServiceAM that container is running but the state of the container in NM is
still {{REINITIALIZING_AWAITING_KILL}}. This is hard to reproduce manually.
Since this is a NM side bug, I am going to open a new ticket to fix it.
was (Author: csingh):
Patch 1 includes support for cancel upgrade.
To test the patch :
1. yarn app -launch test1 sleeper
2. yarn app -upgrade test1 -initiate sleeper_v2.json
3. yarn app -upgrade test1 -instances sleeper-0
4. yarn app -upgrade test1 -cancel
[~eyang] [~gsaha] Could you please review?
Please note that with my changes, the ServiceAM will only send a second reinit
request for the same container until it has received a response from the NM for
previous reinit request.
I see an intermittent issue with NM when there are subsequent reinitialization
requests made for same container. NM sends premature response back to
ServiceAM that container is running but the state of the container in NM is
still {{REINITIALIZING_AWAITING_KILL}}. This is hard to reproduce manually.
Since this is a NM side bug, I am going to open a new ticket to fix it.
> Yarn Service Upgrade: Support cancelling upgrade
> -
>
> Key: YARN-8665
> URL: https://issues.apache.org/jira/browse/YARN-8665
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-8665.001.patch
>
>
> When a service is upgraded without auto-finalization or express upgrade, then
> the upgrade can be cancelled. This provides the user ability to test upgrade
> of a single instance and if that doesn't go well, they get a chance to cancel
> it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[
https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618282#comment-16618282
]
Hadoop QA commented on YARN-7599:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
15s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
29s{color} | {color:green} YARN-7402 passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} The patch fails to run checkstyle in hadoop-yarn
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange} The patch fails to run checkstyle in hadoop-yarn
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
11s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 45s{color}
| {color:red} hadoop-yarn-server-globalpolicygenerator in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.globalpolicygenerator.policygenerator.TestPolicyGenerator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Updated] (YARN-8665) Yarn Service Upgrade: Support cancelling upgrade
[ https://issues.apache.org/jira/browse/YARN-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated YARN-8665: Attachment: YARN-8665.001.patch > Yarn Service Upgrade: Support cancelling upgrade > - > > Key: YARN-8665 > URL: https://issues.apache.org/jira/browse/YARN-8665 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Attachments: YARN-8665.001.patch > > > When a service is upgraded without auto-finalization or express upgrade, then > the upgrade can be cancelled. This provides the user ability to test upgrade > of a single instance and if that doesn't go well, they get a chance to cancel > it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[ https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618241#comment-16618241 ] Botong Huang commented on YARN-7599: Following the [discussion|https://issues.apache.org/jira/browse/YARN-8637?focusedCommentId=16615094=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16615094] in YARN-8637, v3 patch loaded, deciding whether to remove the application solely depends on whether Router/YarnRM still knows about it. > [GPG] ApplicationCleaner in Global Policy Generator > --- > > Key: YARN-7599 > URL: https://issues.apache.org/jira/browse/YARN-7599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Labels: federation, gpg > Attachments: YARN-7599-YARN-7402.v1.patch, > YARN-7599-YARN-7402.v2.patch, YARN-7599-YARN-7402.v3.patch > > > In Federation, we need a cleanup service for StateStore as well as Yarn > Registry. For the former, we need to remove old application records. For the > latter, failed and killed applications might leave records in the Yarn > Registry (see YARN-6128). We plan to do both cleanup work in > ApplicationCleaner in GPG -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8734) Readiness check for remote service
[
https://issues.apache.org/jira/browse/YARN-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618229#comment-16618229
]
Hadoop QA commented on YARN-8734:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
17s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
5s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8734 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940066/YARN-8734.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc
[jira] [Updated] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator
[ https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-7599: --- Attachment: YARN-7599-YARN-7402.v3.patch > [GPG] ApplicationCleaner in Global Policy Generator > --- > > Key: YARN-7599 > URL: https://issues.apache.org/jira/browse/YARN-7599 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Minor > Labels: federation, gpg > Attachments: YARN-7599-YARN-7402.v1.patch, > YARN-7599-YARN-7402.v2.patch, YARN-7599-YARN-7402.v3.patch > > > In Federation, we need a cleanup service for StateStore as well as Yarn > Registry. For the former, we need to remove old application records. For the > latter, failed and killed applications might leave records in the Yarn > Registry (see YARN-6128). We plan to do both cleanup work in > ApplicationCleaner in GPG -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8623) Update Docker examples to use image which exists
[ https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618172#comment-16618172 ] Shane Kumpf commented on YARN-8623: --- The direction I was proposing for the default MR pi example was to use the openjdk:8 image and bind mount /etc/passwd and /etc/group. This should work in a good number of "try it out" use cases without needing to modify the existing hadoop-runner image. We could point to the SSSD instructions we have as an alternative way to manage users. Thoughts? > Update Docker examples to use image which exists > > > Key: YARN-8623 > URL: https://issues.apache.org/jira/browse/YARN-8623 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Craig Condit >Priority: Minor > Labels: Docker > > The example Docker image given in the documentation > (images/hadoop-docker:latest) does not exist. We could change > images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist. > We'd need to do a quick sanity test to see if the image works with YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8623) Update Docker examples to use image which exists
[ https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618157#comment-16618157 ] Eric Yang commented on YARN-8623: - The discussion has not converged toward using apache/hadoop-runner image for documentation example due to limitation of the current images that might not work in most YARN environment. For us to converge to use apache/hadoop-runner image in documentation, the images must be enhanced to do the following: # Hadoop runner image intelligently supports user management using sssd socket when running with normal user uid/gid. # Java and Hadoop binaries are part of hadoop-runner image to ensure the version control of the hadoop docker images are self contained for binaries. (level set expectation, no actual code changes). Do we agree on the docker image enhancement to converge for documentation updates? If agreed upon, I will open JIRA to enable sssd support in hadoop-runner image. If not agreed upon, we can close this as no plan to fix. > Update Docker examples to use image which exists > > > Key: YARN-8623 > URL: https://issues.apache.org/jira/browse/YARN-8623 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Craig Condit >Priority: Minor > Labels: Docker > > The example Docker image given in the documentation > (images/hadoop-docker:latest) does not exist. We could change > images/hadoop-docker:latest to apache/hadoop-runner:latest, which does exist. > We'd need to do a quick sanity test to see if the image works with YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8734) Readiness check for remote service
[ https://issues.apache.org/jira/browse/YARN-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8734: Attachment: YARN-8734.005.patch > Readiness check for remote service > -- > > Key: YARN-8734 > URL: https://issues.apache.org/jira/browse/YARN-8734 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: Dependency check vs.pdf, YARN-8734.001.patch, > YARN-8734.002.patch, YARN-8734.003.patch, YARN-8734.004.patch, > YARN-8734.005.patch > > > When a service is deploying, there can be remote service dependency. It > would be nice to describe ZooKeeper as a dependent service, and the service > has reached a stable state, then deploy HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618115#comment-16618115
]
Eric Yang commented on YARN-8783:
-
[~simonprewo] {quote}
It would nice to describe it here:
https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/DockerContainers.html{quote}
Agree, we can probably add a paragraph in Docker Image Requirements section to
clarify how to use docker.trusted.registries=libraries for top level images.
Patch is welcome.
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: Docker, container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8785) Error Message "Invalid docker rw mount" not helpful
[
https://issues.apache.org/jira/browse/YARN-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618081#comment-16618081
]
Eric Yang commented on YARN-8785:
-
The message change looks good to me.
> Error Message "Invalid docker rw mount" not helpful
> ---
>
> Key: YARN-8785
> URL: https://issues.apache.org/jira/browse/YARN-8785
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.9.1, 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: Docker
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> A user recieves the error message _Invalid docker rw mount_ when a container
> tries to mount a directory which is not configured in property
> *docker.allowed.rw-mounts*.
> {code:java}
> Invalid docker rw mount
> '/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01:/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01',
>
> realpath=/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01{code}
> The error message makes the user think "It is not possible due to a docker
> issue". My suggestion would be to put there a message like *Configuration of
> the container executor does not allow mounting directory.*.
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
> CURRENT:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Invalid docker mount '%s', realpath=%s\n",
> values[i], mount_src);
> ...
> {code}
> NEW:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Configuration of the container executor does not
> allow mounting directory '%s', realpath=%s\n", values[i], mount_src);
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8734) Readiness check for remote service
[
https://issues.apache.org/jira/browse/YARN-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618058#comment-16618058
]
Hadoop QA commented on YARN-8734:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 24s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 51 unchanged - 0 fixed = 52 total (was 51) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
27s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
52s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8734 |
| JIRA Patch URL |
[jira] [Updated] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf updated YARN-8783:
--
Labels: Docker container-executor docker (was: container-executor docker)
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: Docker, container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf updated YARN-8783:
--
Issue Type: Sub-task (was: Bug)
Parent: YARN-8472
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: Docker, container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8785) Error Message "Invalid docker rw mount" not helpful
[
https://issues.apache.org/jira/browse/YARN-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf updated YARN-8785:
--
Issue Type: Sub-task (was: Improvement)
Parent: YARN-8472
> Error Message "Invalid docker rw mount" not helpful
> ---
>
> Key: YARN-8785
> URL: https://issues.apache.org/jira/browse/YARN-8785
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.9.1, 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: Docker
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> A user recieves the error message _Invalid docker rw mount_ when a container
> tries to mount a directory which is not configured in property
> *docker.allowed.rw-mounts*.
> {code:java}
> Invalid docker rw mount
> '/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01:/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01',
>
> realpath=/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01{code}
> The error message makes the user think "It is not possible due to a docker
> issue". My suggestion would be to put there a message like *Configuration of
> the container executor does not allow mounting directory.*.
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
> CURRENT:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Invalid docker mount '%s', realpath=%s\n",
> values[i], mount_src);
> ...
> {code}
> NEW:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Configuration of the container executor does not
> allow mounting directory '%s', realpath=%s\n", values[i], mount_src);
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8785) Error Message "Invalid docker rw mount" not helpful
[
https://issues.apache.org/jira/browse/YARN-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf updated YARN-8785:
--
Labels: Docker (was: )
> Error Message "Invalid docker rw mount" not helpful
> ---
>
> Key: YARN-8785
> URL: https://issues.apache.org/jira/browse/YARN-8785
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.9.1, 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: Docker
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> A user recieves the error message _Invalid docker rw mount_ when a container
> tries to mount a directory which is not configured in property
> *docker.allowed.rw-mounts*.
> {code:java}
> Invalid docker rw mount
> '/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01:/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01',
>
> realpath=/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01{code}
> The error message makes the user think "It is not possible due to a docker
> issue". My suggestion would be to put there a message like *Configuration of
> the container executor does not allow mounting directory.*.
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
> CURRENT:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Invalid docker mount '%s', realpath=%s\n",
> values[i], mount_src);
> ...
> {code}
> NEW:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Configuration of the container executor does not
> allow mounting directory '%s', realpath=%s\n", values[i], mount_src);
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-8785) Error Message "Invalid docker rw mount" not helpful
[
https://issues.apache.org/jira/browse/YARN-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf moved HADOOP-15734 to YARN-8785:
Affects Version/s: (was: 3.1.1)
(was: 2.9.1)
2.9.1
3.1.1
Key: YARN-8785 (was: HADOOP-15734)
Project: Hadoop YARN (was: Hadoop Common)
> Error Message "Invalid docker rw mount" not helpful
> ---
>
> Key: YARN-8785
> URL: https://issues.apache.org/jira/browse/YARN-8785
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.1.1, 2.9.1
>Reporter: Simon Prewo
>Priority: Major
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> A user recieves the error message _Invalid docker rw mount_ when a container
> tries to mount a directory which is not configured in property
> *docker.allowed.rw-mounts*.
> {code:java}
> Invalid docker rw mount
> '/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01:/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01',
>
> realpath=/usr/local/hadoop/logs/userlogs/application_1536476159258_0004/container_1536476159258_0004_02_01{code}
> The error message makes the user think "It is not possible due to a docker
> issue". My suggestion would be to put there a message like *Configuration of
> the container executor does not allow mounting directory.*.
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
> CURRENT:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Invalid docker mount '%s', realpath=%s\n",
> values[i], mount_src);
> ...
> {code}
> NEW:
> {code:java}
> permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts,
> mount_src);
> permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts,
> mount_src);
> if (permitted_ro == -1 || permitted_rw == -1) {
> fprintf(ERRORFILE, "Configuration of the container executor does not
> allow mounting directory '%s', realpath=%s\n", values[i], mount_src);
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617950#comment-16617950 ] Eric Yang commented on YARN-8777: - [~ebadger] The patch is written for specific use case while making remote attack difficult base on the new extension. I thought about code reuse before, and it is possible to use launch_command in cmd file and having flags passed from node manager. However, it is a smaller surface to be exposed to automated remote attack when bash is hard coded without parameter passing instead of leaving it up to node manager with parameter passing. I don't have strong preference for keeping "-it bash" or making them optional for code reuse, and patch 001 shows my preference toward smaller attack surface. It is good to talk about this before we proceed. > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8784) DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk
[ https://issues.apache.org/jira/browse/YARN-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger reassigned YARN-8784: - Assignee: Eric Badger > DockerLinuxContainerRuntime prevents access to distributed cache entries on a > full disk > --- > > Key: YARN-8784 > URL: https://issues.apache.org/jira/browse/YARN-8784 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.2.0, 3.1.1 >Reporter: Jason Lowe >Assignee: Eric Badger >Priority: Major > Labels: Docker > > DockerLinuxContainerRuntime bind mounts the filecache and usercache > directories into the container to allow tasks to access entries in the > distributed cache. However it only bind mounts directories on disks that > are considered good, and disks that are full or bad are not in that list. If > a container tries to run with a distributed cache entry that has been > previously localized to a disk that is now considered full/bad, the dist > cache directory will _not_ be bind-mounted into the container's filesystem > namespace. At that point any symlinks in the container's current working > directory that point to those disks will reference invalid paths. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8784) DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk
[ https://issues.apache.org/jira/browse/YARN-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-8784: -- Labels: Docker (was: ) > DockerLinuxContainerRuntime prevents access to distributed cache entries on a > full disk > --- > > Key: YARN-8784 > URL: https://issues.apache.org/jira/browse/YARN-8784 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.2.0, 3.1.1 >Reporter: Jason Lowe >Priority: Major > Labels: Docker > > DockerLinuxContainerRuntime bind mounts the filecache and usercache > directories into the container to allow tasks to access entries in the > distributed cache. However it only bind mounts directories on disks that > are considered good, and disks that are full or bad are not in that list. If > a container tries to run with a distributed cache entry that has been > previously localized to a disk that is now considered full/bad, the dist > cache directory will _not_ be bind-mounted into the container's filesystem > namespace. At that point any symlinks in the container's current working > directory that point to those disks will reference invalid paths. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8784) DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk
Jason Lowe created YARN-8784: Summary: DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk Key: YARN-8784 URL: https://issues.apache.org/jira/browse/YARN-8784 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.1.1, 3.2.0 Reporter: Jason Lowe DockerLinuxContainerRuntime bind mounts the filecache and usercache directories into the container to allow tasks to access entries in the distributed cache. However it only bind mounts directories on disks that are considered good, and disks that are full or bad are not in that list. If a container tries to run with a distributed cache entry that has been previously localized to a disk that is now considered full/bad, the dist cache directory will _not_ be bind-mounted into the container's filesystem namespace. At that point any symlinks in the container's current working directory that point to those disks will reference invalid paths. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617933#comment-16617933
]
Shane Kumpf commented on YARN-8783:
---
[~simonprewo] - one trick here is that there is an implicit namespace at Docker
hub for these "official" images, it is "library". Changing your registry list
to include "library" and changing your image to "library/centos" should allow
this to work without the need to tag locally.
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617922#comment-16617922
]
Simon Prewo commented on YARN-8783:
---
[~eyang] Thanks a lot for your fast response.
It would nice to describe it here:
https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8734) Readiness check for remote service
[ https://issues.apache.org/jira/browse/YARN-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617911#comment-16617911 ] Eric Yang commented on YARN-8734: - Patch 004 skips initialization of ServiceClient, if there is no remote service dependencies defined. > Readiness check for remote service > -- > > Key: YARN-8734 > URL: https://issues.apache.org/jira/browse/YARN-8734 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: Dependency check vs.pdf, YARN-8734.001.patch, > YARN-8734.002.patch, YARN-8734.003.patch, YARN-8734.004.patch > > > When a service is deploying, there can be remote service dependency. It > would be nice to describe ZooKeeper as a dependent service, and the service > has reached a stable state, then deploy HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8734) Readiness check for remote service
[ https://issues.apache.org/jira/browse/YARN-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8734: Attachment: YARN-8734.004.patch > Readiness check for remote service > -- > > Key: YARN-8734 > URL: https://issues.apache.org/jira/browse/YARN-8734 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: Dependency check vs.pdf, YARN-8734.001.patch, > YARN-8734.002.patch, YARN-8734.003.patch, YARN-8734.004.patch > > > When a service is deploying, there can be remote service dependency. It > would be nice to describe ZooKeeper as a dependent service, and the service > has reached a stable state, then deploy HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8500) Use hbase shaded jars
[ https://issues.apache.org/jira/browse/YARN-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617896#comment-16617896 ] Vrushali C commented on YARN-8500: -- As per my discussion with Sean on HBASE-15666, here are the next steps. We have a separate module for the tests that use the mini-cluster, called as 'hadoop-yarn-server-timelineservice-hbase-tests'. This module only has test code and is not required for cluster deployment. So, in the pom for that module, I will see if we can use the non-shaded client (& dependencies) in test-scope and exclude the shaded jars inherited from other dependent modules. > Use hbase shaded jars > - > > Key: YARN-8500 > URL: https://issues.apache.org/jira/browse/YARN-8500 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8500.0001.patch > > > Move to using hbase shaded jars in atsv2 > Related jira YARN-7213 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617795#comment-16617795
]
Eric Yang commented on YARN-8783:
-
List of registries works fine in my test cluster using Hadoop 3.1.1. However,
centos, and ubuntu images are top level images that does not associate with any
repository. This prevents official images like centos:latest and ubuntu:latest
to be specified as trusted registries. You will need to tag the official
images like centos:latest to my.trusted.registry.com/centos:latest to utilize
the official images from docker hub. Otherwise, container-executor will only
trust images in centos/* and ubuntu/*, which might not be what you wanted.
Container executor forces the detection of [repo]/[image]:[version] format.
For images without [repo]/ prefix, they are not usable by default unless tagged
locally. Sorry for the inconvenience, the current code is written to enforce
format validation instead of a prefix match for safety reasons. In YARN-6456,
there are discussion on how to allow exact image name to run on the cluster.
That might improve the usability without tagging the images locally.
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[
https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617780#comment-16617780
]
Eric Badger commented on YARN-8777:
---
Do we want to always force the exec to be {{-it bash}}? I know the intent is
for interactive shell, just thinking of possible other usages, like a fire and
forget command to all containers across a job where we don't want to keep a
shell open.
Other than that question, patch looks good to me.
> Container Executor C binary change to execute interactive docker command
>
>
> Key: YARN-8777
> URL: https://issues.apache.org/jira/browse/YARN-8777
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zian Chen
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: YARN-8777.001.patch
>
>
> Since Container Executor provides Container execution using the native
> container-executor binary, we also need to make changes to accept new
> “dockerExec” method to invoke the corresponding native function to execute
> docker exec command to the running container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Prewo updated YARN-8783:
--
Labels: container-executor docker (was: )
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> docker.trusted.registries=centos
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> docker.trusted.registries=centos,ubuntu
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Prewo updated YARN-8783:
--
Description:
I am deploying the default yarn distributed shell example:
{code:java}
yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
Having a *single trusted registry configured like this works*:
{code:java}
docker.trusted.registries=centos
{code}
But having *a list of trusted registries configured fails* ("Shell error
output: image: centos is not trusted."):
{code:java}
docker.trusted.registries=centos,ubuntu{code}
was:
I am deploying the default yarn distributed shell example:
{code:java}
yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
Having a *single trusted registry configured like this works*:
docker.trusted.registries=centos
But having *a list of trusted registries configured fails* ("Shell error
output: image: centos is not trusted."):
docker.trusted.registries=centos,ubuntu
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
>
> {code:java}
> docker.trusted.registries=centos
> {code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8783) Property docker.trusted.registries does not work when using a list
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Prewo updated YARN-8783:
--
Description:
I am deploying the default yarn distributed shell example:
{code:java}
yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
Having a *single trusted registry configured like this works*:
{code:java}
docker.trusted.registries=centos{code}
But having *a list of trusted registries configured fails* ("Shell error
output: image: centos is not trusted."):
{code:java}
docker.trusted.registries=centos,ubuntu{code}
was:
I am deploying the default yarn distributed shell example:
{code:java}
yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
Having a *single trusted registry configured like this works*:
{code:java}
docker.trusted.registries=centos
{code}
But having *a list of trusted registries configured fails* ("Shell error
output: image: centos is not trusted."):
{code:java}
docker.trusted.registries=centos,ubuntu{code}
> Property docker.trusted.registries does not work when using a list
> --
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Priority: Major
> Labels: container-executor, docker
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8783) Property docker.trusted.registries does not work when using a list
Simon Prewo created YARN-8783:
-
Summary: Property docker.trusted.registries does not work when
using a list
Key: YARN-8783
URL: https://issues.apache.org/jira/browse/YARN-8783
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Simon Prewo
I am deploying the default yarn distributed shell example:
{code:java}
yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
Having a *single trusted registry configured like this works*:
docker.trusted.registries=centos
But having *a list of trusted registries configured fails* ("Shell error
output: image: centos is not trusted."):
docker.trusted.registries=centos,ubuntu
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8725) Submarine job staging directory has a lot of useless PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple times
[
https://issues.apache.org/jira/browse/YARN-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617711#comment-16617711
]
Hadoop QA commented on YARN-8725:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
in trunk has 4 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine:
The patch generated 40 new + 39 unchanged - 1 fixed = 79 total (was 40) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8725 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939991/YARN-8725-trunk.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 38d9bd8990de 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9a265fa |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/21852/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine-warnings.html
|
| checkstyle |
[jira] [Commented] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617680#comment-16617680
]
Szilard Nemeth commented on YARN-3525:
--
Hi [~zsiegl]!
Thanks for the updated patch!
LGTM, +1 (non-binding)
> Rename fair scheduler properties increment-allocation-mb and
> increment-allocation-vcores
>
>
> Key: YARN-3525
> URL: https://issues.apache.org/jira/browse/YARN-3525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Zoltan Siegl
>Priority: Minor
> Attachments: YARN-3525.001.patch, YARN-3525.003.patch
>
>
> Rename below two properties since only used by fair scheduler
> {color:blue}yarn.scheduler.increment-allocation-mb{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-mb{color}
> {color:blue}yarn.scheduler.increment-allocation-vcores{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
> All other properties only for fair scheduler are using {color:red}
> yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617675#comment-16617675
]
Zoltan Siegl commented on YARN-3525:
Javac bug increase is intentional.
> Rename fair scheduler properties increment-allocation-mb and
> increment-allocation-vcores
>
>
> Key: YARN-3525
> URL: https://issues.apache.org/jira/browse/YARN-3525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Zoltan Siegl
>Priority: Minor
> Attachments: YARN-3525.001.patch, YARN-3525.003.patch
>
>
> Rename below two properties since only used by fair scheduler
> {color:blue}yarn.scheduler.increment-allocation-mb{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-mb{color}
> {color:blue}yarn.scheduler.increment-allocation-vcores{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
> All other properties only for fair scheduler are using {color:red}
> yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617637#comment-16617637 ] Hudson commented on YARN-8782: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14976 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14976/]) YARN-8782. Fix exception message in (wwei: rev 9a265fa673ef1b8774cfd69c76cdd29bf344e79d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Minor > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617616#comment-16617616
]
Hadoop QA commented on YARN-3525:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 19s{color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 2 new + 107 unchanged -
0 fixed = 109 total (was 107) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 10 unchanged - 2 fixed = 10 total (was 12) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 80m
27s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-3525 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939966/YARN-3525.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc
[jira] [Comment Edited] (YARN-8725) Submarine job staging directory has a lot of useless PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple times
[
https://issues.apache.org/jira/browse/YARN-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617600#comment-16617600
]
Zhankun Tang edited comment on YARN-8725 at 9/17/18 2:36 PM:
-
Added a patch which does following:
# add a new option "--keep_staging_dir". It's false by default so that we'll
clean up the staging directory after job finish
# add unit test case through "MockRemoteDirectoryManager".
# Changes(staging dir creation) to existing unit test due to the need for a
real directory in local fs for "cleanupStagingDir" to work
Please help review. [~wangda] [~sunilg] [~yuan_zac]
was (Author: tangzhankun):
Added a patch which does following:
# add a new option "--keep_staging_dir". It's false by default so that we'll
clean up the staging directory after job finish
# added unit test case through "MockRemoteDirectoryManager".
# Changes(staging dir creation) to existing unit test due to the need for a
real directory in local fs for "cleanupStagingDir" to work
Please help review. [~wangda] [~sunilg] [~yuan_zac]
> Submarine job staging directory has a lot of useless
> PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple
> times
> --
>
> Key: YARN-8725
> URL: https://issues.apache.org/jira/browse/YARN-8725
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zac Zhou
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8725-trunk.001.patch
>
>
> Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
> PRIMARY_WORKER-launch-script.sh to staging dir.
> The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job
> is submitted multiple times.
> But PRIMARY_WORKER-launch-script.sh would not be overwritten, as it has
> random numbers in its name.
> The files in the staging dir are as follows：
> {code:java}
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:11
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:02
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:06
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
> -rw-r- 2 hadoop hdfs 15225 2018-08-17 18:46
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-16 20:48
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 14:53
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:16
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
> -rw-r- 2 hadoop hdfs 8815 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
> -rw-r- 2 hadoop hdfs 11583 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
> -rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
> {code}
>
> We should stop the staging dir from growing or have a way to clean it up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617605#comment-16617605 ] Weiwei Yang commented on YARN-8782: --- Just committed the patch to trunk/branch-3.1/branch-3.0, thanks for the contribution [~shuzirra]. > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Minor > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated YARN-8782: -- Issue Type: Bug (was: Improvement) > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Minor > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8725) Submarine job staging directory has a lot of useless PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple times
[
https://issues.apache.org/jira/browse/YARN-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617600#comment-16617600
]
Zhankun Tang commented on YARN-8725:
Added a patch which does following:
# add a new option "--keep_staging_dir". It's false by default so that we'll
clean up the staging directory after job finish
# added unit test case through "MockRemoteDirectoryManager".
# Changes(staging dir creation) to existing unit test due to the need for a
real directory in local fs for "cleanupStagingDir" to work
Please help review. [~wangda] [~sunilg] [~yuan_zac]
> Submarine job staging directory has a lot of useless
> PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple
> times
> --
>
> Key: YARN-8725
> URL: https://issues.apache.org/jira/browse/YARN-8725
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zac Zhou
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8725-trunk.001.patch
>
>
> Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
> PRIMARY_WORKER-launch-script.sh to staging dir.
> The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job
> is submitted multiple times.
> But PRIMARY_WORKER-launch-script.sh would not be overwritten, as it has
> random numbers in its name.
> The files in the staging dir are as follows：
> {code:java}
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:11
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:02
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:06
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
> -rw-r- 2 hadoop hdfs 15225 2018-08-17 18:46
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-16 20:48
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 14:53
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:16
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
> -rw-r- 2 hadoop hdfs 8815 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
> -rw-r- 2 hadoop hdfs 11583 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
> -rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
> {code}
>
> We should stop the staging dir from growing or have a way to clean it up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8725) Submarine job staging directory has a lot of useless PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple times
[
https://issues.apache.org/jira/browse/YARN-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8725:
---
Attachment: YARN-8725-trunk.001.patch
> Submarine job staging directory has a lot of useless
> PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple
> times
> --
>
> Key: YARN-8725
> URL: https://issues.apache.org/jira/browse/YARN-8725
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zac Zhou
>Assignee: Zhankun Tang
>Priority: Major
> Attachments: YARN-8725-trunk.001.patch
>
>
> Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
> PRIMARY_WORKER-launch-script.sh to staging dir.
> The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job
> is submitted multiple times.
> But PRIMARY_WORKER-launch-script.sh would not be overwritten, as it has
> random numbers in its name.
> The files in the staging dir are as follows：
> {code:java}
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:11
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:02
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:06
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
> -rw-r- 2 hadoop hdfs 15225 2018-08-17 18:46
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-16 20:48
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 14:53
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
> -rw-r- 2 hadoop hdfs 580 2018-08-17 10:16
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
> -rw-r- 2 hadoop hdfs 8815 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
> -rw-r- 2 hadoop hdfs 11583 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
> -rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
> {code}
>
> We should stop the staging dir from growing or have a way to clean it up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated YARN-8782: -- Priority: Minor (was: Major) > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Minor > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617582#comment-16617582 ] Weiwei Yang commented on YARN-8782: --- +1, committing shortly. > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617523#comment-16617523 ] Gergely Pollak commented on YARN-8782: -- This is just a typo fix, no actual functionality change is involved. > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[
https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617506#comment-16617506
]
Hadoop QA commented on YARN-8782:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8782 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939970/YARN-8782.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 267d3d66eb61 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33d8327 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21851/testReport/ |
| Max. process+thread count | 406 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/21851/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Fix exception message
[jira] [Updated] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-8782: - Attachment: YARN-8782.001.patch > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-8782.001.patch, YARN-8782.001.patch > > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Siegl updated YARN-3525:
---
Attachment: YARN-3525.003.patch
> Rename fair scheduler properties increment-allocation-mb and
> increment-allocation-vcores
>
>
> Key: YARN-3525
> URL: https://issues.apache.org/jira/browse/YARN-3525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Zoltan Siegl
>Priority: Minor
> Attachments: YARN-3525.001.patch, YARN-3525.003.patch
>
>
> Rename below two properties since only used by fair scheduler
> {color:blue}yarn.scheduler.increment-allocation-mb{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-mb{color}
> {color:blue}yarn.scheduler.increment-allocation-vcores{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
> All other properties only for fair scheduler are using {color:red}
> yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Siegl updated YARN-3525:
---
Attachment: (was: YARN-3525.002.patch)
> Rename fair scheduler properties increment-allocation-mb and
> increment-allocation-vcores
>
>
> Key: YARN-3525
> URL: https://issues.apache.org/jira/browse/YARN-3525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Zoltan Siegl
>Priority: Minor
> Attachments: YARN-3525.001.patch
>
>
> Rename below two properties since only used by fair scheduler
> {color:blue}yarn.scheduler.increment-allocation-mb{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-mb{color}
> {color:blue}yarn.scheduler.increment-allocation-vcores{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
> All other properties only for fair scheduler are using {color:red}
> yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Siegl updated YARN-3525:
---
Attachment: YARN-3525.002.patch
> Rename fair scheduler properties increment-allocation-mb and
> increment-allocation-vcores
>
>
> Key: YARN-3525
> URL: https://issues.apache.org/jira/browse/YARN-3525
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Zoltan Siegl
>Priority: Minor
> Attachments: YARN-3525.001.patch
>
>
> Rename below two properties since only used by fair scheduler
> {color:blue}yarn.scheduler.increment-allocation-mb{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-mb{color}
> {color:blue}yarn.scheduler.increment-allocation-vcores{color} to
> {color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
> All other properties only for fair scheduler are using {color:red}
> yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617402#comment-16617402
]
Szilard Nemeth commented on YARN-8750:
--
Test failures seem unrelated.
> Refactor TestQueueMetrics
> -
>
> Key: YARN-8750
> URL: https://issues.apache.org/jira/browse/YARN-8750
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-8750.001.patch, YARN-8750.002.patch,
> YARN-8750.003.patch
>
>
> {{TestQueueMetrics#checkApps}} and {{TestQueueMetrics#checkResources}} have 8
> and 14 parameters, respectively.
> It is very hard to read the testcases that are using these methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
[ https://issues.apache.org/jira/browse/YARN-8782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak reassigned YARN-8782: Assignee: Gergely Pollak > Fix exception message in Resource.throwExceptionWhenArrayOutOfBound > > > Key: YARN-8782 > URL: https://issues.apache.org/jira/browse/YARN-8782 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Gergely Pollak >Priority: Major > > The exception message contains "please check double check". > This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8468) Limit container sizes per queue in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617371#comment-16617371
]
Weiwei Yang commented on YARN-8468:
---
Generally took a look, it seems good. But it would be helpful if [~bsteinbach]
can upload a up-to-date patch so I can apply to take a closer look. v12 patch
doesn't apply anymore.
Some comments:
1) There are too many checkstyle issues need to be fixed
2) Suggest to add a NPE check in case app is null in following changes in
PlacementConstraintProcessor
{code}
+ String queueName =
+ scheduler.getApplicationAttempt(appAttemptId).getQueueName();
+ Resource maxAllocation = scheduler.getMaximumResourceCapability(queueName);
{code}
Thanks
> Limit container sizes per queue in FairScheduler
>
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch,
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch,
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch,
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch,
> YARN-8468.011.patch, YARN-8468.012.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
>
> The goal of this ticket is to allow this value to be set on a per queue basis.
>
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
>
> Suggested solution:
>
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability() in both FSParentQueue and
> FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * write JUnit tests.
> * update the scheduler documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8468) Limit container sizes per queue in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617371#comment-16617371
]
Weiwei Yang edited comment on YARN-8468 at 9/17/18 11:00 AM:
-
Generally took a look, it seems good. But it would be helpful if [~bsteinbach]
can upload a up-to-date patch so I can apply to take a closer look. v12 patch
doesn't apply anymore.
Some comments:
1) There are too many checkstyle issues need to be fixed
2) Suggest to add a NPE check in case app is null in following changes in
PlacementConstraintProcessor
{code:java}
+ String queueName =
+ scheduler.getApplicationAttempt(appAttemptId).getQueueName();
+ Resource maxAllocation = scheduler.getMaximumResourceCapability(queueName);
{code}
Thanks
was (Author: cheersyang):
Generally took a look, it seems good. But it would be helpful if [~bsteinbach]
can upload a up-to-date patch so I can apply to take a closer look. v12 patch
doesn't apply anymore.
Some comments:
1) There are too many checkstyle issues need to be fixed
2) Suggest to add a NPE check in case app is null in following changes in
PlacementConstraintProcessor
{code}
+ String queueName =
+ scheduler.getApplicationAttempt(appAttemptId).getQueueName();
+ Resource maxAllocation = scheduler.getMaximumResourceCapability(queueName);
{code}
Thanks
> Limit container sizes per queue in FairScheduler
>
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch,
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch,
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch,
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch,
> YARN-8468.011.patch, YARN-8468.012.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
>
> The goal of this ticket is to allow this value to be set on a per queue basis.
>
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
>
> Suggested solution:
>
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability() in both FSParentQueue and
> FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * write JUnit tests.
> * update the scheduler documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8657) User limit calculation should be read-lock-protected within LeafQueue
[
https://issues.apache.org/jira/browse/YARN-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617342#comment-16617342
]
Weiwei Yang commented on YARN-8657:
---
[~leftnoteasy], [~sunilg]
Sorry I just got time to this.
The patch looks fine. My only concern is the performance.
Since computeUserLimitAndSetHeadroom was called in both scheduling-phase and
commit-phase, so even the user limit was stale a bit in the scheduling-phase,
the check put in the commit-phase should be able to guarantee there is no
violations. If we add this sync block, it may reduce the performance as a
side-effect. If I remember correctly, when we were analyzing the conflicts a
few months ago, this was not one of the top-3 cause of conflicts.
BTW, the patch no longer applies to trunk.
Thanks
> User limit calculation should be read-lock-protected within LeafQueue
> -
>
> Key: YARN-8657
> URL: https://issues.apache.org/jira/browse/YARN-8657
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sumana Sathish
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-8657.001.patch
>
>
> When async scheduling is enabled, user limit calculation could be wrong:
> It is possible that scheduler calculated a user_limit, but inside
> {{canAssignToUser}} it becomes staled.
> We need to protect user limit calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617333#comment-16617333
]
Hadoop QA commented on YARN-8750:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 26s{color} | {color:orange} root: The patch generated 1 new + 45 unchanged -
71 fixed = 46 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
42s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8750 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939929/YARN-8750.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 72dc61bb7b62 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Created] (YARN-8782) Fix exception message in Resource.throwExceptionWhenArrayOutOfBound
Szilard Nemeth created YARN-8782: Summary: Fix exception message in Resource.throwExceptionWhenArrayOutOfBound Key: YARN-8782 URL: https://issues.apache.org/jira/browse/YARN-8782 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth The exception message contains "please check double check". This needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications
[
https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617317#comment-16617317
]
Hadoop QA commented on YARN-8775:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
The patch generated 1 new + 7 unchanged - 1 fixed = 8 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
3s{color} | {color:green} hadoop-yarn-server-tests in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8775 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12939945/YARN-8775.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9c795bf34ace 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 33d8327 |
| maven |
[jira] [Commented] (YARN-8775) TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications
[
https://issues.apache.org/jira/browse/YARN-8775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617299#comment-16617299
]
Szilard Nemeth commented on YARN-8775:
--
Hi [~bsteinbach]!
Thanks for this patch!
Couple of comments:
1. {{prepareDirToFail}} now throws an {{IOException}}, this could be removed.
2. All calls to {{prepareDirToFail}} supply 1 as a value of {{tryCounter}}.
Creating a separate method that receives just the directory name and defining a
new method to execute the recursion would make the client code of
{{prepareDirToFail}} more readable.
3. The loop in {{waitForTwoDiskHealthCheck}} runs until {{i <=
maxTimeoutIteration}}. For me, it is more readable to run until {{i <
maxTimeoutIteration}} and then you can move the {{Assert.fail}} call and its
condition outside the loop. This way, you need to provide the loop index as a
separate statement.
> TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File
> modifications
> --
>
> Key: YARN-8775
> URL: https://issues.apache.org/jira/browse/YARN-8775
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn
>Affects Versions: 3.0.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Major
> Attachments: YARN-8775.001.patch
>
>
> The test can fail sometimes when file operations were done during the check
> done by the thread in _LocalDirsHandlerService._
> {code:java}
> java.lang.AssertionError: NodeManager could not identify disk failure.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.yarn.server.TestDiskFailures.verifyDisksHealth(TestDiskFailures.java:239)
> at
> org.apache.hadoop.yarn.server.TestDiskFailures.testDirsFailures(TestDiskFailures.java:202)
> at
> org.apache.hadoop.yarn.server.TestDiskFailures.testLocalDirsFailures(TestDiskFailures.java:99)
> Stderr
> 2018-09-13 08:21:49,822 INFO [main] server.TestDiskFailures
> (TestDiskFailures.java:prepareDirToFail(277)) - Prepared
> /tmp/dist-test-taskjUrf0_/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/org.apache.hadoop.yarn.server.TestDiskFailures/org.apache.hadoop.yarn.server.TestDiskFailures-logDir-nm-0_1
> to fail.
> 2018-09-13 08:21:49,823 INFO [main] server.TestDiskFailures
> (TestDiskFailures.java:prepareDirToFail(277)) - Prepared
> /tmp/dist-test-taskjUrf0_/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/org.apache.hadoop.yarn.server.TestDiskFailures/org.apache.hadoop.yarn.server.TestDiskFailures-logDir-nm-0_3
> to fail.
> 2018-09-13 08:21:49,823 WARN [DiskHealthMonitor-Timer]
> nodemanager.DirectoryCollection (DirectoryCollection.java:checkDirs(283)) -
> Directory
> /tmp/dist-test-taskjUrf0_/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/org.apache.hadoop.yarn.server.TestDiskFailures/org.apache.hadoop.yarn.server.TestDiskFailures-logDir-nm-0_1
> error, Not a directory:
> /tmp/dist-test-taskjUrf0_/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/org.apache.hadoop.yarn.server.TestDiskFailures/org.apache.hadoop.yarn.server.TestDiskFailures-logDir-nm-0_1,
> removing from list of valid directories
> 2018-09-13 08:21:49,824 WARN [DiskHealthMonitor-Timer]
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:initializeLogDir(1329)) - Could not
> initialize log dir
> /tmp/dist-test-taskjUrf0_/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/org.apache.hadoop.yarn.server.TestDiskFailures/org.apache.hadoop.yarn.server.TestDiskFailures-logDir-nm-0_3
> java.io.FileNotFoundException: Destination exists and is not a directory:
> /tmp/dist-test-taskjUrf0_/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/org.apache.hadoop.yarn.server.TestDiskFailures/org.apache.hadoop.yarn.server.TestDiskFailures-logDir-nm-0_3
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:515)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:1081)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:178)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:205)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:747)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:743)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:743)
> at
>
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617258#comment-16617258
]
Sunil Govindan commented on YARN-7505:
--
[~templedf] As code freeze for 3.2 is crossed, moving this Jira to 3.3. Please
feel free to revert if you have concern. Thank you.
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil Govindan updated YARN-7505:
-
Target Version/s: 3.3.0 (was: 3.2.0)
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8340) Capacity Scheduler Intra Queue Preemption Should Work When 3rd or more resources enabled.
[ https://issues.apache.org/jira/browse/YARN-8340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617251#comment-16617251 ] Sunil Govindan commented on YARN-8340: -- As code freeze for 3.2 is crossed, moving this Jira to 3.3. Please feel free to revert if anyone has concerns. Thank you. > Capacity Scheduler Intra Queue Preemption Should Work When 3rd or more > resources enabled. > - > > Key: YARN-8340 > URL: https://issues.apache.org/jira/browse/YARN-8340 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Priority: Critical > > Refer to comment from [~eepayne] and discussion below that: > https://issues.apache.org/jira/browse/YARN-8292?focusedCommentId=16482689=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16482689 > for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8340) Capacity Scheduler Intra Queue Preemption Should Work When 3rd or more resources enabled.
[ https://issues.apache.org/jira/browse/YARN-8340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8340: - Target Version/s: 3.3.0 (was: 3.2.0) > Capacity Scheduler Intra Queue Preemption Should Work When 3rd or more > resources enabled. > - > > Key: YARN-8340 > URL: https://issues.apache.org/jira/browse/YARN-8340 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Priority: Critical > > Refer to comment from [~eepayne] and discussion below that: > https://issues.apache.org/jira/browse/YARN-8292?focusedCommentId=16482689=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16482689 > for details. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8468) Limit container sizes per queue in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617240#comment-16617240 ] Sunil Govindan commented on YARN-8468: -- [~cheersyang], could u also pls take a look. As we crossed 3.2 code freeze, I would like to get this in today or move to next release. Thanks > Limit container sizes per queue in FairScheduler > > > Key: YARN-8468 > URL: https://issues.apache.org/jira/browse/YARN-8468 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Antal Bálint Steinbach >Assignee: Antal Bálint Steinbach >Priority: Critical > Attachments: YARN-8468.000.patch, YARN-8468.001.patch, > YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch, > YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch, > YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch, > YARN-8468.011.patch, YARN-8468.012.patch > > > When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" > to limit the overall size of a container. This applies globally to all > containers and cannot be limited by queue or and is not scheduler dependent. > > The goal of this ticket is to allow this value to be set on a per queue basis. > > The use case: User has two pools, one for ad hoc jobs and one for enterprise > apps. User wants to limit ad hoc jobs to small containers but allow > enterprise apps to request as many resources as needed. Setting > yarn.scheduler.maximum-allocation-mb sets a default value for maximum > container size for all queues and setting maximum resources per queue with > “maxContainerResources” queue config value. > > Suggested solution: > > All the infrastructure is already in the code. We need to do the following: > * add the setting to the queue properties for all queue types (parent and > leaf), this will cover dynamically created queues. > * if we set it on the root we override the scheduler setting and we should > not allow that. > * make sure that queue resource cap can not be larger than scheduler max > resource cap in the config. > * implement getMaximumResourceCapability(String queueName) in the > FairScheduler > * implement getMaximumResourceCapability() in both FSParentQueue and > FSLeafQueue as follows > * expose the setting in the queue information in the RM web UI. > * expose the setting in the metrics etc for the queue. > * write JUnit tests. > * update the scheduler documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8771) CapacityScheduler fails to unreserve when cluster resource contains empty resource type
[
https://issues.apache.org/jira/browse/YARN-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617162#comment-16617162
]
Weiwei Yang commented on YARN-8771:
---
[~Tao Yang], the patch looks good to me. Using isAnyMajorResourceAboveZero
check against the unreserve resource looks reasonable. Comments:
* Instead of adding a new "resource-types-1.xml", can we use
TestResourceUtils#addNewTypesToResources for the tests? I think it doesn't
matter to test with existing gpu or fpga resource correct?
Since isAnyMajorResourceAboveZero was added via YARN-8292, cc-ing [~jlowe],
[~eepayne] and [~leftnoteasy] for cross-check.
> CapacityScheduler fails to unreserve when cluster resource contains empty
> resource type
> ---
>
> Key: YARN-8771
> URL: https://issues.apache.org/jira/browse/YARN-8771
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.2.0
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Critical
> Attachments: YARN-8771.001.patch, YARN-8771.002.patch
>
>
> We found this problem when cluster is almost but not exhausted (93% used),
> scheduler kept allocating for an app but always fail to commit, this can
> blocking requests from other apps and parts of cluster resource can't be used.
> Reproduce this problem:
> (1) use DominantResourceCalculator
> (2) cluster resource has empty resource type, for example: gpu=0
> (3) scheduler allocates container for app1 who has reserved containers and
> whose queue limit or user limit reached(used + required > limit).
> Reference codes in RegularContainerAllocator#assignContainer:
> {code:java}
> // How much need to unreserve equals to:
> // max(required - headroom, amountNeedUnreserve)
> Resource headRoom = Resources.clone(currentResoureLimits.getHeadroom());
> Resource resourceNeedToUnReserve =
> Resources.max(rc, clusterResource,
> Resources.subtract(capability, headRoom),
> currentResoureLimits.getAmountNeededUnreserve());
> boolean needToUnreserve =
> Resources.greaterThan(rc, clusterResource,
> resourceNeedToUnReserve, Resources.none());
> {code}
> For example, resourceNeedToUnReserve can be <8GB, -6 cores, 0 gpu> when
> {{headRoom=<0GB, 8 vcores, 0 gpu>}} and {{capacity=<8GB, 2 vcores, 0 gpu>}},
> needToUnreserve which is the result of {{Resources#greaterThan}} will be
> {{false}}. This is not reasonable because required resource did exceed the
> headroom and unreserve is needed.
> After that, when reaching the unreserve process in
> RegularContainerAllocator#assignContainer, unreserve process will be skipped
> when shouldAllocOrReserveNewContainer is true (when required containers >
> reserved containers) and needToUnreserve is wrongly calculated to be false:
> {code:java}
> if (availableContainers > 0) {
> if (rmContainer == null && reservationsContinueLooking
> && node.getLabels().isEmpty()) {
> // unreserve process can be wrongly skipped when
> shouldAllocOrReserveNewContainer=true and needToUnreserve=false but required
> resource did exceed the headroom
> if (!shouldAllocOrReserveNewContainer || needToUnreserve) {
> ...
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617156#comment-16617156
]
Szilard Nemeth commented on YARN-8750:
--
Fixed the checkstyle issues with patch 003
> Refactor TestQueueMetrics
> -
>
> Key: YARN-8750
> URL: https://issues.apache.org/jira/browse/YARN-8750
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-8750.001.patch, YARN-8750.002.patch,
> YARN-8750.003.patch
>
>
> {{TestQueueMetrics#checkApps}} and {{TestQueueMetrics#checkResources}} have 8
> and 14 parameters, respectively.
> It is very hard to read the testcases that are using these methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-8750:
-
Attachment: YARN-8750.003.patch
> Refactor TestQueueMetrics
> -
>
> Key: YARN-8750
> URL: https://issues.apache.org/jira/browse/YARN-8750
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-8750.001.patch, YARN-8750.002.patch,
> YARN-8750.003.patch
>
>
> {{TestQueueMetrics#checkApps}} and {{TestQueueMetrics#checkResources}} have 8
> and 14 parameters, respectively.
> It is very hard to read the testcases that are using these methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
