[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457573#comment-15457573 ] Naganarasimha G R commented on YARN-5600: - +1 for the approach. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457570#comment-15457570 ] Naganarasimha G R commented on YARN-5549: - Thanks for the patch [~templedf] and [~kasha] for the comments for the config. Given that there is already jira raised to ATSV2 to log the command which can be securedly access later makes sense, but i am +0 for this patch because to get this log it requires a restart of the NM to enable the modified config and as [~jlowe] mentioned we will not be sure where the container will be relaunched again in this node on failure. So temporarily if the node has issues this log and config will be helpfull but in case app's command has issues this approach offers no help. Other than this patch looks fine ! > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch, > YARN-5549.006.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5576) Core change to localize resource while container is running
[
https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457505#comment-15457505
]
Hadoop QA commented on YARN-5576:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 6 new + 704 unchanged - 15 fixed = 710 total (was 719)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 31s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 2s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826765/YARN-5576.4.patch |
| JIRA Issue | YARN-5576 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a1ae6b7e23a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0690f09 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13006/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13006/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Core change to localize resource while container is running
> ---
>
> Key: YARN-5576
> URL:
[jira] [Commented] (YARN-5552) Add Builder methods for common yarn API records
[
https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457470#comment-15457470
]
Hadoop QA commented on YARN-5552:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 36
new + 115 unchanged - 8 fixed = 151 total (was 123) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 38s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 1s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 5s
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| |
org.apache.hadoop.yarn.client.api.AMRMClient$ContainerRequest$ContainerRequestBuilder.nodes(String[])
may expose internal representation by storing an externally mutable object
into AMRMClient$ContainerRequest$ContainerRequestBuilder.nodes At
AMRMClient.java:by storing an externally mutable object into
AMRMClient$ContainerRequest$ContainerRequestBuilder.nodes At
AMRMClient.java:[line 398] |
| |
org.apache.hadoop.yarn.client.api.AMRMClient$ContainerRequest$ContainerRequestBuilder.racks(String[])
[jira] [Comment Edited] (YARN-5576) Core change to localize resource while container is running
[ https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457446#comment-15457446 ] Jian He edited comment on YARN-5576 at 9/2/16 4:08 AM: --- Thanks for the review, Arun ! bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) It is not allowed, the method in ContainerManagerImpl only allows localization while running bq. I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status. bq. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE. I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it. bq. : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest That's done by IDE auto fixing some unused imports, I edited it manually. was (Author: jianhe): bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) It is not allowed, the method in ContainerManagerImpl only allows localization while running bq. I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status. bq. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE. I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it. bq. : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest That's done by IDE auto fixing some unused imports, I edited it manually. > Core change to localize resource while container is running > --- > > Key: YARN-5576 > URL: https://issues.apache.org/jira/browse/YARN-5576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5576.1.patch, YARN-5576.2.patch, YARN-5576.3.patch, > YARN-5576.4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5576) Core change to localize resource while container is running
[ https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-5576: -- Attachment: YARN-5576.4.patch > Core change to localize resource while container is running > --- > > Key: YARN-5576 > URL: https://issues.apache.org/jira/browse/YARN-5576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5576.1.patch, YARN-5576.2.patch, YARN-5576.3.patch, > YARN-5576.4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5576) Core change to localize resource while container is running
[ https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457446#comment-15457446 ] Jian He edited comment on YARN-5576 at 9/2/16 4:06 AM: --- bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) It is not allowed, the method in ContainerManagerImpl only allows localization while running bq. I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status. bq. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE. I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it. bq. : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest That's done by IDE auto fixing some unused imports, I edited it manually. was (Author: jianhe): bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) It is not allowed, the method in ContainerManagerImpl only allows localization while running bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) bq. I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status. bq. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE. I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it. bq. : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest That's done by IDE auto fixing some unused imports, I edited it manually. > Core change to localize resource while container is running > --- > > Key: YARN-5576 > URL: https://issues.apache.org/jira/browse/YARN-5576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5576.1.patch, YARN-5576.2.patch, YARN-5576.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5576) Core change to localize resource while container is running
[ https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457446#comment-15457446 ] Jian He commented on YARN-5576: --- bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) It is not allowed, the method in ContainerManagerImpl only allows localization while running bq. We need to override the ContainerManagerImpl::localize() method in the QueuingContainerManagerImpl. Re-localization should not be allowed if the container is currently queued (not yet running) bq. I only see entries added to ResourceSet::resourcesFailedToBeLocalized set. Shouldnt we remove these once the AM is notified of the failure ? Also, Shouldn't these be notified back to the AM ? or we are just relying on the diagnostic string sent to the AM in the GetContainerStatus response to notify the AM ? The status part is not yet implemented as mentioned in the parent jira. It'll will be done once the requirement is clear. Earlier I was thinking these will be sent as part of container status. bq. wondering if we should have another RE_LOCALIZE_CONTAINER_RESOURCE event in the ResourceLocalizationService to distinguish from the localization needed for container initialization and correspondingly send different events to the Container. Or maybe for the timebeing, we should just rename INIT_CONTAINER_RESOURCE to LOCALIZE_CONTAINER_RESOURCE. I don't think adding new events type for doing the same thing is necessary at this point. This will also add additional complexity as you need to conditionally sends different types of events. The goal is to reuse existing code. I can rename it. bq. : spurious change in the imports of ContainerImpl and BaseAMRMProxyTest That's done by IDE auto fixing some unused imports, I edited it manually. > Core change to localize resource while container is running > --- > > Key: YARN-5576 > URL: https://issues.apache.org/jira/browse/YARN-5576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5576.1.patch, YARN-5576.2.patch, YARN-5576.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5552) Add Builder methods for common yarn API records
[ https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457307#comment-15457307 ] Tao Jie commented on YARN-5552: --- Attached patch fixed checkstyle and failed test. [~asuresh], [~kasha], would you mind giving it a review? Thank you! > Add Builder methods for common yarn API records > --- > > Key: YARN-5552 > URL: https://issues.apache.org/jira/browse/YARN-5552 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Tao Jie > Attachments: YARN-5552.000.patch, YARN-5552.001.patch, > YARN-5552.002.patch, YARN-5552.003.patch > > > Currently yarn API records such as ResourceRequest, AllocateRequest/Respone > as well as AMRMClient.ContainerRequest have multiple constructors / > newInstance methods. This makes it very difficult to add new fields to these > records. > It would probably be better if we had Builder classes for many of these > records, which would make evolution of these records a bit easier. > (suggested by [~kasha]) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5552) Add Builder methods for common yarn API records
[ https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-5552: -- Attachment: YARN-5552.003.patch > Add Builder methods for common yarn API records > --- > > Key: YARN-5552 > URL: https://issues.apache.org/jira/browse/YARN-5552 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Tao Jie > Attachments: YARN-5552.000.patch, YARN-5552.001.patch, > YARN-5552.002.patch, YARN-5552.003.patch > > > Currently yarn API records such as ResourceRequest, AllocateRequest/Respone > as well as AMRMClient.ContainerRequest have multiple constructors / > newInstance methods. This makes it very difficult to add new fields to these > records. > It would probably be better if we had Builder classes for many of these > records, which would make evolution of these records a bit easier. > (suggested by [~kasha]) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4948) Support node labels store in zookeeper
[ https://issues.apache.org/jira/browse/YARN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457265#comment-15457265 ] Naganarasimha G R commented on YARN-4948: - Sorry [~wjlei], my bad! thought had replied for it. As [~wangda] mentioned had discussecd on this issue and to add on top of this Constraint label[YARN-3409] would add more data for each node, so thought we will wait once the approach is finalized for it and then take account of size of data and frequency of updates and then confirm whether ZK store for Node Labels would be ideal. Because of these open points held on to further review. > Support node labels store in zookeeper > -- > > Key: YARN-4948 > URL: https://issues.apache.org/jira/browse/YARN-4948 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: jialei weng >Assignee: jialei weng > Attachments: YARN-4948.001.patch, YARN-4948.002.patch, > YARN-4948.003.patch, YARN-4948.006.patch, YARN-4948.007.patch > > > Support node labels store in zookeeper. The main scenario for this is to give > a way to decouple yarn with HDFS. Since nodelabel is a very important data > for yarn, if hdfs down, yarn will fail to start up,too. So it is meaningful > for make yarn much independence when user serve both yarn and HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5525) Make log aggregation service class configurable
[
https://issues.apache.org/jira/browse/YARN-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457179#comment-15457179
]
Hadoop QA commented on YARN-5525:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 36
new + 450 unchanged - 8 fixed = 486 total (was 458) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 13s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 15s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826747/YARN-5525.v3.patch |
| JIRA Issue | YARN-5525 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 468b5edb7832 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0690f09 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/13004/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457133#comment-15457133
]
Hadoop QA commented on YARN-5331:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 43s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826739/YARN-5331.002.patch |
| JIRA Issue | YARN-5331 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ff6fc1804d7f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0690f09 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13003/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13003/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Extend RLESparseResourceAllocation with period for supporting recurring
> reservations in YARN ReservationSystem
> --
>
> Key: YARN-5331
> URL: https://issues.apache.org/jira/browse/YARN-5331
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Sangeetha Abdu
[jira] [Updated] (YARN-5525) Make log aggregation service class configurable
[ https://issues.apache.org/jira/browse/YARN-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-5525: --- Attachment: YARN-5525.v3.patch > Make log aggregation service class configurable > --- > > Key: YARN-5525 > URL: https://issues.apache.org/jira/browse/YARN-5525 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Reporter: Giovanni Matteo Fumarola >Assignee: Botong Huang >Priority: Minor > Attachments: YARN-5525.v1.patch, YARN-5525.v2.patch, > YARN-5525.v3.patch > > > Make the log aggregation class configurable and extensible, so that > alternative log aggregation behaviors like app specific log aggregation > directory, log aggregation format can be implemented and plugged in. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5613) Fair Scheduler can assign containers from blacklisted nodes
[
https://issues.apache.org/jira/browse/YARN-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457093#comment-15457093
]
Daniel Templeton commented on YARN-5613:
Hold the phone. Unfortunately, I'm persistent, and with more testing I was
eventually able to reproduce the error, even with this patch. [sigh]
> Fair Scheduler can assign containers from blacklisted nodes
> ---
>
> Key: YARN-5613
> URL: https://issues.apache.org/jira/browse/YARN-5613
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5613.001.patch
>
>
> The {{FairScheduler.allocate()}} makes its resource request before it updates
> the blacklist. If the scheduler processes the resource request before the
> allocating thread updates the blacklist, the scheduler can assign containers
> that are on nodes in the blacklist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5585) [Atsv2] Add a new filter fromId in REST endpoints
[ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457081#comment-15457081 ] Li Lu commented on YARN-5585: - Thanks [~varun_saxena]! I think the discussion so far indicates that implementing fromId on containers seems to be hard? [~rohithsharma] could you please verify if Varun's current plan works for your use case? I'm trying to get a big picture for the use cases of fromId. Thanks! > [Atsv2] Add a new filter fromId in REST endpoints > - > > Key: YARN-5585 > URL: https://issues.apache.org/jira/browse/YARN-5585 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > TimelineReader REST API's provides lot of filters to retrieve the > applications. Along with those, it would be good to add new filter i.e fromId > so that entities can be retrieved after the fromId. > Example : If applications are stored database, app-1 app-2 ... app-10. > *getApps?limit=5* gives app-1 to app-10. But to retrieve next 5 apps, it is > difficult. > So proposal is to have fromId in the filter like > *getApps?limit=5&=app-5* which gives list of apps from app-6 to > app-10. > This is very useful for pagination in web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5601) Make the RM epoch base value configurable
[ https://issues.apache.org/jira/browse/YARN-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457070#comment-15457070 ] Subru Krishnan commented on YARN-5601: -- The test case failure seems unrelated to the patch. > Make the RM epoch base value configurable > - > > Key: YARN-5601 > URL: https://issues.apache.org/jira/browse/YARN-5601 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5601-YARN-2915-v1.patch, > YARN-5601-YARN-2915-v2.patch, YARN-5601-YARN-2915-v3.patch > > > Currently the epoch always starts from zero. This can cause container ids to > conflict for an application under Federation that spans multiple RMs > concurrently. This JIRA proposes to make the RM epoch base value configurable > which will allow us to avoid conflicts by setting different values for each > RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4231) Node Label should support pluggable storage
[ https://issues.apache.org/jira/browse/YARN-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan resolved YARN-4231. -- Resolution: Duplicate This is already included by: YARN-4405, closing as dup. > Node Label should support pluggable storage > --- > > Key: YARN-4231 > URL: https://issues.apache.org/jira/browse/YARN-4231 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan > > We need to support pluggable storage for node label. Currently it requires > file system supports append. Some filesystem doesn't support append, for > example, Azure FileSystem. We'd better make storage implementation pluggable > so different filesystem can choose their best approach to implement node > label storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4948) Support node labels store in zookeeper
[ https://issues.apache.org/jira/browse/YARN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457059#comment-15457059 ] Wangda Tan commented on YARN-4948: -- Sorry for the late response, I just discussed with [~subru], and talked to [~naganarasimha...@apache.org] few months before. Thanks for taking up this JIRA, [~wjlei], I would suggest: - If you want it works with WASB file system (or other file system doesn't support append), you can try to use NonAppendableFSNodeLabelStore. - If you don't want YARN fails because of HDFS failures, you can setup yarn.node-labels.fs-store.retry-policy-spec - Reasonable number of nodes and reasonable frequency of node label updating is already supported by existing stores. There will be a lot of issue if #nodemanager goes beyond 2+, for example, scheduler itself becomes a bottleneck. I would say adding the new ZK store impl will add extra overhead to code maintenance. And since pluggable node label store is supported, if you really want to run a ZK store, you can configure it in yarn-site.xml. So I suggest to keep this open until this becomes a common requirement for node label. > Support node labels store in zookeeper > -- > > Key: YARN-4948 > URL: https://issues.apache.org/jira/browse/YARN-4948 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: jialei weng >Assignee: jialei weng > Attachments: YARN-4948.001.patch, YARN-4948.002.patch, > YARN-4948.003.patch, YARN-4948.006.patch, YARN-4948.007.patch > > > Support node labels store in zookeeper. The main scenario for this is to give > a way to decouple yarn with HDFS. Since nodelabel is a very important data > for yarn, if hdfs down, yarn will fail to start up,too. So it is meaningful > for make yarn much independence when user serve both yarn and HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[
https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457045#comment-15457045
]
Li Lu commented on YARN-5561:
-
I've got offline discussions with several folks on if we should have concepts
like "app-attempt" and "container". From a designer's perspective, app-attempts
and containers should not be included in timeline APIs, but from YARN users
perspective, requesting app-attempt and container level information seems to be
very natural operations, especially since both concepts are top level concepts
in YARN. So I'm relatively fine with having terms like "containers" and
"app-attempts" exposed in timeline APIs, but we may want to be very careful to
not to give an impression that attempts and containers are on the hierarchical
order as flows and flowruns.
So how about having two different hierarchical orders:
Order 1, native timeline order: cluster, user, flow, flow-run, application,
entity
Order 2, YARN application order: application, app-attempt, container
Once we're not mixing the two orders in APIs, the logic should be clear.
Thoughts?
> [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and
> entities via REST
> ---
>
> Key: YARN-5561
> URL: https://issues.apache.org/jira/browse/YARN-5561
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Attachments: YARN-5561.patch, YARN-5561.v0.patch
>
>
> ATSv2 model lacks retrieval of {{list-of-all-apps}},
> {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via
> REST API's. And also it is required to know about all the entities in an
> applications.
> It is pretty much highly required these URLs for Web UI.
> New REST URL would be
> # GET {{/ws/v2/timeline/apps}}
> # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}.
> # GET
> {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}}
> # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of
> entities that can be queried.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS
[ https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457039#comment-15457039 ] Yufei Gu commented on YARN-3432: The last patch seems good to me generally. I wonder if we can add unit tests for these since the different logics between FS and CS and these logic might change sometimes. > Cluster metrics have wrong Total Memory when there is reserved memory on CS > --- > > Key: YARN-3432 > URL: https://issues.apache.org/jira/browse/YARN-3432 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Thomas Graves >Assignee: Brahma Reddy Battula > Attachments: YARN-3432-002.patch, YARN-3432-003.patch, YARN-3432.patch > > > I noticed that when reservations happen when using the Capacity Scheduler, > the UI and web services report the wrong total memory. > For example. I have a 300GB of total memory in my cluster. I allocate 50 > and I reserve 10. The cluster metrics for total memory get reported as 290GB. > This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps > there is a difference between fair scheduler and capacity scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5601) Make the RM epoch base value configurable
[
https://issues.apache.org/jira/browse/YARN-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457034#comment-15457034
]
Hadoop QA commented on YARN-5601:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 44s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 54s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s
{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 48s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 2
new + 424 unchanged - 1 fixed = 426 total (was 425) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 44s {color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 49s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826719/YARN-5601-YARN-2915-v3.patch
|
| JIRA Issue | YARN-5601 |
| Optional Tests | asflicense findbugs xml compile javac javadoc
mvninstall mvnsite
[jira] [Updated] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangeetha Abdu Jyothi updated YARN-5331: Attachment: YARN-5331.002.patch > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5331.001.patch, YARN-5331.002.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5323) Policies APIs (for Router and AMRMProxy policies)
[
https://issues.apache.org/jira/browse/YARN-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456959#comment-15456959
]
Hadoop QA commented on YARN-5323:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} YARN-2915 passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s {color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 11s
{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 17s {color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 15s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826726/YARN-5323-YARN-2915.08.patch
|
| JIRA Issue | YARN-5323 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a512bd6960b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / bd906b2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| checkstyle |
[jira] [Commented] (YARN-5323) Policies APIs (for Router and AMRMProxy policies)
[ https://issues.apache.org/jira/browse/YARN-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456922#comment-15456922 ] Carlo Curino commented on YARN-5323: [~giovanni.fumarola] thanks for the comments, I addressed them in the latest (.08) patch. > Policies APIs (for Router and AMRMProxy policies) > - > > Key: YARN-5323 > URL: https://issues.apache.org/jira/browse/YARN-5323 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-5323-YARN-2915.05.patch, > YARN-5323-YARN-2915.06.patch, YARN-5323-YARN-2915.07.patch, > YARN-5323-YARN-2915.08.patch, YARN-5323.01.patch, YARN-5323.02.patch, > YARN-5323.03.patch, YARN-5323.04.patch > > > This JIRA tracks APIs for the policies that will guide the Router and > AMRMProxy decisions on where to fwd the jobs submission/query requests as > well as ResourceRequests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5323) Policies APIs (for Router and AMRMProxy policies)
[ https://issues.apache.org/jira/browse/YARN-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-5323: --- Attachment: YARN-5323-YARN-2915.08.patch > Policies APIs (for Router and AMRMProxy policies) > - > > Key: YARN-5323 > URL: https://issues.apache.org/jira/browse/YARN-5323 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-5323-YARN-2915.05.patch, > YARN-5323-YARN-2915.06.patch, YARN-5323-YARN-2915.07.patch, > YARN-5323-YARN-2915.08.patch, YARN-5323.01.patch, YARN-5323.02.patch, > YARN-5323.03.patch, YARN-5323.04.patch > > > This JIRA tracks APIs for the policies that will guide the Router and > AMRMProxy decisions on where to fwd the jobs submission/query requests as > well as ResourceRequests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5601) Make the RM epoch base value configurable
[ https://issues.apache.org/jira/browse/YARN-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5601: - Attachment: YARN-5601-YARN-2915-v3.patch Fixing findbugs exclusion syntax > Make the RM epoch base value configurable > - > > Key: YARN-5601 > URL: https://issues.apache.org/jira/browse/YARN-5601 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5601-YARN-2915-v1.patch, > YARN-5601-YARN-2915-v2.patch, YARN-5601-YARN-2915-v3.patch > > > Currently the epoch always starts from zero. This can cause container ids to > conflict for an application under Federation that spans multiple RMs > concurrently. This JIRA proposes to make the RM epoch base value configurable > which will allow us to avoid conflicts by setting different values for each > RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5567) Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang resolved YARN-5567.
--
Resolution: Fixed
Hadoop Flags: Incompatible change,Reviewed (was: Reviewed)
Release Note: Prior to this fix, the NodeManager will ignore any non-zero
exit code for any script in the yarn.nodemanager.health-checker.script.path
property. With this change, any syntax errors in the health checking script
will get flagged as an error in the same fashion (likely exit code 1) that the
script detecting a health issue. (was: Prior to this fix, the NodeManager will
ignore any non-zero exit code for any script in the
yarn.nodemanager.health-checker.script.path property.)
Thanks [~andrew.wang] for the info. Thanks to [~wilfreds] for bringing up the
issue and thanks again to [~yufeigu] and [~Naganarasimha] for your comments.
Reverted from branch-2.8 and branch-2. Marked as incompatible.
> Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
> --
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS
[ https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456787#comment-15456787 ] Karthik Kambatla commented on YARN-3432: The last patch here seems reasonable. [~yufeigu] has offered to confirm the same. > Cluster metrics have wrong Total Memory when there is reserved memory on CS > --- > > Key: YARN-3432 > URL: https://issues.apache.org/jira/browse/YARN-3432 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Thomas Graves >Assignee: Brahma Reddy Battula > Attachments: YARN-3432-002.patch, YARN-3432-003.patch, YARN-3432.patch > > > I noticed that when reservations happen when using the Capacity Scheduler, > the UI and web services report the wrong total memory. > For example. I have a 300GB of total memory in my cluster. I allocate 50 > and I reserve 10. The cluster metrics for total memory get reported as 290GB. > This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps > there is a difference between fair scheduler and capacity scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5613) Fair Scheduler can assign containers from blacklisted nodes
[
https://issues.apache.org/jira/browse/YARN-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456757#comment-15456757
]
Hadoop QA commented on YARN-5613:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 27s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 28s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826688/YARN-5613.001.patch |
| JIRA Issue | YARN-5613 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 139262f7841a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 76cd81f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12999/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12999/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fair Scheduler can assign containers from blacklisted nodes
> ---
>
> Key: YARN-5613
> URL: https://issues.apache.org/jira/browse/YARN-5613
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>
[jira] [Comment Edited] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456302#comment-15456302 ] Karthik Kambatla edited comment on YARN-5549 at 9/1/16 10:01 PM: - The patch looks good, except for the following nits: # The static imports make sense, but the rest of the code doesn't do it. Can we leave them out? If we choose to keep it, when logging "REDACTED etc." should use the statically imported version. # Some of the changes are unrelated to the patch. To minimize conflicts, I would like for us to leave these out. was (Author: kasha): The patch looks good, except for the following nits: # The static imports make sense, but the rest of the code doesn't do it. Can we leave out. If we choose to keep it, when logging "REDACTED etc." # Some of the changes are unrelated to the patch. To minimize conflicts, I would like for us to leave these out. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch, > YARN-5549.006.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5566) Client-side NM graceful decom is not triggered when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456768#comment-15456768 ] Hudson commented on YARN-5566: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10387 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10387/]) YARN-5566. Client-side NM graceful decom is not triggered when jobs (kasha: rev 74f4bae45597f4794e99e33309130ddff647b21f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java > Client-side NM graceful decom is not triggered when jobs finish > --- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5566) Client-side NM graceful decom is not triggered when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-5566: --- Fix Version/s: 3.0.0-alpha2 2.9.0 > Client-side NM graceful decom is not triggered when jobs finish > --- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5567) Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456763#comment-15456763
]
Andrew Wang commented on YARN-5567:
---
Looking at git log, it looks like this will also be included in alpha1. I
rebranched right before sending the RC, and we picked up this JIRA as part of
that.
So, I think we're good?
> Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
> --
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5566) Client-side NM graceful decom is not triggered when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456746#comment-15456746 ] Karthik Kambatla commented on YARN-5566: Thanks [~djp] for the review, and [~rkanter] for the patch. Just committed this to trunk and branch-2. Leaving the JIRA open for 2.8 patch. > Client-side NM graceful decom is not triggered when jobs finish > --- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456727#comment-15456727
]
Wangda Tan commented on YARN-4945:
--
bq. Trying to do this coordination seems to me to be quite complicated.
Using logic similar to {{deductPreemptableResourcesBasedSelectedCandidates}}
should be able to achieve this, and I think it doesn't bring too many
complexities to the implementation.
bq. Would it be sufficient to just avoid preempting during the intra-queue
policies if there are already containers in the selectedContainers list?
If we want to avoid excessive preemption, It may not sufficient to me. We need
to adjust ideal / to-preempt resource properly as well.
> [Umbrella] Capacity Scheduler Preemption Within a queue
> ---
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5567) Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456719#comment-15456719
]
Ray Chiang commented on YARN-5567:
--
I think they've already started the vote on alpha-1 RC0. I believe it will
show up in alpha2 automatically.
[~andrew.wang], let me know how to handle this situation. Thanks.
> Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
> --
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456717#comment-15456717 ] Wangda Tan commented on YARN-4945: -- bq. I think we do need several intra-queue configs that are separate from the existing (inter-queue) ones. For inter-queue vs. intra-queue, I think we need a separate one at least for total_preemption_per_round and max_ignored_over_capacity, and maybe even for natural_termination_factor and max_wait_before_kill. We definitely need some parameter for per-queue preemption setting, max-to-have in my mind is: - Minimum queue's used capacity to trigger preemption - Total preemption per round - Max ignored over capacity (for user limit) I suggest to add only must-to-have parameters, more options make a feature harder to be used. So I would prefer to not add things like natural-termination-factor / max-wait-before-kill for now. > [Umbrella] Capacity Scheduler Preemption Within a queue > --- > > Key: YARN-4945 > URL: https://issues.apache.org/jira/browse/YARN-4945 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan > Attachments: Intra-Queue Preemption Use Cases.pdf, > IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch, > YARN-2009-wip.patch > > > This is umbrella ticket to track efforts of preemption within a queue to > support features like: > YARN-2009. YARN-2113. YARN-4781. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5566) Client-side NM graceful decom doesn't trigger when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-5566: --- Summary: Client-side NM graceful decom doesn't trigger when jobs finish (was: client-side NM graceful decom doesn't trigger when jobs finish) > Client-side NM graceful decom doesn't trigger when jobs finish > -- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5566) Client-side NM graceful decom is not triggered when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-5566: --- Summary: Client-side NM graceful decom is not triggered when jobs finish (was: Client-side NM graceful decom doesn't trigger when jobs finish) > Client-side NM graceful decom is not triggered when jobs finish > --- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456698#comment-15456698
]
Hadoop QA commented on YARN-5608:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 1s
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826704/YARN-5608.004.patch |
| JIRA Issue | YARN-5608 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f56c1ed793f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 76cd81f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13000/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13000/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch,
> YARN-5608.004.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
>
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[
https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456686#comment-15456686
]
Hadoop QA commented on YARN-5549:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 59s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 39s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826691/YARN-5549.006.patch |
| JIRA Issue | YARN-5549 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 807283a6d847 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3069df7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12996/testReport/ |
| modules | C:
[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS
[
https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456683#comment-15456683
]
Hadoop QA commented on YARN-3432:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m 21s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12777063/YARN-3432-003.patch |
| JIRA Issue | YARN-3432 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b98b75af083b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3069df7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12997/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12997/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Cluster metrics have wrong Total Memory when there is reserved memory on CS
> ---
>
> Key: YARN-3432
> URL: https://issues.apache.org/jira/browse/YARN-3432
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, resourcemanager
>Affects
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456672#comment-15456672 ] Karthik Kambatla commented on YARN-5549: +1 pending Jenkins. Will commit this tomorrow morning (PT) if no one has any objections. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch, > YARN-5549.006.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[
https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-5139:
-
Attachment: Explanantions of Global Scheduling (YARN-5139)
Implementation.pdf
I've uploaded a document to explain global scheduling implementation in the
attached patch.
Hope it can make you easier understand the overall logic / changes without
reviewing the half-MB patch.
This is the same doc on github:
https://github.com/leftnoteasy/hadoop/blob/global-scheduling-3/global-scheduling-explaination.md.
Hopefully it has better syntax highlight support than PDF.
Thanks [~vinodkv] for offline suggestions.
> [Umbrella] Move YARN scheduler towards global scheduler
> ---
>
> Key: YARN-5139
> URL: https://issues.apache.org/jira/browse/YARN-5139
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: Explanantions of Global Scheduling (YARN-5139)
> Implementation.pdf,
> YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf,
> YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf,
> wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, wip-3.YARN-5139.patch,
> wip-4.YARN-5139.patch
>
>
> Existing YARN scheduler is based on node heartbeat. This can lead to
> sub-optimal decisions because scheduler can only look at one node at the time
> when scheduling resources.
> Pseudo code of existing scheduling logic looks like:
> {code}
> for node in allNodes:
>Go to parentQueue
> Go to leafQueue
> for application in leafQueue.applications:
>for resource-request in application.resource-requests
> try to schedule on node
> {code}
> Considering future complex resource placement requirements, such as node
> constraints (give me "a && b || c") or anti-affinity (do not allocate HBase
> regionsevers and Storm workers on the same host), we may need to consider
> moving YARN scheduler towards global scheduling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456618#comment-15456618 ] Ray Chiang commented on YARN-5549: -- I like the actionable log message in v6 better too. +1 > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch, > YARN-5549.006.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5608:
---
Attachment: YARN-5608.004.patch
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch,
> YARN-5608.004.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5608:
---
Attachment: (was: YARN-5608.004.patch)
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5608:
---
Attachment: YARN-5608.004.patch
Looks like the bind exception was spurious. Upping the retry count appears to
resolve the issue.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456586#comment-15456586 ] Jason Lowe commented on YARN-5549: -- +1 lgtm. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch, > YARN-5549.006.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5601) Make the RM epoch base value configurable
[ https://issues.apache.org/jira/browse/YARN-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5601: - Attachment: YARN-5601-YARN-2915-v2.patch Adding findbug exclusion as suggested by [~jianhe] > Make the RM epoch base value configurable > - > > Key: YARN-5601 > URL: https://issues.apache.org/jira/browse/YARN-5601 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5601-YARN-2915-v1.patch, > YARN-5601-YARN-2915-v2.patch > > > Currently the epoch always starts from zero. This can cause container ids to > conflict for an application under Federation that spans multiple RMs > concurrently. This JIRA proposes to make the RM epoch base value configurable > which will allow us to avoid conflicts by setting different values for each > RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456550#comment-15456550
]
Hadoop QA commented on YARN-5612:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
43s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 35s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 9s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826689/YARN-5612-YARN-2915.v2.patch
|
| JIRA Issue | YARN-5612 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0e8bfca9805d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / c77269d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12995/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12995/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Return SubClusterId in
> FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
> --
>
> Key: YARN-5612
> URL: https://issues.apache.org/jira/browse/YARN-5612
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni
[jira] [Commented] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS
[ https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456537#comment-15456537 ] Nathan Roberts commented on YARN-3432: -- Recently ran into this issue again. Just seems wrong that totalMB fluctuates significantly due to reservedMB moving around. Since we can't reserve beyond the size of the cluster, it seems fine to have Total MB = Available MB + Allocated MB + Reserved MB for the capacity scheduler. I guess this could be an incompatible change to anyone who's worked around the problem by adding reservedMB to totalMB. [~vinodkv], [~ka...@cloudera.com], others have comments on this aspect? > Cluster metrics have wrong Total Memory when there is reserved memory on CS > --- > > Key: YARN-3432 > URL: https://issues.apache.org/jira/browse/YARN-3432 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Thomas Graves >Assignee: Brahma Reddy Battula > Attachments: YARN-3432-002.patch, YARN-3432-003.patch, YARN-3432.patch > > > I noticed that when reservations happen when using the Capacity Scheduler, > the UI and web services report the wrong total memory. > For example. I have a 300GB of total memory in my cluster. I allocate 50 > and I reserve 10. The cluster metrics for total memory get reported as 290GB. > This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps > there is a difference between fair scheduler and capacity scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
[ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456530#comment-15456530 ] Arun Suresh commented on YARN-5079: --- Given that now the Slider core is being merged with Yarn. Should new features requests for slider be filed as a YARN or SLIDER jira ? > [Umbrella] Native YARN framework layer for services and beyond > -- > > Key: YARN-5079 > URL: https://issues.apache.org/jira/browse/YARN-5079 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > > (See overview doc at YARN-4692, modifying and copy-pasting some of the > relevant pieces and sub-section 3.3.1 to track the specific sub-item.) > (This is a companion to YARN-4793 in our effort to simplify the entire story, > but focusing on APIs) > So far, YARN by design has restricted itself to having a very low-level API > that can support any type of application. Frameworks like Apache Hadoop > MapReduce, Apache Tez, Apache Spark, Apache REEF, Apache Twill, Apache Helix > and others ended up exposing higher level APIs that end-users can directly > leverage to build their applications on top of YARN. On the services side, > Apache Slider has done something similar. > With our current attention on making services first-class and simplified, > it's time to take a fresh look at how we can make Apache Hadoop YARN support > services well out of the box. Beyond the functionality that I outlined in the > previous sections in the doc on how NodeManagers can be enhanced to help > services, the biggest missing piece is the framework itself. There is a lot > of very important functionality that a services' framework can own together > with YARN in executing services end-to-end. > In this JIRA I propose we look at having a native Apache Hadoop framework for > running services natively on YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5549: --- Attachment: YARN-5549.006.patch Here's a patch minus my spurious improvements and the static imports. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch, > YARN-5549.006.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[
https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456471#comment-15456471
]
Subru Krishnan commented on YARN-5612:
--
Thanks [~giovanni.fumarola] for the fix, it looks fairly straightforward.
Can you revert the redudant change (whitespaces) in
{{TestFederationStateStoreFacade}} as that will raise checkstyle warning:
{code}
.asList(new Boolean[][] { { Boolean.FALSE }, { Boolean.TRUE } });
{code}
> Return SubClusterId in
> FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
> --
>
> Key: YARN-5612
> URL: https://issues.apache.org/jira/browse/YARN-5612
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-5612-YARN-2915.v1.patch
>
>
> This JIRA tracks the returning of SubClusterId from
> FederationStateStoreFacade#addApplicationHomeSubCluster.
> in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659],
> to handle better fail-over scenario the response needs SubClusterId. This is
> bubbling up the change in YARN-5519.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5613) Fair Scheduler can assign containers from blacklisted nodes
Daniel Templeton created YARN-5613:
--
Summary: Fair Scheduler can assign containers from blacklisted
nodes
Key: YARN-5613
URL: https://issues.apache.org/jira/browse/YARN-5613
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.8.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton
The {{FairScheduler.allocate()}} makes its resource request before it updates
the blacklist. If the scheduler processes the resource request before the
allocating thread updates the blacklist, the scheduler can assign containers
that are on nodes in the blacklist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5613) Fair Scheduler can assign containers from blacklisted nodes
[
https://issues.apache.org/jira/browse/YARN-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5613:
---
Attachment: YARN-5613.001.patch
The issue shows up when running
{{TestAMRMClient.testAMRMClientWithBlacklist()}} using fair scheduler as the
default. The error is:
{noformat}
expected:<0> but was:<1>
Stacktrace
java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocationWithBlacklist(TestAMRMClient.java:537)
Standard Output
{noformat}
This patch appears to resolve the issue, but I'm still testing.
> Fair Scheduler can assign containers from blacklisted nodes
> ---
>
> Key: YARN-5613
> URL: https://issues.apache.org/jira/browse/YARN-5613
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5613.001.patch
>
>
> The {{FairScheduler.allocate()}} makes its resource request before it updates
> the blacklist. If the scheduler processes the resource request before the
> allocating thread updates the blacklist, the scheduler can assign containers
> that are on nodes in the blacklist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3854) Add localization support for docker images
[ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456432#comment-15456432 ] Shane Kumpf commented on YARN-3854: --- Today, when credential issues occur during the implicit docker pull, the "image not found" error is hidden away in the NM logs. Initially my thought was that failing fast and exposing the error to the user via the application logs would be sufficient without credential validation, however, after testing, docker pull has some differing behaviors regarding various credential issue scenerios. If the credentials file is missing or the image does not exist in the registry, docker pull returns an "image not found" error (and rc=1). As a result, it will be difficult to validate if the user supplied a non-existent image or if it's a missing credentials problem. This deficiency could have been easily addressed via a log message and some documentation though. If the credentials exist, but are incorrect (incorrect auth token), it's more challenging to address as docker falls back to the interactive docker login prompt. Ideally, docker would provide a command line option to avoid this fall back to make the behavior consistent, and it may be worth starting that discussion within the docker community towards adding this option. Otherwise, we would need to try to handle detecting the difference between a long running docker pull and a docker pull that is waiting for the user to input credentials. I'll note that, unfortunately, I'm not aware of any options in the docker client CLI to validate credentials. It may be possible by resorting to checks against the registry REST API, but ideally, we want to stick with the docker client CLI as we've done so far. I'll look into a clean way to address validation. > Add localization support for docker images > -- > > Key: YARN-3854 > URL: https://issues.apache.org/jira/browse/YARN-3854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-3854-branch-2.8.001.patch, > YARN-3854_Localization_support_for_Docker_image_v1.pdf, > YARN-3854_Localization_support_for_Docker_image_v2.pdf, > YARN-3854_Localization_support_for_Docker_image_v3.pdf > > > We need the ability to localize docker images when those images aren't > already available locally. There are various approaches that could be used > here with different trade-offs/issues : image archives on HDFS + docker load > , docker pull during the localization phase or (automatic) docker pull > during the run/launch phase. > We also need the ability to clean-up old/stale, unused images. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5567) Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456445#comment-15456445
]
Yufei Gu commented on YARN-5567:
I think we should push it 3.0.0-alpha1, otherwise there will be a
incompatibility in Hadoop 3.
> Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
> --
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5598) [YARN-3368] Fix create-release to be able to generate bits for the new yarn-ui
[ https://issues.apache.org/jira/browse/YARN-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456435#comment-15456435 ] Sunil G commented on YARN-5598: --- Patch looks good for me. I will commit the same if there are no objections. > [YARN-3368] Fix create-release to be able to generate bits for the new yarn-ui > -- > > Key: YARN-5598 > URL: https://issues.apache.org/jira/browse/YARN-5598 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn, yarn-ui-v2 >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5598-YARN-3368.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456428#comment-15456428
]
Daniel Templeton commented on YARN-5608:
Doesn't look like this patch is quite enough. I'm still seeing the failure.
After bumping up the number of retries to 20, I saw a bind exception instead.
I don't know if that was related or not. I'll keep playing with it.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3224) Notify AM with containers (on decommissioning node) could be preempted after timeout.
[
https://issues.apache.org/jira/browse/YARN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456403#comment-15456403
]
Hadoop QA commented on YARN-3224:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} YARN-3224 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12750692/0001-YARN-3224.patch |
| JIRA Issue | YARN-3224 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12992/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Notify AM with containers (on decommissioning node) could be preempted after
> timeout.
> -
>
> Key: YARN-3224
> URL: https://issues.apache.org/jira/browse/YARN-3224
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful
>Reporter: Junping Du
>Assignee: Sunil G
> Attachments: 0001-YARN-3224.patch, 0002-YARN-3224.patch
>
>
> We should leverage YARN preemption framework to notify AM that some
> containers will be preempted after a timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-5612: --- Attachment: YARN-5612-YARN-2915.v1.patch > Return SubClusterId in > FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover > -- > > Key: YARN-5612 > URL: https://issues.apache.org/jira/browse/YARN-5612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-5612-YARN-2915.v1.patch > > > This JIRA tracks the returning of SubClusterId from > FederationStateStoreFacade#addApplicationHomeSubCluster. > in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], > to handle better fail-over scenario the response needs SubClusterId. This is > bubbling up the change in YARN-5519. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3224) Notify AM with containers (on decommissioning node) could be preempted after timeout.
[ https://issues.apache.org/jira/browse/YARN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456396#comment-15456396 ] Hua Liu commented on YARN-3224: --- This feature is very helpful. AM can choose to stop sending new tasks to the to-be-preempted containers on decommissioning hosts. Will the author do a rebase and try to commit it? > Notify AM with containers (on decommissioning node) could be preempted after > timeout. > - > > Key: YARN-3224 > URL: https://issues.apache.org/jira/browse/YARN-3224 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Assignee: Sunil G > Attachments: 0001-YARN-3224.patch, 0002-YARN-3224.patch > > > We should leverage YARN preemption framework to notify AM that some > containers will be preempted after a timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5612: - Description: This JIRA tracks the returning of SubClusterId from FederationStateStoreFacade#addApplicationHomeSubCluster. in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], to handle better fail-over scenario the response needs SubClusterId. This is bubbling up the change in YARN-5519. was: This JIRA tracks the returning of SubClusterId from FederationStateStoreFacade#addApplicationHomeSubCluster. in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], to handle better fail-over scenario the response needs SubClusterId. > Return SubClusterId in > FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover > -- > > Key: YARN-5612 > URL: https://issues.apache.org/jira/browse/YARN-5612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > > This JIRA tracks the returning of SubClusterId from > FederationStateStoreFacade#addApplicationHomeSubCluster. > in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], > to handle better fail-over scenario the response needs SubClusterId. This is > bubbling up the change in YARN-5519. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-5612: --- Description: This JIRA tracks the returning of SubClusterId from FederationStateStoreFacade#addApplicationHomeSubCluster. in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], to handle better fail-over scenario the response needs SubClusterId. was: This JIRA tracks the returning of SubClusterId from . in the design of YARN-3659, to handle better fail-over scenario the response needs SubclusterId as field. > Return SubClusterId in > FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover > -- > > Key: YARN-5612 > URL: https://issues.apache.org/jira/browse/YARN-5612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > > This JIRA tracks the returning of SubClusterId from > FederationStateStoreFacade#addApplicationHomeSubCluster. > in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], > to handle better fail-over scenario the response needs SubClusterId. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-5612: --- Description: This JIRA tracks the returning of SubClusterId from . in the design of YARN-3659, to handle better fail-over scenario the response needs SubclusterId as field. was: This JIRA tracks the addition of SubClusterId into AddApplicationHomeSubClusterResponse. in the design of YARN-3659, to handle better fail-over scenario the response needs SubclusterId as field. > Return SubClusterId in > FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover > -- > > Key: YARN-5612 > URL: https://issues.apache.org/jira/browse/YARN-5612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > > This JIRA tracks the returning of SubClusterId from . > in the design of YARN-3659, to handle better fail-over scenario the response > needs SubclusterId as field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
[ https://issues.apache.org/jira/browse/YARN-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-5612: --- Description: This JIRA tracks the addition of SubClusterId into AddApplicationHomeSubClusterResponse. in the design of YARN-3659, to handle better fail-over scenario the response needs SubclusterId as field. > Return SubClusterId in > FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover > -- > > Key: YARN-5612 > URL: https://issues.apache.org/jira/browse/YARN-5612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > > This JIRA tracks the addition of SubClusterId into > AddApplicationHomeSubClusterResponse. > in the design of YARN-3659, to handle better fail-over scenario the response > needs SubclusterId as field. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456309#comment-15456309 ] Karthik Kambatla commented on YARN-5549: [~vinodkv], [~jlowe], [~Naganarasimha Garla] - are you okay with the approach in the last patch here? I would like to get this in soon. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5612) Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover
Giovanni Matteo Fumarola created YARN-5612: -- Summary: Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover Key: YARN-5612 URL: https://issues.apache.org/jira/browse/YARN-5612 Project: Hadoop YARN Issue Type: Sub-task Reporter: Giovanni Matteo Fumarola Assignee: Giovanni Matteo Fumarola -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5549) AMLauncher.createAMContainerLaunchContext() should not log the command to be launched indiscriminately
[ https://issues.apache.org/jira/browse/YARN-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456302#comment-15456302 ] Karthik Kambatla commented on YARN-5549: The patch looks good, except for the following nits: # The static imports make sense, but the rest of the code doesn't do it. Can we leave out. If we choose to keep it, when logging "REDACTED etc." # Some of the changes are unrelated to the patch. To minimize conflicts, I would like for us to leave these out. > AMLauncher.createAMContainerLaunchContext() should not log the command to be > launched indiscriminately > -- > > Key: YARN-5549 > URL: https://issues.apache.org/jira/browse/YARN-5549 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-5549.001.patch, YARN-5549.002.patch, > YARN-5549.003.patch, YARN-5549.004.patch, YARN-5549.005.patch > > > The command could contain sensitive information, such as keystore passwords > or AWS credentials or other. Instead of logging it as INFO, we should log it > as DEBUG and include a property to disable logging it at all. Logging it to > a different logger would also be viable and may create a smaller > administrative footprint. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5576) Core change to localize resource while container is running
[
https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456223#comment-15456223
]
Arun Suresh commented on YARN-5576:
---
Thanks for the patch [~jianhe]!
Couple of comments:
# We need to override the {{ContainerManagerImpl::localize()}} method in the
{{QueuingContainerManagerImpl}}. Re-localization should not be allowed if the
container is currently queued (not yet running)
# I only see entries added to {{ResourceSet::resourcesFailedToBeLocalized}}
set. Shouldnt we remove these once the AM is notified of the failure ? Also,
Shouldn't these be notified back to the AM ? or we are just relying on the
diagnostic string sent to the AM in the GetContainerStatus response to notify
the AM ?
# wondering if we should have another *RE_LOCALIZE_CONTAINER_RESOURCE* event in
the {{ResourceLocalizationService}} to distinguish from the localization needed
for container initialization and correspondingly send different events to the
Container. Or maybe for the timebeing, we should just rename
*INIT_CONTAINER_RESOURCE* to *LOCALIZE_CONTAINER_RESOURCE*.
# Nit : spurious change in the imports of {{ContainerImpl}} and
{{BaseAMRMProxyTest}}
> Core change to localize resource while container is running
> ---
>
> Key: YARN-5576
> URL: https://issues.apache.org/jira/browse/YARN-5576
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5576.1.patch, YARN-5576.2.patch, YARN-5576.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out
[
https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456100#comment-15456100
]
Hadoop QA commented on YARN-4205:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 14
new + 472 unchanged - 2 fixed = 486 total (was 474) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 10s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 16s
{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 7s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Possible null pointer dereference of application in
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppLifetimeMonitor$RMAppLifeTimeMonitorThread.run()
on exception path Dereferenced at RMAppLifetimeMonitor.java:application in
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppLifetimeMonitor$RMAppLifeTimeMonitorThread.run()
on exception path Dereferenced at
[jira] [Commented] (YARN-5610) Initial code for native services REST API
[
https://issues.apache.org/jira/browse/YARN-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456061#comment-15456061
]
Jian He commented on YARN-5610:
---
Some more comments:
- Application#get/setNumberOfContainers, this API is a bit inconsistent
-- when submit, it means the default number of containers for each component
-- when query app status, it means the total number of containers acroos all
components.
I feel it's better to make it consistent, for the number of running containers,
user can just infer it from the size of containers list or add a new filed
called containers_running
- Question on the quickLinks API, why does the component need to have
quicklinks field? IIUC, the quicklinks is mainly to show on the UI? In that
case, having it in the application object is enough?
- Can you upload the new Swagger sepcification ?
- Configuration object:
-- how are the properties filed passed to the container ?
- ConfigFile#dest_file (The absolute path that this configuration file should
be mounted as)
The user cannot assume an abritrary absolute path is valid in the remote
container ? It needs to be a relative path to the container local dir?
- ConfigFile#src_file: I think this could support all kinds of config files on
HDFS, so that user doesn't need to specify all configs in the request payload.
e.g. user simply provide a URI of the configFile on HDFS, YARN will localize
the file for container to use
- I dont see the place where queuName is set, also why is queueName set to
label_expression ?
{code}
if (queueName != null) {
resCompOptTriples.addAll(Arrays.asList(compName,
ResourceKeys.YARN_LABEL_EXPRESSION, queueName));
}
{code}
- Given that APP_NAME is staic and specified by the user, why do we need to
provide a place_holder for App_name and then substitue? why can't user specify
the app_name in the first place.
{code}
Map appQuicklinks = application.getQuicklinks();
Map placeholders = new HashMap<>();
placeholders.put(PLACEHOLDER_APP_NAME, application.getName());
if (appQuicklinks != null) {
for (Map.Entry quicklink : appQuicklinks.entrySet()) {
JsonObject export = new JsonObject();
export.addProperty("name", quicklink.getKey());
export.addProperty("value",
replacePlaceholders(quicklink.getValue(), placeholders));
exportsArray.add(export);
}
}
{code}
- All these null checks are not needed, because they are validated upfront
already
{code}
if (comp.getArtifact() != null && comp.getArtifact().getType() != null
&& comp.getArtifact().getType() == Artifact.TypeEnum.DOCKER) {
{code}
similarly,
{code}
comp.getArtifact().getId() == null ? application.getArtifact()
.getId() : comp.getArtifact().getId());
{code}
- invokeSliderClientRunnable: I don't quite understand why we need to set the
setContextClassLoader and then reset back, could you explain
- why is this code needed to be called in every rest API?
{code}
File sliderJarFile = SliderUtils
.findContainingJar(SliderClient.class);
if (sliderJarFile != null) {
logger.debug("slider.libdir={}", sliderJarFile.getParentFile()
.getAbsolutePath());
System.setProperty("slider.libdir", sliderJarFile.getParentFile()
.getAbsolutePath());
}
} catch (Throwable t) {
logger.warn("Unable to determine 'slider.libdir' path", t);
}
{code}
- remove unused destroySliderClient method
- deleteApplication API: listing all apps from RM is an expensive call. Maybe
we can directly call kill/stop application and handle the Exception and return
proper return-code.
{code}
// Check if application exists in any state
try {
int applicationsFound = getSliderList(appName, false);
if (applicationsFound < 0) {
return Response.status(Status.NOT_FOUND).build();
}
} catch (Exception e) {
logger.error("Delete application failed", e);
return Response.status(Status.NOT_FOUND).build();
}
try {
int livenessCheck = getSliderList(appName);
if (livenessCheck == 0) {
stopSliderApplication(appName);
while (getSliderList(appName) == 0) {
Thread.sleep(3000); // don't use thread sleep
}
}
{code}
- getApplication API: listing all apps is an expensive call in YARN. Instead We
can handle the ApplicationNotFoundExcetion from Yarn if the app does not exist.
{code}
// Check if app exists
try {
int livenessCheck = getSliderList(appName);
if (livenessCheck < 0) {
logger.info("Application not running");
ApplicationStatus applicationStatus = new ApplicationStatus();
applicationStatus.setErrorMessage(ERROR_APPLICATION_NOT_RUNNING);
[jira] [Commented] (YARN-5552) Add Builder methods for common yarn API records
[
https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456046#comment-15456046
]
Hadoop QA commented on YARN-5552:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 110
new + 210 unchanged - 8 fixed = 320 total (was 218) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 40s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 25s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 43s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 59s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| |
org.apache.hadoop.yarn.client.api.AMRMClient$ContainerRequest$ContainerRequestBuilder.nodes(String[])
may expose internal representation by storing an externally mutable object
into AMRMClient$ContainerRequest$ContainerRequestBuilder.nodes At
AMRMClient.java:by storing an externally mutable object into
AMRMClient$ContainerRequest$ContainerRequestBuilder.nodes At
AMRMClient.java:[line 394] |
| |
[jira] [Commented] (YARN-3854) Add localization support for docker images
[ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456021#comment-15456021 ] Varun Vasudev commented on YARN-3854: - [~tangzhankun] - the general approach looks ok to me. Do you want to go ahead and file tickets for the tasks and start the implementation? > Add localization support for docker images > -- > > Key: YARN-3854 > URL: https://issues.apache.org/jira/browse/YARN-3854 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-3854-branch-2.8.001.patch, > YARN-3854_Localization_support_for_Docker_image_v1.pdf, > YARN-3854_Localization_support_for_Docker_image_v2.pdf, > YARN-3854_Localization_support_for_Docker_image_v3.pdf > > > We need the ability to localize docker images when those images aren't > already available locally. There are various approaches that could be used > here with different trade-offs/issues : image archives on HDFS + docker load > , docker pull during the localization phase or (automatic) docker pull > during the run/launch phase. > We also need the ability to clean-up old/stale, unused images. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5610) Initial code for native services REST API
[
https://issues.apache.org/jira/browse/YARN-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456016#comment-15456016
]
Jian He commented on YARN-5610:
---
Copy over comments from parent jira.
*API Models*
- {artifact, resource, launch_command, number_of_containers} in Application
seems duplicated with those inside the component. I feel in this scenario, a
default global setting for artifacts, launch_command etc. is not that
appropriate, different components may likely have different requirements.
IMHO, we only need the ones in Component, this makes the interface cleaner and
underlying implementation simpler?
- unique_component_support: what is the primary use-case to have distinct
component name ?
- What is the BaseResource object for? Why does Application, ApplicationStatus,
Container, Resource need to extend this class?
- What does the Artifact#APPLICATION mean ?
- ApplicationState: What is difference RUNNNG vs STARTED, FINISHED vs STOPPED
{code}
ACCEPTED, RUNNING, FINISHED, FAILED, STOPPED, STARTED;
{code}
- Application#lifetime: it is String type. Does this mean we have to define a
scheme for user to specify the time in string format? How about just using long
type ?
- ApplicationStatus#errorMessage, how about call it diagnostics ? sometimes we
may also return non-error messages.
*Implementation*
- “hadoop-yarn-services-api” should be under hadoop-yarn-slider module as peer
to hadoop-yarn-slider-core
- why the changes needed in hadoop-project/pom.xml
- We should not use a deprecated getPort() method {{logger.info("Listening at
port = {}", applicationApiServer.getPort());}}, jenkins will report error.
- couple of things for below code
{code}
HADOOP_CONFIG = getHadoopConfigs();
SLIDER_CONFIG = getSliderClientConfiguration();
{code}
-- We cannot load hdfs config, that's for hdfs servers. Any reason you need the
hdfs configs?
-- Instead of calling these two methods, I think we can just call
{{YarnConfiguration yarnConf = new YarnConfiguration()}}. This will
automatically load the yarn-site and core-site configs.
- Why do we need to explicitly call initHadoopBinding, which is already called
the super.init() previously.
{code}
SliderClient client = new SliderClient() {
@Override
public void init(org.apache.hadoop.conf.Configuration conf) {
super.init(conf);
try {
initHadoopBinding();
} catch (SliderException e) {
throw new RuntimeException(
"Unable to automatically init Hadoop binding", e);
} catch (IOException e) {
throw new RuntimeException(
"Unable to automatically init Hadoop binding", e);
}
}
};
{code}
- These two catch clauses are identical, and Exception extends Throwable, so we
only need catch Throwable, if that's desired.
{code}
} catch (Exception e) {
logger.error("Unable to create SliderClient", e);
throw new RuntimeException(e.getMessage(), e);
} catch (Throwable e) {
logger.error("Unable to create SliderClient", e);
throw new RuntimeException(e.getMessage(), e);
}
{code}
- This will never return null, because the numberOfContainers is intialized as
1. you might want to check zero ?
{code}
// container size
if (application.getNumberOfContainers() == null) {
throw new IllegalArgumentException(ERROR_CONTAINERS_COUNT_INVALID);
}
{code}
- The lifetime field will never be null, because it is intilized as
"unlimited" by default
{code}
// Application lifetime if not specified, is set to unlimited lifetime
if (application.getLifetime() == null) {
application.setLifetime(DEFAULT_UNLIMITED_LIFETIME);
}
{code}
- IIUC, all these code are not needed, because appOptions is only used for
logging, uniqueGlobalPropertyCache is not used logically, Python is not
required any more in yarn-slider
{code}
if (application.getConfiguration() != null
&& application.getConfiguration().getProperties() != null) {
for (Map.Entry propEntry : application.getConfiguration()
.getProperties().entrySet()) {
if (PROPERTY_PYTHON_PATH.equals(propEntry.getKey())) {
addOptionsIfNotPresent(appOptions, uniqueGlobalPropertyCache,
SliderXmlConfKeys.PYTHON_EXECUTABLE_PATH, propEntry.getValue());
continue;
}
addOptionsIfNotPresent(appOptions, uniqueGlobalPropertyCache,
propEntry.getKey(), propEntry.getValue());
}
}
{code}
- In agent-less world, the status command is probably not required. We need a
different mechanism to determine container status. let's remove this for now
{code}
appConfOptTriples.addAll(Arrays.asList(compName, configPrefix.toLowerCase()
+ ".statusCommand", DEFAULT_STATUS_CMD));
{code}
- remove the unused parameter globalConf in createAppConfigComponent
- remove unused method
[jira] [Commented] (YARN-4793) [Umbrella] Simplified API layer for services and beyond
[ https://issues.apache.org/jira/browse/YARN-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456000#comment-15456000 ] Varun Vasudev commented on YARN-4793: - [~grey] - you raise some interesting points on scheduling. I see you've left a similar comment on YARN-3926. Let's continue the discussion there(since it seems to be the more relevant ticket). > [Umbrella] Simplified API layer for services and beyond > --- > > Key: YARN-4793 > URL: https://issues.apache.org/jira/browse/YARN-4793 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Gour Saha > Attachments: 20160603-YARN-Simplified-V1-API-Examples.adoc, > 20160603-YARN-Simplified-V1-API-Layer-For-Services.pdf, > 20160603-YARN-Simplified-V1-API-Layer-For-Services.yaml, > YARN-4793-yarn-native-services.001.patch > > > [See overview doc at YARN-4692, modifying and copy-pasting some of the > relevant pieces and sub-section 3.3.2 to track the specific sub-item.] > Bringing a new service on YARN today is not a simple experience. The APIs of > existing frameworks are either too low level (native YARN), require writing > new code (for frameworks with programmatic APIs ) or writing a complex spec > (for declarative frameworks). > In addition to building critical building blocks inside YARN (as part of > other efforts at YARN-4692), we should also look to simplifying the user > facing story for building services. Experience of projects like Slider > building real-life services like HBase, Storm, Accumulo, Solr etc gives us > some very good learnings on how simplified APIs for building services will > look like. > To this end, we should look at a new simple-services API layer backed by REST > interfaces. The REST layer can act as a single point of entry for creation > and lifecycle management of YARN services. Services here can range from > simple single-component apps to the most complex, multi-component > applications needing special orchestration needs. > We should also look at making this a unified REST based entry point for other > important features like resource-profile management (YARN-3926), > package-definitions' lifecycle-management and service-discovery (YARN-913 / > YARN-4757). We also need to flesh out its relation to our present much lower > level REST APIs (YARN-1695) in YARN for application-submission and > management. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455996#comment-15455996 ] Junping Du edited comment on YARN-5566 at 9/1/16 4:59 PM: -- >From above description, it seems the root cause is RM receive container status >after RMApp do App Finish Transition (will remove app from >runningApplications), then it add back the application to RMNode's >runningApplications but never remove it again. I am not 100% sure as RM log is >not included. [~rkanter], if you can check the timestamp for calling "runningApplications.add(containerAppId);" (in RMNodeImpl) and AppFinishedTransition (in RMAppImpl) for the same app when this issue happens, you should get the same answer. Current fix is a right one as we should always check application's status in context before we adding it to RMNode's runningApplication. +1. 004 patch LGTM. [~kasha], please feel free to commit it today or I will commit it tomorrow. BTW, patch for branch-2.8 should be slightly different. Robert, can you deliver one for 2.8 also? Thx! was (Author: djp): >From above description, it seems the root cause is RM receive container status >after RMApp do App Finish Transition (will app from runningApplications), then >it add back the application to RMNode's runningApplications but never remove >it again. I am not 100% sure as RM log is not included. [~rkanter], if you can check the timestamp for calling "runningApplications.add(containerAppId);" (in RMNodeImpl) and AppFinishedTransition (in RMAppImpl) for the same app when this issue happens, you should get the same answer. Current fix is a right one as we should always check application's status in context before we adding it to RMNode's runningApplication. +1. 004 patch LGTM. [~kasha], please feel free to commit it today or I will commit it tomorrow. BTW, patch for branch-2.8 should be slightly different. Robert, can you deliver one for 2.8 also? Thx! > client-side NM graceful decom doesn't trigger when jobs finish > -- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5566) client-side NM graceful decom doesn't trigger when jobs finish
[ https://issues.apache.org/jira/browse/YARN-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455996#comment-15455996 ] Junping Du commented on YARN-5566: -- >From above description, it seems the root cause is RM receive container status >after RMApp do App Finish Transition (will app from runningApplications), then >it add back the application to RMNode's runningApplications but never remove >it again. I am not 100% sure as RM log is not included. [~rkanter], if you can check the timestamp for calling "runningApplications.add(containerAppId);" (in RMNodeImpl) and AppFinishedTransition (in RMAppImpl) for the same app when this issue happens, you should get the same answer. Current fix is a right one as we should always check application's status in context before we adding it to RMNode's runningApplication. +1. 004 patch LGTM. [~kasha], please feel free to commit it today or I will commit it tomorrow. BTW, patch for branch-2.8 should be slightly different. Robert, can you deliver one for 2.8 also? Thx! > client-side NM graceful decom doesn't trigger when jobs finish > -- > > Key: YARN-5566 > URL: https://issues.apache.org/jira/browse/YARN-5566 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Attachments: YARN-5566.001.patch, YARN-5566.002.patch, > YARN-5566.003.patch, YARN-5566.004.patch > > > I was testing the client-side NM graceful decommission and noticed that it > was always waiting for the timeout, even if all jobs running on that node (or > even the cluster) had already finished. > For example: > # JobA is running with at least one container on NodeA > # User runs client-side decom on NodeA at 5:00am with a timeout of 3 hours > --> NodeA enters DECOMMISSIONING state > # JobA finishes at 6:00am and there are no other jobs running on NodeA > # User's client reaches the timeout at 8:00am, and forcibly decommissions > NodeA > NodeA should have decommissioned at 6:00am. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5611) Provide an API to update lifetime of an application.
Rohith Sharma K S created YARN-5611: --- Summary: Provide an API to update lifetime of an application. Key: YARN-5611 URL: https://issues.apache.org/jira/browse/YARN-5611 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S YARN-4205 monitors an Lifetime of an applications is monitored if required. Add an client api to update lifetime of an application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
[
https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455930#comment-15455930
]
Varun Vasudev commented on YARN-5366:
-
Thanks for the patch [~shaneku...@gmail.com]!
1)
{code}
+ // Validate the configured value
+ if (!keepContainer.equalsIgnoreCase("true") &&
+ !keepContainer.equalsIgnoreCase("false")) {
+throw new IllegalArgumentException("Only true and false are valid for"
++ " YARN_CONTAINER_RUNTIME_DOCKER_KEEP_CONTAINER_ON_EXIT");
+ }
{code}
We should validate the value on submission - not when we signal the container
2)
We should add a configuration option to let admins allow/disallow this behavior
- users shouldn’t be allowed to keep containers around because they feel like it
3)
{code}
+LOG.debug("Docker container is not being removed due to user request. "
++ "ContainerId: " + containerId);
{code}
I think for now this should be logged at info level. What do you think?
4)
{code}
+ String msg =
+ "Liveliness check failed for PID: " + ctx.getExecutionAttribute(PID)
+ + ". Container may have already completed.";
+ LOG.warn(msg);
{code}
I think this ends up double logging the message because the some one in the
caller chain also logs the same message?
5)
{code}
+ public enum StatusState {
{code}
Rename to either DockerContainerStatus or DockerContainerState
6)
{code}
+ if (currentContainerStatus == null) {
+return StatusState.UNKNOWN;
+ } else if (currentContainerStatus.equals(StatusState.RUNNING.getName()))
{
+return StatusState.RUNNING;
+ } else if (currentContainerStatus.equals(StatusState.STOPPED.getName()))
{
+return StatusState.STOPPED;
+ } else if (currentContainerStatus.equals(StatusState.EXITED.getName())) {
+return StatusState.EXITED;
+ } else {
+return StatusState.UNKNOWN;
+ }
{code}
Minor nit - but maybe change to switch/case?
7)
{code}
+// Allow for injecting the container's status for testing.
+if (statusState != null) {
+ status = statusState.getName();
+}
{code}
Remove this. What you can do in your testing use cases is to create a class
that inherits from DockerContainerStatusHandler and returns the status that
you’ve set, but the the code snippet above shouldn’t be in
DockerContainerStatusHandler
8)
Does MockContainerExecutorBinary need to be it’s own class - the pattern is
used in other places so we should either get everyone else to use this class or
move it into TestDockerContainerRuntime.
9)
{code}
+File f = new File("./src/test/resources/mock-container-executor");
{code}
The path of the file looks incorrect. It really should be in the target
directory. You should create a directory in the target directory and create the
file in that directory. It also looks like you create the mock executor but
don’t clean it up?
10)
Rename PrivilegedOperationCaptor to MockPrivilegedOperationCaptor? I would like
the name to reflect that it only works in the testing case.
> Add support for toggling the removal of completed and failed docker containers
> --
>
> Key: YARN-5366
> URL: https://issues.apache.org/jira/browse/YARN-5366
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Attachments: YARN-5366.001.patch, YARN-5366.002.patch
>
>
> Currently, completed and failed docker containers are removed by
> container-executor. Add a job level environment variable to
> DockerLinuxContainerRuntime to allow the user to toggle whether they want the
> container deleted or not and remove the logic from container-executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455929#comment-15455929 ] Rohith Sharma K S commented on YARN-4205: - [~sunilg] [~jianhe] [~leftnoteasy] requesting your review on rebased patch. kindly review > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4205.patch, YARN-4205_01.patch, > YARN-4205_02.patch, YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4205: Attachment: 0001-YARN-4205.patch Rebased the patch against trunk. And I have done few changes than earlier patches. # Kept consistently lifetime across all the usage. # Configuration name I modified with prefix yarn.resourcemanager since it is rm specific configuration. # Other minor changes is done here and there. > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-4205.patch, YARN-4205_01.patch, > YARN-4205_02.patch, YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-4205) Add a service for monitoring application life time out
[ https://issues.apache.org/jira/browse/YARN-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-4205: --- Assignee: Rohith Sharma K S (was: nijel) > Add a service for monitoring application life time out > -- > > Key: YARN-4205 > URL: https://issues.apache.org/jira/browse/YARN-4205 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: nijel >Assignee: Rohith Sharma K S > Attachments: YARN-4205_01.patch, YARN-4205_02.patch, > YARN-4205_03.patch > > > This JIRA intend to provide a lifetime monitor service. > The service will monitor the applications where the life time is configured. > If the application is running beyond the lifetime, it will be killed. > The lifetime will be considered from the submit time. > The thread monitoring interval is configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5610) Initial code for native services REST API
[ https://issues.apache.org/jira/browse/YARN-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-5610: Attachment: YARN-4793-yarn-native-services.001.patch Re-uploading the 001 patch to this sub-task > Initial code for native services REST API > - > > Key: YARN-5610 > URL: https://issues.apache.org/jira/browse/YARN-5610 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Attachments: YARN-4793-yarn-native-services.001.patch > > > This task will be used to submit and review patches for the initial code drop > for the native services REST API -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5610) Initial code for native services REST API
Gour Saha created YARN-5610: --- Summary: Initial code for native services REST API Key: YARN-5610 URL: https://issues.apache.org/jira/browse/YARN-5610 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha This task will be used to submit and review patches for the initial code drop for the native services REST API -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5552) Add Builder methods for common yarn API records
[ https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-5552: -- Attachment: YARN-5552.002.patch > Add Builder methods for common yarn API records > --- > > Key: YARN-5552 > URL: https://issues.apache.org/jira/browse/YARN-5552 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Tao Jie > Attachments: YARN-5552.000.patch, YARN-5552.001.patch, > YARN-5552.002.patch > > > Currently yarn API records such as ResourceRequest, AllocateRequest/Respone > as well as AMRMClient.ContainerRequest have multiple constructors / > newInstance methods. This makes it very difficult to add new fields to these > records. > It would probably be better if we had Builder classes for many of these > records, which would make evolution of these records a bit easier. > (suggested by [~kasha]) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5582) SchedulerUtils#validate vcores even for DefaultResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455725#comment-15455725
]
Hadoop QA commented on YARN-5582:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch
generated 0 new + 37 unchanged - 1 fixed = 37 total (was 38) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m 49s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 5s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826635/YARN-5582.0002.patch |
| JIRA Issue | YARN-5582 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 658c67609d96 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f55cc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12986/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn |
| Console output |
[jira] [Commented] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455666#comment-15455666
]
Daniel Templeton commented on YARN-5608:
The first couple of patches had a bug. There should be one more Jenkins run
pending with the last patch.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5576) Core change to localize resource while container is running
[ https://issues.apache.org/jira/browse/YARN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455654#comment-15455654 ] Varun Vasudev commented on YARN-5576: - +1 for the latest patch. I'll commit it tomorrow if no one objects. > Core change to localize resource while container is running > --- > > Key: YARN-5576 > URL: https://issues.apache.org/jira/browse/YARN-5576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5576.1.patch, YARN-5576.2.patch, YARN-5576.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455648#comment-15455648
]
Hadoop QA commented on YARN-5608:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 57s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826638/YARN-5608.003.patch |
| JIRA Issue | YARN-5608 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7ba1f835a14a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f55cc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12988/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12988/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12988/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12988/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL:
[jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455570#comment-15455570
]
Eric Payne commented on YARN-4945:
--
Thanks very much [~sunilg] and [~leftnoteasy].
{quote}
1.
I think we might need to come with a limit on how much resource can be
preempted from over-utilizing users's apps. WE do have
max-preemption-per-round. But sometimes it may be more as it may be configured
for inter-queue. Since we are sharing this config, i think we can have a config
to limit the preemption for user-limit. For priority, i have considered a
certain limit to control this scenario. Thoughts?
{quote}
I think we do need several intra-queue configs that are separate from the
existing (inter-queue) ones. For inter-queue vs. intra-queue, I think we need a
separate one at least for {{total_preemption_per_round}} and
{{max_ignored_over_capacity}}, and maybe even for
{{natural_termination_factor}} and {{max_wait_before_kill}}.
Are you also suggesting that we need these configs to also be spearate between
user-limit-percent preemption and priority preemption within intra queue? I
don't have a strong opinion either way, but if we can keep all configs the same
between intra-queue preemption policies, I would like to do that, just to avoid
confusion and complication.
bq. I will not consider preemption demand from a high priority if that app is
already crossing the user-limit.
I just want to make sure we are talking about the same thing. In the case I am
worried about, the high priority app is _*not*_ over any limit. There is an
inversion happening because the lower priority app has containers and the high
priority app wants them. But, if the low priority app is from a user that is at
or below its {{minimum-user-limit-percent}}, the higher priority app must not
continue to preempt from the lower priority app. This only can happen when the
two apps are from different users.
{quote}
I think normalization for inter-queue / intra-queue preemption is one of the
top priority goal for this feature.
If you take a look at existing preemption code, it normalizes preempt-able
resource for reserved-container-candidate-selector and fifo-candidate-selector.
We can do the similar normalization for inter/intra-queue preemption.
{quote}
Trying to do this coordination seems to me to be quite complicated. Would it be
sufficient to just avoid preempting during the intra-queue policies if there
are already containers in the {{selectedContainers}} list?
> [Umbrella] Capacity Scheduler Preemption Within a queue
> ---
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455580#comment-15455580
]
Hadoop QA commented on YARN-5608:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 38s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826634/YARN-5608.003.patch |
| JIRA Issue | YARN-5608 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 515420cb3ab1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f55cc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12987/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12987/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12987/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12987/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL:
[jira] [Updated] (YARN-5582) SchedulerUtils#validate vcores even for DefaultResourceCalculator
[
https://issues.apache.org/jira/browse/YARN-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-5582:
---
Attachment: YARN-5582.0002.patch
Updated patch using Resources
> SchedulerUtils#validate vcores even for DefaultResourceCalculator
> -
>
> Key: YARN-5582
> URL: https://issues.apache.org/jira/browse/YARN-5582
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: YARN-5582.0001.patch, YARN-5582.0002.patch
>
>
> Configure Memory=20 GB core 3 Vcores
> Submit request for 5 containers with memory 4 Gb and 5 core each from
> mapreduce application.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
> Invalid resource request, requested virtual cores < 0, or requested virtual
> cores > max configured, requestedVirtualCores=5, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:274)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:105)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:703)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:65)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:115)
> {noformat}
> Shouldnot validate core when resource calculator is
> {{org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5608:
---
Attachment: YARN-5608.003.patch
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455521#comment-15455521
]
Hadoop QA commented on YARN-5608:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 40s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826631/YARN-5608.002.patch |
| JIRA Issue | YARN-5608 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 85cb30e53abd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08f55cc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12985/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12985/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12985/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12985/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL:
[jira] [Updated] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5608:
---
Attachment: (was: YARN-5608.003.patch)
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5608.002.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5608) TestAMRMClient.setup() fails with ArrayOutOfBoundsException
[
https://issues.apache.org/jira/browse/YARN-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-5608:
---
Attachment: YARN-5608.003.patch
Here's a slight refactoring based on the {{sleep()}} method I just found
already in the code.
> TestAMRMClient.setup() fails with ArrayOutOfBoundsException
> ---
>
> Key: YARN-5608
> URL: https://issues.apache.org/jira/browse/YARN-5608
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Rohith Sharma K S
> Attachments: YARN-5608.002.patch, YARN-5608.003.patch, YARN-5608.patch
>
>
> After 39 runs the {{TestAMRMClient}} test, I encountered:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.setup(TestAMRMClient.java:144)
> {noformat}
> I see it shows up occasionally in the error emails as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
