[jira] [Commented] (YARN-4598) Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL
[
https://issues.apache.org/jira/browse/YARN-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105369#comment-15105369
]
tangshangwen commented on YARN-4598:
I think we should add a transition , have any Suggestions?
{noformat}
.addTransition(ContainerState.CONTAINER_CLEANEDUP_AFTER_KILL,
ContainerState.CONTAINER_CLEANEDUP_AFTER_KILL,
ContainerEventType.RESOURCE_FAILED)
{noformat}
> Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL
>
>
> Key: YARN-4598
> URL: https://issues.apache.org/jira/browse/YARN-4598
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
>
> In our cluster, I found that the container has some problems in state
> transition，this is my log
> {noformat}
> 2016-01-12 17:42:50,088 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1452588902899_0001_01_87 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2016-01-12 17:42:50,088 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Can't handle this event at current state: Current:
> [CONTAINER_CLEANEDUP_AFTER_KILL], eventType: [RESOURCE_FAILED]
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL
>
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
>
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1127)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:83)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1078)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1071)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
>
>
> at java.lang.Thread.run(Thread.java:744)
>
>
> 2016-01-12 17:42:50,089 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1452588902899_0001_01_94 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to null
> 2016-01-12 17:42:50,089 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_1452588902899_0001
> CONTAINERID=container_1452588902899_0001_01_94
>
> 2016-01-12 17:42:50,089 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1452588902899_0001_01_94 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-3215: Attachment: YARN-3215.v2.001.patch Hi [~wangda], Have done following modifications from the previous patch * Stored the labels for an app in appSchedulingInfo * Limited the headroom based on the available resources for a label * added test case for headroom calculation with labels * corrected other test cases where ever it was breaking for new api in csContext. > Respect labels in CapacityScheduler when computing headroom > --- > > Key: YARN-3215 > URL: https://issues.apache.org/jira/browse/YARN-3215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Naganarasimha G R > Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch > > > In existing CapacityScheduler, when computing headroom of an application, it > will only consider "non-labeled" nodes of this application. > But it is possible the application is asking for labeled resources, so > headroom-by-label (like 5G resource available under node-label=red) is > required to get better resource allocation and avoid deadlocks such as > MAPREDUCE-5928. > This JIRA could involve both API changes (such as adding a > label-to-available-resource map in AllocateResponse) and also internal > changes in CapacityScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4557) Few issues in scheduling with Node Labels
[ https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4557: Description: * When queue has * as accessibility, then the queue ordering was not happening properly. (was: * When app has submitted requests for multiple priority in default partition, then if one of the priority requests has missed non-partitioned-resource-request equivalent to cluster size then container needs to be allocated. Currently if the higher priority requests doesn't satisfy the condition, then whole application is getting skipped instead the priority * When queue has * as accessibility, then the queue ordering was not happening properly. ) > Few issues in scheduling with Node Labels > - > > Key: YARN-4557 > URL: https://issues.apache.org/jira/browse/YARN-4557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch, > YARN-4557.v2.002.patch > > > * When queue has * as accessibility, then the queue ordering was not > happening properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4557) Few issues in scheduling with Node Labels
[ https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4557: Description: * When queue has * as accessibility, then the queue ordering was not happening properly. Few Small nits * In AppSchedulingInfo comparator field doesn't have generics * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions has unused variable was:* When queue has * as accessibility, then the queue ordering was not happening properly. > Few issues in scheduling with Node Labels > - > > Key: YARN-4557 > URL: https://issues.apache.org/jira/browse/YARN-4557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch, > YARN-4557.v2.002.patch > > > * When queue has * as accessibility, then the queue ordering was not > happening properly. > Few Small nits > * In AppSchedulingInfo comparator field doesn't have generics > * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions > has unused variable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4557) Few issues in scheduling with Node Labels
[ https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4557: Description: * When queue has * as accessibility, then the queue ordering was not happening properly. Few Small nits * In AppSchedulingInfo comparator field doesn't have generics * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions has unused variable was: * When queue has * as accessibility, then the queue ordering was not happening properly. Few Small nits * In AppSchedulingInfo comparator field doesn't have generics * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions has unused variable > Few issues in scheduling with Node Labels > - > > Key: YARN-4557 > URL: https://issues.apache.org/jira/browse/YARN-4557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch, > YARN-4557.v2.002.patch > > > * When queue has * as accessibility, then the queue ordering was not > happening properly. > Few Small nits > * In AppSchedulingInfo comparator field doesn't have generics > * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions > has unused variable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4557) Improper Queues sorting in PartitionedQueueComparator when accessible node labels is configured as ANY
[ https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4557: Summary: Improper Queues sorting in PartitionedQueueComparator when accessible node labels is configured as ANY (was: Few issues in scheduling with Node Labels) > Improper Queues sorting in PartitionedQueueComparator when accessible node > labels is configured as ANY > -- > > Key: YARN-4557 > URL: https://issues.apache.org/jira/browse/YARN-4557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch, > YARN-4557.v2.002.patch > > > * When queue has * as accessibility, then the queue ordering was not > happening properly. > Few Small nits > * In AppSchedulingInfo comparator field doesn't have generics > * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions > has unused variable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4557) Improper Queues sorting in PartitionedQueueComparator when accessible node labels is configured as ANY
[ https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4557: Attachment: YARN-4557.v3.001.patch Hi [~wangda], Have removed modifications for bq. When app has submitted requests for multiple priority in default partition, then if one of the priority requests has missed non-partitioned-resource-request equivalent to cluster size then container needs to be allocated. Currently if the higher priority requests doesn't satisfy the condition, then whole application is getting skipped instead the priority Please check whether latest patch is fine > Improper Queues sorting in PartitionedQueueComparator when accessible node > labels is configured as ANY > -- > > Key: YARN-4557 > URL: https://issues.apache.org/jira/browse/YARN-4557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch, > YARN-4557.v2.002.patch, YARN-4557.v3.001.patch > > > * When queue has * as accessibility, then the queue ordering was not > happening properly. > Few Small nits > * In AppSchedulingInfo comparator field doesn't have generics > * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions > has unused variable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4307) Blacklisted nodes for AM container is not getting displayed in the Web UI
[ https://issues.apache.org/jira/browse/YARN-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105408#comment-15105408 ] Naganarasimha G R commented on YARN-4307: - Hi [~vvasudev], if you some cycles can you take a look at this patch ? > Blacklisted nodes for AM container is not getting displayed in the Web UI > - > > Key: YARN-4307 > URL: https://issues.apache.org/jira/browse/YARN-4307 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, webapp >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: AppInfoPage.png, RMappAttempt.png, > YARN-4307.v1.001.patch, YARN-4307.v1.002.patch, YARN-4307.v1.003.patch, > webpage.png, yarn-capacity-scheduler-debug.log > > > In pseudo cluster had 2 NM's and had launched app with incorrect > configuration *./hadoop org.apache.hadoop.mapreduce.SleepJob > -Dmapreduce.job.node-label-expression=labelX > -Dyarn.app.mapreduce.am.env=JAVA_HOME=/no/jvm/here -m 5 -mt 1200*. > First attempt failed and 2nd attempt was launched, but the application was > hung. In the scheduler logs found that localhost was blacklisted but in the > UI (app& apps listing page) count was shown as zero and as well no hosts > listed in the app page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105410#comment-15105410 ] Naganarasimha G R commented on YARN-4183: - Hi [~sjlee0], Any updates/reviews for this patch ? > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Naganarasimha G R > Attachments: YARN-4183.1.patch, YARN-4183.v1.001.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4598) Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL
[
https://issues.apache.org/jira/browse/YARN-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tangshangwen updated YARN-4598:
---
Attachment: YARN-4598.1.patch
I submitted a patch
> Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL
>
>
> Key: YARN-4598
> URL: https://issues.apache.org/jira/browse/YARN-4598
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: YARN-4598.1.patch
>
>
> In our cluster, I found that the container has some problems in state
> transition，this is my log
> {noformat}
> 2016-01-12 17:42:50,088 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1452588902899_0001_01_87 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> 2016-01-12 17:42:50,088 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Can't handle this event at current state: Current:
> [CONTAINER_CLEANEDUP_AFTER_KILL], eventType: [RESOURCE_FAILED]
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL
>
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
>
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1127)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:83)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1078)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1071)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:175)
>
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
>
>
> at java.lang.Thread.run(Thread.java:744)
>
>
> 2016-01-12 17:42:50,089 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1452588902899_0001_01_94 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to null
> 2016-01-12 17:42:50,089 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hadoop
> OPERATION=Container Finished - Killed TARGET=ContainerImpl
> RESULT=SUCCESS APPID=application_1452588902899_0001
> CONTAINERID=container_1452588902899_0001_01_94
>
> 2016-01-12 17:42:50,089 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1452588902899_0001_01_94 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4600) More general services provided to application/container by YARN
Junping Du created YARN-4600: Summary: More general services provided to application/container by YARN Key: YARN-4600 URL: https://issues.apache.org/jira/browse/YARN-4600 Project: Hadoop YARN Issue Type: New Feature Components: applications, resourcemanager Reporter: Junping Du Priority: Critical There are more general services like HA, message/notification, should be supported by YARN to containers to better support colorful applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4600) More general services provided to application/container by YARN
[ https://issues.apache.org/jira/browse/YARN-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du reassigned YARN-4600: Assignee: Junping Du > More general services provided to application/container by YARN > --- > > Key: YARN-4600 > URL: https://issues.apache.org/jira/browse/YARN-4600 > Project: Hadoop YARN > Issue Type: New Feature > Components: applications, resourcemanager >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > > There are more general services like HA, message/notification, should be > supported by YARN to containers to better support colorful applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4601) HA as a general YARN service to highlighted container by application.
Junping Du created YARN-4601: Summary: HA as a general YARN service to highlighted container by application. Key: YARN-4601 URL: https://issues.apache.org/jira/browse/YARN-4601 Project: Hadoop YARN Issue Type: Sub-task Components: applications Reporter: Junping Du Assignee: Junping Du Priority: Critical For LRS (long running services) on YARN, get rid of single point failure for critical container failure may not be necessary. Some applications would like to build its own HA architecture. However, it would be ideal to provide some fundamental support to HA service in YARN, like: launching container marked with active/standby, monitor/trigger out failed over, provide end point for shring information between active/standby container, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file
[
https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104910#comment-15104910
]
Hadoop QA commented on YARN-4577:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 55s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 1 new +
258 unchanged - 0 fixed = 259 total (was 258) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s {color}
| {color:red} hadoop-yarn-api in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 45s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 23s {color}
| {color:red} hadoop-yarn-api in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green}
[jira] [Commented] (YARN-4502) Fix two AM containers get allocated when AM restart
[
https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104925#comment-15104925
]
Karthik Kambatla commented on YARN-4502:
I tried backporting YARN-4524 to branch-2.8. Unfortunately, branch-2.8
compilation is broken.
[~leftnoteasy] - mind cherry-picking it to 2.8 once the compilation is fixed.
> Fix two AM containers get allocated when AM restart
> ---
>
> Key: YARN-4502
> URL: https://issues.apache.org/jira/browse/YARN-4502
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: YARN-4502-20160114.txt, YARN-4502-20160212.txt
>
>
> Scenario :
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
> yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-*.jar
> -attempt_failures_validity_interval 6 -shell_command "sleep 150"
> -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_02
> INFO impl.TimelineClientImpl: Timeline service address:
> http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/10.10.10.10:
> Total number of containers :2
> Container-Id Start Time Finish Time
> StateHost Node Http Address
>LOG-URL
> container_e12_1450825622869_0001_02_02 Tue Dec 22 23:07:35 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_02/hrt_qa
> container_e12_1450825622869_0001_02_01 Tue Dec 22 23:07:34 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_01/hrt_qa
> {code}
> * look for new AM pid
> Here, 2nd AM container was suppose to be started on
> container_e12_1450825622869_0001_02_01. But AM was not launched on
> container_e12_1450825622869_0001_02_01. It was in AQUIRED state.
> On other hand, container_e12_1450825622869_0001_02_02 got the AM running.
> Expected behavior: RM should not start 2 containers for starting AM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4584) RM startup failure when AM attempts greater than max-attempts
[
https://issues.apache.org/jira/browse/YARN-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4584:
---
Attachment: 0003-YARN-4584.patch
> RM startup failure when AM attempts greater than max-attempts
> -
>
> Key: YARN-4584
> URL: https://issues.apache.org/jira/browse/YARN-4584
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-YARN-4584.patch, 0002-YARN-4584.patch,
> 0003-YARN-4584.patch
>
>
> Configure 3 queue in cluster with 8 GB
> # queue 40%
> # queue 50%
> # default 10%
> * Submit applications to all 3 queue with container size as 1024MB (sleep job
> with 50 containers on all queues)
> * AM that gets assigned to default queue and gets preempted immediately after
> 20 preemption kill all application
> Due resource limit in default queue AM got prempted about 20 times
> On RM restart RM fails to restart
> {noformat}
> 2016-01-12 10:49:04,081 DEBUG org.apache.hadoop.service.AbstractService:
> noteFailure java.lang.NullPointerException
> 2016-01-12 10:49:04,081 INFO org.apache.hadoop.service.AbstractService:
> Service RMActiveServices failed in state STARTED; cause:
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.recover(RMAppAttemptImpl.java:887)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recover(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:953)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:946)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:786)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:328)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:464)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1232)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:594)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1022)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1062)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1058)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1705)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1058)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:877)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:467)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-12 10:49:04,082 DEBUG org.apache.hadoop.service.AbstractService:
> Service: RMActiveServices entered state STOPPED
> 2016-01-12 10:49:04,082 DEBUG org.apache.hadoop.service.CompositeService:
> RMActiveServices: stopping services, size=16
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4584) RM startup failure when AM attempts greater than max-attempts
[
https://issues.apache.org/jira/browse/YARN-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104930#comment-15104930
]
Bibin A Chundatt commented on YARN-4584:
[~rohithsharma]
# Attached patch after updating comments.
> RM startup failure when AM attempts greater than max-attempts
> -
>
> Key: YARN-4584
> URL: https://issues.apache.org/jira/browse/YARN-4584
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-YARN-4584.patch, 0002-YARN-4584.patch,
> 0003-YARN-4584.patch
>
>
> Configure 3 queue in cluster with 8 GB
> # queue 40%
> # queue 50%
> # default 10%
> * Submit applications to all 3 queue with container size as 1024MB (sleep job
> with 50 containers on all queues)
> * AM that gets assigned to default queue and gets preempted immediately after
> 20 preemption kill all application
> Due resource limit in default queue AM got prempted about 20 times
> On RM restart RM fails to restart
> {noformat}
> 2016-01-12 10:49:04,081 DEBUG org.apache.hadoop.service.AbstractService:
> noteFailure java.lang.NullPointerException
> 2016-01-12 10:49:04,081 INFO org.apache.hadoop.service.AbstractService:
> Service RMActiveServices failed in state STARTED; cause:
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.recover(RMAppAttemptImpl.java:887)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recover(RMAppImpl.java:826)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:953)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:946)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:786)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:328)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:464)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1232)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:594)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1022)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1062)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1058)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1705)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1058)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:323)
> at
> org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:127)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:877)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:467)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> 2016-01-12 10:49:04,082 DEBUG org.apache.hadoop.service.AbstractService:
> Service: RMActiveServices entered state STOPPED
> 2016-01-12 10:49:04,082 DEBUG org.apache.hadoop.service.CompositeService:
> RMActiveServices: stopping services, size=16
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4524) Cleanup AppSchedulingInfo
[ https://issues.apache.org/jira/browse/YARN-4524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4524: - Fix Version/s: (was: 2.9.0) 2.8.0 > Cleanup AppSchedulingInfo > - > > Key: YARN-4524 > URL: https://issues.apache.org/jira/browse/YARN-4524 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.8.0 > > Attachments: yarn-4524-1.patch, yarn-4524-2.patch > > > The AppSchedulingInfo class has become very hard to grok with some pretty > long methods. It needs some cleaning up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4502) Fix two AM containers get allocated when AM restart
[
https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104966#comment-15104966
]
Wangda Tan commented on YARN-4502:
--
[~kasha],
Thanks for reply,
I tried to compile branch-2.8, I didn't see any compilation issue, I just
pushed it to 2.8.
> Fix two AM containers get allocated when AM restart
> ---
>
> Key: YARN-4502
> URL: https://issues.apache.org/jira/browse/YARN-4502
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: YARN-4502-20160114.txt, YARN-4502-20160212.txt
>
>
> Scenario :
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
> yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-*.jar
> -attempt_failures_validity_interval 6 -shell_command "sleep 150"
> -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_02
> INFO impl.TimelineClientImpl: Timeline service address:
> http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/10.10.10.10:
> Total number of containers :2
> Container-Id Start Time Finish Time
> StateHost Node Http Address
>LOG-URL
> container_e12_1450825622869_0001_02_02 Tue Dec 22 23:07:35 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_02/hrt_qa
> container_e12_1450825622869_0001_02_01 Tue Dec 22 23:07:34 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_01/hrt_qa
> {code}
> * look for new AM pid
> Here, 2nd AM container was suppose to be started on
> container_e12_1450825622869_0001_02_01. But AM was not launched on
> container_e12_1450825622869_0001_02_01. It was in AQUIRED state.
> On other hand, container_e12_1450825622869_0001_02_02 got the AM running.
> Expected behavior: RM should not start 2 containers for starting AM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2885) Create AMRMProxy request interceptor for distributed scheduling decisions for queueable containers
[
https://issues.apache.org/jira/browse/YARN-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-2885:
--
Attachment: YARN-2885-yarn-2877.v6.patch
Thanks for the comments [~kishorch] and [~giovanni.fumarola].
I have updated the patch based on your suggestions:
# Updated the {{RequestInterceptor}} to implement the
{{DistributedSchedulingProtocol}} instead of the {{ApplicationMasterProtocol}}
: This essentially does not require any much change in the implementing
interceptors since 1) the former extends the latter and 2) the two extra
methods have a default implementation in the {{AbstractRequestInterceptor}}, so
if it not concerned with Distributed Scheduling, it will not have to deal with
it.
# Updated the {{DefaultRequestInterceptor}} to talk to the RM using the
{{DistributedSchedulingProtocol}} instead of the {{ApplicationMasterProtocol}}.
# Removed {{LocalScheduler}} from the end of the chain.. and put back the
DefaultRequestInterceptor
Do let me know what you think..
> Create AMRMProxy request interceptor for distributed scheduling decisions for
> queueable containers
> --
>
> Key: YARN-2885
> URL: https://issues.apache.org/jira/browse/YARN-2885
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Konstantinos Karanasos
>Assignee: Arun Suresh
> Attachments: YARN-2885-yarn-2877.001.patch,
> YARN-2885-yarn-2877.002.patch, YARN-2885-yarn-2877.full-2.patch,
> YARN-2885-yarn-2877.full-3.patch, YARN-2885-yarn-2877.full.patch,
> YARN-2885-yarn-2877.v4.patch, YARN-2885-yarn-2877.v5.patch,
> YARN-2885-yarn-2877.v6.patch, YARN-2885_api_changes.patch
>
>
> We propose to add a Local ResourceManager (LocalRM) to the NM in order to
> support distributed scheduling decisions.
> Architecturally we leverage the RMProxy, introduced in YARN-2884.
> The LocalRM makes distributed decisions for queuable containers requests.
> Guaranteed-start requests are still handled by the central RM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4526) Make SystemClock singleton so AppSchedulingInfo could use it
[ https://issues.apache.org/jira/browse/YARN-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105048#comment-15105048 ] Hudson commented on YARN-4526: -- FAILURE: Integrated in Hadoop-trunk-Commit #9131 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9131/]) YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha: rev d40859fab1ad977636457a6cc96b6a4f9b903afc) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/TestFairSchedulerQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/SystemClock.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestShuffleProvider.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttemptContainerRequest.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/ControlledClock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestQueueManager.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptFinishingMonitor.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSParentQueue.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java *
[jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105094#comment-15105094 ] Rohith Sharma K S commented on YARN-4265: - The same issue is there in branch-2 also. > Provide new timeline plugin storage to support fine-grained entity caching > -- > > Key: YARN-4265 > URL: https://issues.apache.org/jira/browse/YARN-4265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch, > YARN-4265-trunk.003.patch, YARN-4265-trunk.004.patch, > YARN-4265-trunk.005.patch, YARN-4265-trunk.006.patch, > YARN-4265-trunk.007.patch, YARN-4265-trunk.008.patch, > YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch > > > To support the newly proposed APIs in YARN-4234, we need to create a new > plugin timeline store. The store may have similar behavior as the > EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id > granularity, instead of application id granularity. Let's have this storage > as a standalone one, instead of updating EntityFileTimelineStore, to keep the > existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4597) Add SCHEDULE to NM container lifecycle
[ https://issues.apache.org/jira/browse/YARN-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104950#comment-15104950 ] Karthik Kambatla commented on YARN-4597: Makes a lot of sense. Thanks for filing this, Chris. > Add SCHEDULE to NM container lifecycle > -- > > Key: YARN-4597 > URL: https://issues.apache.org/jira/browse/YARN-4597 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Chris Douglas > > Currently, the NM immediately launches containers after resource > localization. Several features could be more cleanly implemented if the NM > included a separate stage for reserving resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104958#comment-15104958 ] Karthik Kambatla commented on YARN-4265: This seems to break branch-2.8 compilation (mvn clean install -DskipTests). Can someone look into it? > Provide new timeline plugin storage to support fine-grained entity caching > -- > > Key: YARN-4265 > URL: https://issues.apache.org/jira/browse/YARN-4265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch, > YARN-4265-trunk.003.patch, YARN-4265-trunk.004.patch, > YARN-4265-trunk.005.patch, YARN-4265-trunk.006.patch, > YARN-4265-trunk.007.patch, YARN-4265-trunk.008.patch, > YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch > > > To support the newly proposed APIs in YARN-4234, we need to create a new > plugin timeline store. The store may have similar behavior as the > EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id > granularity, instead of application id granularity. Let's have this storage > as a standalone one, instead of updating EntityFileTimelineStore, to keep the > existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4502) Fix two AM containers get allocated when AM restart
[ https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104986#comment-15104986 ] Hudson commented on YARN-4502: -- FAILURE: Integrated in Hadoop-trunk-Commit #9130 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9130/]) Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (wangda: rev adf260a728df427eb729abe8fb9ad7248991ea54) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/SchedulerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriority.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/ContainerRescheduledEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/PreemptableResourceScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMDispatcher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerPreemptEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/ContainerPreemptEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod (wangda: rev 3fe57285635e8058c34aa40a103845b49ca7d6ff) *
[jira] [Commented] (YARN-4524) Cleanup AppSchedulingInfo
[ https://issues.apache.org/jira/browse/YARN-4524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104985#comment-15104985 ] Hudson commented on YARN-4524: -- FAILURE: Integrated in Hadoop-trunk-Commit #9130 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9130/]) Move YARN-4524 from 2.9 to 2.8 (wangda: rev 01603be97bcd45fe40d576520aaa01d7bb5bbfac) * hadoop-yarn-project/CHANGES.txt > Cleanup AppSchedulingInfo > - > > Key: YARN-4524 > URL: https://issues.apache.org/jira/browse/YARN-4524 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.8.0 > > Attachments: yarn-4524-1.patch, yarn-4524-2.patch > > > The AppSchedulingInfo class has become very hard to grok with some pretty > long methods. It needs some cleaning up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4526) Make SystemClock singleton so AppSchedulingInfo could use it
[ https://issues.apache.org/jira/browse/YARN-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104956#comment-15104956 ] Karthik Kambatla commented on YARN-4526: Will check this in soon, based on Arun's +1 > Make SystemClock singleton so AppSchedulingInfo could use it > > > Key: YARN-4526 > URL: https://issues.apache.org/jira/browse/YARN-4526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-4526-1.patch, yarn-4526-2.patch, yarn-4526-2.patch > > > To track the time a request is received, we need to get current system time. > For better testability of this, we are likely better off using a Clock > instance that uses SystemClock by default. Instead of creating umpteen > instances of SystemClock, we should just reuse the same instance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4502) Fix two AM containers get allocated when AM restart
[
https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-4502:
-
Fix Version/s: (was: 2.9.0)
2.8.0
> Fix two AM containers get allocated when AM restart
> ---
>
> Key: YARN-4502
> URL: https://issues.apache.org/jira/browse/YARN-4502
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: YARN-4502-20160114.txt, YARN-4502-20160212.txt
>
>
> Scenario :
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
> yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-*.jar
> -attempt_failures_validity_interval 6 -shell_command "sleep 150"
> -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_02
> INFO impl.TimelineClientImpl: Timeline service address:
> http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/10.10.10.10:
> Total number of containers :2
> Container-Id Start Time Finish Time
> StateHost Node Http Address
>LOG-URL
> container_e12_1450825622869_0001_02_02 Tue Dec 22 23:07:35 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_02/hrt_qa
> container_e12_1450825622869_0001_02_01 Tue Dec 22 23:07:34 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_01/hrt_qa
> {code}
> * look for new AM pid
> Here, 2nd AM container was suppose to be started on
> container_e12_1450825622869_0001_02_01. But AM was not launched on
> container_e12_1450825622869_0001_02_01. It was in AQUIRED state.
> On other hand, container_e12_1450825622869_0001_02_02 got the AM running.
> Expected behavior: RM should not start 2 containers for starting AM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4599) Set OOM control for memory cgroups
Karthik Kambatla created YARN-4599: -- Summary: Set OOM control for memory cgroups Key: YARN-4599 URL: https://issues.apache.org/jira/browse/YARN-4599 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Karthik Kambatla Assignee: Karthik Kambatla YARN-1876 adds memory cgroups enforcing support. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. Today, one could set the swappiness to control this, but clusters with swap turned off exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105089#comment-15105089 ] Rohith Sharma K S commented on YARN-4265: - I found the issue, failure is because newly added project pom.xml version still 3.0.0-SNAPSHOT in branch-2.8. It should be 2.8.0-SNAPSHOT. So, dependency download will fail. After making change to 2.8.0-SNAPSHOT, it passes me locally. > Provide new timeline plugin storage to support fine-grained entity caching > -- > > Key: YARN-4265 > URL: https://issues.apache.org/jira/browse/YARN-4265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch, > YARN-4265-trunk.003.patch, YARN-4265-trunk.004.patch, > YARN-4265-trunk.005.patch, YARN-4265-trunk.006.patch, > YARN-4265-trunk.007.patch, YARN-4265-trunk.008.patch, > YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch > > > To support the newly proposed APIs in YARN-4234, we need to create a new > plugin timeline store. The store may have similar behavior as the > EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id > granularity, instead of application id granularity. Let's have this storage > as a standalone one, instead of updating EntityFileTimelineStore, to keep the > existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4526) Make SystemClock singleton so AppSchedulingInfo could use it
[ https://issues.apache.org/jira/browse/YARN-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104955#comment-15104955 ] Karthik Kambatla commented on YARN-4526: The test failures seem unrelated, and the RAT check fails on left over files from a compile/test run. > Make SystemClock singleton so AppSchedulingInfo could use it > > > Key: YARN-4526 > URL: https://issues.apache.org/jira/browse/YARN-4526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-4526-1.patch, yarn-4526-2.patch, yarn-4526-2.patch > > > To track the time a request is received, we need to get current system time. > For better testability of this, we are likely better off using a Clock > instance that uses SystemClock by default. Instead of creating umpteen > instances of SystemClock, we should just reuse the same instance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4599) Set OOM control for memory cgroups
[ https://issues.apache.org/jira/browse/YARN-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4599: --- Affects Version/s: 2.9.0 > Set OOM control for memory cgroups > -- > > Key: YARN-4599 > URL: https://issues.apache.org/jira/browse/YARN-4599 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-1856 adds memory cgroups enforcing support. We should also explicitly > set OOM control so that containers are not killed as soon as they go over > their usage. Today, one could set the swappiness to control this, but > clusters with swap turned off exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4599) Set OOM control for memory cgroups
[ https://issues.apache.org/jira/browse/YARN-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-4599: --- Description: YARN-1856 adds memory cgroups enforcing support. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. Today, one could set the swappiness to control this, but clusters with swap turned off exist. (was: YARN-1876 adds memory cgroups enforcing support. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. Today, one could set the swappiness to control this, but clusters with swap turned off exist.) > Set OOM control for memory cgroups > -- > > Key: YARN-4599 > URL: https://issues.apache.org/jira/browse/YARN-4599 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-1856 adds memory cgroups enforcing support. We should also explicitly > set OOM control so that containers are not killed as soon as they go over > their usage. Today, one could set the swappiness to control this, but > clusters with swap turned off exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
[
https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105542#comment-15105542
]
Hadoop QA commented on YARN-3215:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 3 new + 212 unchanged - 5 fixed = 215 total (was 217) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 22s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 29s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 46s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Unused field:CapacityHeadroomProvider.java |
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| |
[jira] [Commented] (YARN-4577) Enable aux services to have their own custom classpath/jar file
[ https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105561#comment-15105561 ] Xuan Gong commented on YARN-4577: - [~sjlee0] bq. how important is it to support non-local classpaths We can do it separately. Right now, for this ticket, we are focusing on supporting the local classpath. bq. other types of classloading Do we have any examples for this ? I can only find the MRApp and RunJar are using ApplicationClassLoader. > Enable aux services to have their own custom classpath/jar file > --- > > Key: YARN-4577 > URL: https://issues.apache.org/jira/browse/YARN-4577 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4577.1.patch, YARN-4577.2.patch, YARN-4577.3.patch, > YARN-4577.3.rebase.patch, YARN-4577.4.patch > > > Right now, users have to add their jars to the NM classpath directly, thus > put them on the system classloader. But if multiple versions of the plugin > are present on the classpath, there is no control over which version actually > gets loaded. Or if there are any conflicts between the dependencies > introduced by the auxiliary service and the NM itself, they can break the NM, > the auxiliary service, or both. > The solution could be: to instantiate aux services using a classloader that > is different from the system classloader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4557) Improper Queues sorting in PartitionedQueueComparator when accessible node labels is configured as ANY
[
https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105578#comment-15105578
]
Hadoop QA commented on YARN-4557:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 52s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 33s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782888/YARN-4557.v3.001.patch
|
| JIRA Issue | YARN-4557 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle
[jira] [Commented] (YARN-4584) RM startup failure when AM attempts greater than max-attempts
[
https://issues.apache.org/jira/browse/YARN-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105116#comment-15105116
]
Hadoop QA commented on YARN-4584:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 27s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 35s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782827/0003-YARN-4584.patch |
| JIRA Issue | YARN-4584 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105166#comment-15105166 ] Rohith Sharma K S commented on YARN-4265: - Committed addendum patch to branch-2 and branch-2.8 respectively to unblock compilation issue asap.. > Provide new timeline plugin storage to support fine-grained entity caching > -- > > Key: YARN-4265 > URL: https://issues.apache.org/jira/browse/YARN-4265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YANR-4265-branch-2.8-addendum.patch, > YARN-4265-branch-2-addendum.patch, YARN-4265-trunk.001.patch, > YARN-4265-trunk.002.patch, YARN-4265-trunk.003.patch, > YARN-4265-trunk.004.patch, YARN-4265-trunk.005.patch, > YARN-4265-trunk.006.patch, YARN-4265-trunk.007.patch, > YARN-4265-trunk.008.patch, YARN-4265.YARN-4234.001.patch, > YARN-4265.YARN-4234.002.patch > > > To support the newly proposed APIs in YARN-4234, we need to create a new > plugin timeline store. The store may have similar behavior as the > EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id > granularity, instead of application id granularity. Let's have this storage > as a standalone one, instead of updating EntityFileTimelineStore, to keep the > existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4265: Attachment: YANR-4265-branch-2.8-addendum.patch YARN-4265-branch-2-addendum.patch Updating the addendum patch fixing compilation issue in branch-2/branch-2.8 > Provide new timeline plugin storage to support fine-grained entity caching > -- > > Key: YARN-4265 > URL: https://issues.apache.org/jira/browse/YARN-4265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YANR-4265-branch-2.8-addendum.patch, > YARN-4265-branch-2-addendum.patch, YARN-4265-trunk.001.patch, > YARN-4265-trunk.002.patch, YARN-4265-trunk.003.patch, > YARN-4265-trunk.004.patch, YARN-4265-trunk.005.patch, > YARN-4265-trunk.006.patch, YARN-4265-trunk.007.patch, > YARN-4265-trunk.008.patch, YARN-4265.YARN-4234.001.patch, > YARN-4265.YARN-4234.002.patch > > > To support the newly proposed APIs in YARN-4234, we need to create a new > plugin timeline store. The store may have similar behavior as the > EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id > granularity, instead of application id granularity. Let's have this storage > as a standalone one, instead of updating EntityFileTimelineStore, to keep the > existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4502) Fix two AM containers get allocated when AM restart
[
https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105191#comment-15105191
]
Rohith Sharma K S commented on YARN-4502:
-
branch-2/branch-2.8 was broken by YARN-4265, uploaded addendum patch and
committed it.
Most probably, guessing in your case that compilation is done for trunk first
where in 3.0.0.SNAPSHOT jar has got published to local repository. And later
trying to compile in branch-2/branch-2.8 fetches jars from local repository.
> Fix two AM containers get allocated when AM restart
> ---
>
> Key: YARN-4502
> URL: https://issues.apache.org/jira/browse/YARN-4502
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: YARN-4502-20160114.txt, YARN-4502-20160212.txt
>
>
> Scenario :
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
> yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-*.jar
> -attempt_failures_validity_interval 6 -shell_command "sleep 150"
> -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_02
> INFO impl.TimelineClientImpl: Timeline service address:
> http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/10.10.10.10:
> Total number of containers :2
> Container-Id Start Time Finish Time
> StateHost Node Http Address
>LOG-URL
> container_e12_1450825622869_0001_02_02 Tue Dec 22 23:07:35 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_02/hrt_qa
> container_e12_1450825622869_0001_02_01 Tue Dec 22 23:07:34 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_01/hrt_qa
> {code}
> * look for new AM pid
> Here, 2nd AM container was suppose to be started on
> container_e12_1450825622869_0001_02_01. But AM was not launched on
> container_e12_1450825622869_0001_02_01. It was in AQUIRED state.
> On other hand, container_e12_1450825622869_0001_02_02 got the AM running.
> Expected behavior: RM should not start 2 containers for starting AM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching
[ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105238#comment-15105238 ] Junping Du commented on YARN-4265: -- It is weird that my build on branch-2/branch-2.8 locally get passed before check in... may due to local cache from build against trunk before？ Anyway, thanks [~ka...@cloudera.com] and [~rohithsharma] to report the issue and fix it. > Provide new timeline plugin storage to support fine-grained entity caching > -- > > Key: YARN-4265 > URL: https://issues.apache.org/jira/browse/YARN-4265 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YANR-4265-branch-2.8-addendum.patch, > YARN-4265-branch-2-addendum.patch, YARN-4265-trunk.001.patch, > YARN-4265-trunk.002.patch, YARN-4265-trunk.003.patch, > YARN-4265-trunk.004.patch, YARN-4265-trunk.005.patch, > YARN-4265-trunk.006.patch, YARN-4265-trunk.007.patch, > YARN-4265-trunk.008.patch, YARN-4265.YARN-4234.001.patch, > YARN-4265.YARN-4234.002.patch > > > To support the newly proposed APIs in YARN-4234, we need to create a new > plugin timeline store. The store may have similar behavior as the > EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id > granularity, instead of application id granularity. Let's have this storage > as a standalone one, instead of updating EntityFileTimelineStore, to keep the > existing store (EntityFileTimelineStore) stable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4465) SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled
[
https://issues.apache.org/jira/browse/YARN-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4465:
---
Attachment: 0001-YARN-4465.patch
Hi [~sunilg]/[~leftnoteasy]
Attaching patch for review. Patch updates the below
# Typo correction for message
# If cluster node labels are not available throw InvalidLabelresourceexception
# Reset request label if Node Labels are not enabled
> SchedulerUtils#validateRequest for Label check should happen only when
> nodelabel enabled
>
>
> Key: YARN-4465
> URL: https://issues.apache.org/jira/browse/YARN-4465
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4465.patch
>
>
> Disable label from rm side yarn.nodelabel.enable=false
> Capacity scheduler label configuration for queue is available as below
> default label for queue = b1 as 3 and accessible labels as 1,3
> Submit application to queue A .
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
> Invalid resource request, queue=b1 doesn't have permission to access all
> labels in resource request. labelExpression of resource request=3. Queue
> labels=1,3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:216)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:401)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:340)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:602)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:247)
> {noformat}
> # Ignore default label expression when label is disabled *or*
> # NormalizeResourceRequest we can set label expression to
> when node label is not enabled *or*
> # Improve message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4600) More general services provided to application/container by YARN
[ https://issues.apache.org/jira/browse/YARN-4600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated YARN-4600: - Assignee: (was: Junping Du) > More general services provided to application/container by YARN > --- > > Key: YARN-4600 > URL: https://issues.apache.org/jira/browse/YARN-4600 > Project: Hadoop YARN > Issue Type: New Feature > Components: applications, resourcemanager >Reporter: Junping Du >Priority: Critical > > There are more general services like HA, message/notification, should be > supported by YARN to containers to better support colorful applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4602) Message/notification service between containers
Junping Du created YARN-4602: Summary: Message/notification service between containers Key: YARN-4602 URL: https://issues.apache.org/jira/browse/YARN-4602 Project: Hadoop YARN Issue Type: Sub-task Reporter: Junping Du Assignee: Junping Du Currently, mostly communications among YARN daemons, services and applications are go through RPC. In almost all cases, logic running inside of containers are RPC client but not server because it get launched inflight. The only special case is AM container, because it get launched earlier than any other containers so it can be RPC server and tell new coming containers server address in application logic (like MR AM). The side effects are: 1. When AM container get failed, the new AM attempts will get launched with new address/port, so previous RPC are broken. 2. Application's requirement are variable, there could be other dependency between containers (not AM), so some container failed over will affect other containers' running logic. It is better to have some message/notification mechanism between containers for handle above cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4545) Allow YARN distributed shell to use ATS v1.5 APIs
[ https://issues.apache.org/jira/browse/YARN-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105508#comment-15105508 ] Junping Du commented on YARN-4545: -- Thanks for uploading a patch, [~gtCarrera9]. Given YARN-4265 already commit in, can you rebase the patch against trunk? Thanks! > Allow YARN distributed shell to use ATS v1.5 APIs > - > > Key: YARN-4545 > URL: https://issues.apache.org/jira/browse/YARN-4545 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4545-YARN-4265.001.patch > > > We can use YARN distributed shell as a demo for the ATS v1.5 APIs. We need to > allow distributed shell post data with ATS v1.5 API if 1.5 is enabled in the > system. We also need to provide a sample plugin to read those data out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4465) SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled
[
https://issues.apache.org/jira/browse/YARN-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105680#comment-15105680
]
Hadoop QA commented on YARN-4465:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 0 new + 75 unchanged - 2 fixed = 75 total (was 77) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 40s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91
with JDK v1.7.0_91 generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 4s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 37s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
[jira] [Commented] (YARN-4601) HA as a general YARN service to highlighted container by application.
[ https://issues.apache.org/jira/browse/YARN-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105698#comment-15105698 ] Allen Wittenauer commented on YARN-4601: Critical priority? That's a bit surprising... > HA as a general YARN service to highlighted container by application. > - > > Key: YARN-4601 > URL: https://issues.apache.org/jira/browse/YARN-4601 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > > For LRS (long running services) on YARN, get rid of single point failure for > critical container failure may not be necessary. Some applications would like > to build its own HA architecture. However, it would be ideal to provide some > fundamental support to HA service in YARN, like: launching container marked > with active/standby, monitor/trigger out failed over, provide end point for > shring information between active/standby container, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4599) Set OOM control for memory cgroups
[ https://issues.apache.org/jira/browse/YARN-4599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105694#comment-15105694 ] Allen Wittenauer commented on YARN-4599: bq. We should also explicitly set OOM control so that containers are not killed as soon as they go over their usage. We need this as something that can be set and I'd propose that the default be off given that administrators are going to be very confused when they see container usage go above the limit. FWIW, we have jobs running on our clusters that the NM's memory control doesn't stop from breaking nodes because it is too slow. > Set OOM control for memory cgroups > -- > > Key: YARN-4599 > URL: https://issues.apache.org/jira/browse/YARN-4599 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > YARN-1856 adds memory cgroups enforcing support. We should also explicitly > set OOM control so that containers are not killed as soon as they go over > their usage. Today, one could set the swappiness to control this, but > clusters with swap turned off exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4219) New levelDB cache storage for timeline v1.5
[ https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-4219: Attachment: YARN-4219-trunk.004.patch Rebase the patch to latest trunk. > New levelDB cache storage for timeline v1.5 > --- > > Key: YARN-4219 > URL: https://issues.apache.org/jira/browse/YARN-4219 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4219-YARN-4265.001.patch, > YARN-4219-YARN-4265.002.patch, YARN-4219-YARN-4265.003.patch, > YARN-4219-trunk.001.patch, YARN-4219-trunk.002.patch, > YARN-4219-trunk.003.patch, YARN-4219-trunk.004.patch > > > We need to have an "offline" caching storage for timeline server v1.5 after > the changes in YARN-3942. The in memory timeline storage may run into OOM > issues when used as a cache storage for entity file timeline storage. We can > refactor the code and have a level db based caching storage for this use > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (YARN-4502) Fix two AM containers get allocated when AM restart
[
https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reopened YARN-4502:
---
[~leftnoteasy], looks like you committed the wrong patch, likely because of the
wrong names I gave to the patch. You should look at YARN-4502-20160114.txt as
the final patch.
I'll let you fix this as there seem to be other things you did on other
branches, reopening it for now.
> Fix two AM containers get allocated when AM restart
> ---
>
> Key: YARN-4502
> URL: https://issues.apache.org/jira/browse/YARN-4502
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: YARN-4502-20160114.txt, YARN-4502-20160212.txt
>
>
> Scenario :
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
> yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-*.jar
> -attempt_failures_validity_interval 6 -shell_command "sleep 150"
> -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_02
> INFO impl.TimelineClientImpl: Timeline service address:
> http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/10.10.10.10:
> Total number of containers :2
> Container-Id Start Time Finish Time
> StateHost Node Http Address
>LOG-URL
> container_e12_1450825622869_0001_02_02 Tue Dec 22 23:07:35 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_02/hrt_qa
> container_e12_1450825622869_0001_02_01 Tue Dec 22 23:07:34 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_01/hrt_qa
> {code}
> * look for new AM pid
> Here, 2nd AM container was suppose to be started on
> container_e12_1450825622869_0001_02_01. But AM was not launched on
> container_e12_1450825622869_0001_02_01. It was in AQUIRED state.
> On other hand, container_e12_1450825622869_0001_02_02 got the AM running.
> Expected behavior: RM should not start 2 containers for starting AM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-4502) Fix two AM containers get allocated when AM restart
[
https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan resolved YARN-4502.
--
Resolution: Fixed
Thanks [~vinodkv], just reverted & committed it to branch-2/2.8/trunk.
> Fix two AM containers get allocated when AM restart
> ---
>
> Key: YARN-4502
> URL: https://issues.apache.org/jira/browse/YARN-4502
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: YARN-4502-20160114.txt, YARN-4502-20160212.txt
>
>
> Scenario :
> * set yarn.resourcemanager.am.max-attempts = 2
> * start dshell application
> {code}
> yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> hadoop-yarn-applications-distributedshell-*.jar
> -attempt_failures_validity_interval 6 -shell_command "sleep 150"
> -num_containers 16
> {code}
> * Kill AM pid
> * Print container list for 2nd attempt
> {code}
> yarn container -list appattempt_1450825622869_0001_02
> INFO impl.TimelineClientImpl: Timeline service address:
> http://xxx:port/ws/v1/timeline/
> INFO client.RMProxy: Connecting to ResourceManager at xxx/10.10.10.10:
> Total number of containers :2
> Container-Id Start Time Finish Time
> StateHost Node Http Address
>LOG-URL
> container_e12_1450825622869_0001_02_02 Tue Dec 22 23:07:35 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_02/hrt_qa
> container_e12_1450825622869_0001_02_01 Tue Dec 22 23:07:34 + 2015
> N/A RUNNINGxxx:25454 http://xxx:8042
> http://xxx:8042/node/containerlogs/container_e12_1450825622869_0001_02_01/hrt_qa
> {code}
> * look for new AM pid
> Here, 2nd AM container was suppose to be started on
> container_e12_1450825622869_0001_02_01. But AM was not launched on
> container_e12_1450825622869_0001_02_01. It was in AQUIRED state.
> On other hand, container_e12_1450825622869_0001_02_02 got the AM running.
> Expected behavior: RM should not start 2 containers for starting AM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4596) SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities
[ https://issues.apache.org/jira/browse/YARN-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106055#comment-15106055 ] Hudson commented on YARN-4596: -- FAILURE: Integrated in Hadoop-trunk-Commit #9133 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9133/]) YARN-4596. SystemMetricPublisher should not swallow error messages from (jianhe: rev f385851141522633184ce394899c659af5ace92a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * hadoop-yarn-project/CHANGES.txt > SystemMetricPublisher should not swallow error messages from > TimelineClient#putEntities > --- > > Key: YARN-4596 > URL: https://issues.apache.org/jira/browse/YARN-4596 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YARN-4596-trunk.001.patch, YARN-4596-trunk.002.patch > > > We should report error messages from the returned TimelineResponse when > posting timeline entities through system metric publisher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4597) Add SCHEDULE to NM container lifecycle
[ https://issues.apache.org/jira/browse/YARN-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reassigned YARN-4597: - Assignee: Arun Suresh > Add SCHEDULE to NM container lifecycle > -- > > Key: YARN-4597 > URL: https://issues.apache.org/jira/browse/YARN-4597 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Chris Douglas >Assignee: Arun Suresh > > Currently, the NM immediately launches containers after resource > localization. Several features could be more cleanly implemented if the NM > included a separate stage for reserving resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4502) Fix two AM containers get allocated when AM restart
[ https://issues.apache.org/jira/browse/YARN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106100#comment-15106100 ] Hudson commented on YARN-4502: -- FAILURE: Integrated in Hadoop-trunk-Commit #9134 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9134/]) Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (wangda: rev 150f5ae0343e872ee8bef39c57008c1389f0ba9e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriority.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerPreemptEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/SchedulerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMDispatcher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/ContainerRescheduledEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/ContainerPreemptEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/PreemptableResourceScheduler.java YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod (wangda: rev a44ce3f14fd940601f984fbf7980aa6fdc8f23b7) *
[jira] [Updated] (YARN-4603) FairScheduler should mention user requested queuename in error message when failed in queue ACL check
[
https://issues.apache.org/jira/browse/YARN-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Jie updated YARN-4603:
--
Attachment: YARN-4603.001.patch
> FairScheduler should mention user requested queuename in error message when
> failed in queue ACL check
> -
>
> Key: YARN-4603
> URL: https://issues.apache.org/jira/browse/YARN-4603
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.7.1
>Reporter: Tao Jie
>Assignee: Tao Jie
>Priority: Trivial
> Attachments: YARN-4603.001.patch
>
>
> When submit a job to a queue does not exist (maybe misspelling), scheduler
> will try to submit it to another queue according to QueuePlacementPolicy. For
> instance, when I submit a job to queue root.abc, return error message is like:
> {quote}
> java.io.IOException: Failed to run job : User user1 cannot submit
> applications to queue root.default
> {quote}
> We'd better add original user requested queue name to this error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4557) Improper Queues sorting in PartitionedQueueComparator when accessible node labels is configured as ANY
[
https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106186#comment-15106186
]
Wangda Tan commented on YARN-4557:
--
Looks good, +1. Thanks [~Naganarasimha],
One nit is:
{code}
1671// Test case 1
1672// Both a/b has used_capacity(x) = 0, when doing exclusive
allocation, b
1673// will go first since b has more capacity(x)
{code}
It should be "a should go first" according to your test case.
> Improper Queues sorting in PartitionedQueueComparator when accessible node
> labels is configured as ANY
> --
>
> Key: YARN-4557
> URL: https://issues.apache.org/jira/browse/YARN-4557
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Minor
> Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch,
> YARN-4557.v2.002.patch, YARN-4557.v3.001.patch
>
>
> * When queue has * as accessibility, then the queue ordering was not
> happening properly.
> Few Small nits
> * In AppSchedulingInfo comparator field doesn't have generics
> * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions
> has unused variable
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4596) SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities
[ https://issues.apache.org/jira/browse/YARN-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106248#comment-15106248 ] Naganarasimha G R commented on YARN-4596: - Thanks [~gtCarrera] for considering them ! > SystemMetricPublisher should not swallow error messages from > TimelineClient#putEntities > --- > > Key: YARN-4596 > URL: https://issues.apache.org/jira/browse/YARN-4596 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YARN-4596-trunk.001.patch, YARN-4596-trunk.002.patch > > > We should report error messages from the returned TimelineResponse when > posting timeline entities through system metric publisher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4565) When sizeBasedWeight enabled for FairOrderingPolicy in CapacityScheduler, Sometimes lead to situation where all queue resources consumed by AMs only
[ https://issues.apache.org/jira/browse/YARN-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106273#comment-15106273 ] Hudson commented on YARN-4565: -- FAILURE: Integrated in Hadoop-trunk-Commit #9136 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9136/]) YARN-4565. Fix a bug that leads to AM resource limit not hornored when (jianhe: rev edc43a9097530fd469dee47d4fefd091818331e5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/CHANGES.txt > When sizeBasedWeight enabled for FairOrderingPolicy in CapacityScheduler, > Sometimes lead to situation where all queue resources consumed by AMs only > > > Key: YARN-4565 > URL: https://issues.apache.org/jira/browse/YARN-4565 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, capacityscheduler >Affects Versions: 2.8.0 >Reporter: Karam Singh >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4565.1.patch, YARN-4565.2.patch, YARN-4565.3.patch > > > When sizeBasedWeight enabled for FairOrderingPolicy in CapacityScheduler, > Sometimes lead to situation where all queue resources consumed by AMs only, > So from users perpective it appears that all application in queue are stuck, > whole queue capacity is comsumed by AMs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4465) SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled
[
https://issues.apache.org/jira/browse/YARN-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106204#comment-15106204
]
Wangda Tan commented on YARN-4465:
--
Hi [~bibinchundatt]
Thanks for working on this, some comments:
1)
{code}
public static void normalizeAndValidateRequests(List ask,
Resource maximumResource, String queueName, YarnScheduler scheduler,
RMContext rmContext)
{code}
You can avoid get queue info if node label is disabled. Same to
{code}
public static void normalizeAndValidateRequest(ResourceRequest resReq,
Resource maximumResource, String queueName, YarnScheduler scheduler,
boolean isRecovery, RMContext rmContext, QueueInfo queueInfo)
{code}
2) In
{code}
private static void normalizeNodeLabelExpressionInRequest(
ResourceRequest resReq, QueueInfo queueInfo, RMContext rmContext) {
{code}
Instead of passing rmContext as parameter, you can pass a boolean to indicate
if node label is enabled.
3)
{{LOG.debug("Resetti}} should be wrapped by "isDebugEnabled"
4) In
{code}
private static void validateResourceRequest(ResourceRequest resReq,
Resource maximumResource, QueueInfo queueInfo, RMContext rmContext)
{code}
You can use a "if (nodeLabelEnabled) \{...\}" to wrap code after
{code}
String labelExp = resReq.getNodeLabelExpression();
...
{code}
> SchedulerUtils#validateRequest for Label check should happen only when
> nodelabel enabled
>
>
> Key: YARN-4465
> URL: https://issues.apache.org/jira/browse/YARN-4465
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4465.patch
>
>
> Disable label from rm side yarn.nodelabel.enable=false
> Capacity scheduler label configuration for queue is available as below
> default label for queue = b1 as 3 and accessible labels as 1,3
> Submit application to queue A .
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
> Invalid resource request, queue=b1 doesn't have permission to access all
> labels in resource request. labelExpression of resource request=3. Queue
> labels=1,3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:216)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:401)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:340)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:283)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:602)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:247)
> {noformat}
> # Ignore default label expression when label is disabled *or*
> # NormalizeResourceRequest we can set label expression to
> when node label is not enabled *or*
> # Improve message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4219) New levelDB cache storage for timeline v1.5
[ https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated YARN-4219: Attachment: YARN-4219-trunk.005.patch Thanks [~xgong] for the comment! Upload a new patch to address the findbugs warning. > New levelDB cache storage for timeline v1.5 > --- > > Key: YARN-4219 > URL: https://issues.apache.org/jira/browse/YARN-4219 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4219-YARN-4265.001.patch, > YARN-4219-YARN-4265.002.patch, YARN-4219-YARN-4265.003.patch, > YARN-4219-trunk.001.patch, YARN-4219-trunk.002.patch, > YARN-4219-trunk.003.patch, YARN-4219-trunk.004.patch, > YARN-4219-trunk.005.patch > > > We need to have an "offline" caching storage for timeline server v1.5 after > the changes in YARN-3942. The in memory timeline storage may run into OOM > issues when used as a cache storage for entity file timeline storage. We can > refactor the code and have a level db based caching storage for this use > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4380) TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated YARN-4380:
Release Note: (was: Committed this to trunk, branch-2, and branch-2.7.
Thanks Varun Saxena for your contribution.)
> TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails
> intermittently
>
>
> Key: YARN-4380
> URL: https://issues.apache.org/jira/browse/YARN-4380
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 2.8.0, 2.7.1
>Reporter: Tsuyoshi Ozawa
>Assignee: Varun Saxena
> Fix For: 2.7.3, 2.6.4
>
> Attachments: YARN-4380.01.patch,
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell-output.2.txt,
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService-output.txt
>
>
> {quote}
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.361 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
> testDownloadingResourcesOnContainerKill(org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService)
> Time elapsed: 0.109 sec <<< FAILURE!
> org.mockito.exceptions.verification.junit.ArgumentsAreDifferent:
> Argument(s) are different! Wanted:
> deletionService.delete(
> "user0",
> null,
>
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.testDownloadingResourcesOnContainerKill(TestResourceLocalizationService.java:1322)
> Actual invocation has different arguments:
> deletionService.delete(
> "user0",
>
> /home/ubuntu/hadoop-dev/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService/0/usercache/user0/appcache/application_314159265358979_0003/container_314159265358979_0003_01_42
> );
> -> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.testDownloadingResourcesOnContainerKill(TestResourceLocalizationService.java:1296)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService.testDownloadingResourcesOnContainerKill(TestResourceLocalizationService.java:1322)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4559) Make leader elector and zk store share the same curator client
[ https://issues.apache.org/jira/browse/YARN-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106212#comment-15106212 ] Xuan Gong commented on YARN-4559: - The last patch looks good to me. Looks like the test cases failures are not related. Manually kick the Jenkins again. [~kasha] Do you have any other comments ? > Make leader elector and zk store share the same curator client > -- > > Key: YARN-4559 > URL: https://issues.apache.org/jira/browse/YARN-4559 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-4559.1.patch, YARN-4559.2.patch, YARN-4559.3.patch, > YARN-4559.4.patch > > > After YARN-4438, we can reuse the same curator client for leader elector and > zk store -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4219) New levelDB cache storage for timeline v1.5
[ https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106213#comment-15106213 ] Xuan Gong commented on YARN-4219: - Thanks for the patch. [~gtCarrera9]. The latest patch looks good to me. Can we fix the findbugs warning, please ? [~jlowe] Do you have any other comments ? > New levelDB cache storage for timeline v1.5 > --- > > Key: YARN-4219 > URL: https://issues.apache.org/jira/browse/YARN-4219 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4219-YARN-4265.001.patch, > YARN-4219-YARN-4265.002.patch, YARN-4219-YARN-4265.003.patch, > YARN-4219-trunk.001.patch, YARN-4219-trunk.002.patch, > YARN-4219-trunk.003.patch, YARN-4219-trunk.004.patch > > > We need to have an "offline" caching storage for timeline server v1.5 after > the changes in YARN-3942. The in memory timeline storage may run into OOM > issues when used as a cache storage for entity file timeline storage. We can > refactor the code and have a level db based caching storage for this use > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4604) TimelineDataManager should return gracefully when one entity's id or type is empty
Li Lu created YARN-4604: --- Summary: TimelineDataManager should return gracefully when one entity's id or type is empty Key: YARN-4604 URL: https://issues.apache.org/jira/browse/YARN-4604 Project: Hadoop YARN Issue Type: Bug Reporter: Li Lu Assignee: Li Lu As discussed in YARN-4596, when the timeline data manager hit one entity whose id and/or type fields are empty, it should not directly throw exception. It should at least let the client side know which entities have been posted to the timeline server, and which ones haven't. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4603) FairScheduler should mention user requested queuename in error message when failed in queue ACL check
Tao Jie created YARN-4603:
-
Summary: FairScheduler should mention user requested queuename in
error message when failed in queue ACL check
Key: YARN-4603
URL: https://issues.apache.org/jira/browse/YARN-4603
Project: Hadoop YARN
Issue Type: Improvement
Components: fairscheduler
Affects Versions: 2.7.1
Reporter: Tao Jie
Assignee: Tao Jie
Priority: Trivial
When submit a job to a queue does not exist (maybe misspelling), scheduler will
try to submit it to another queue according to QueuePlacementPolicy. For
instance, when I submit a job to queue root.abc, return error message is like:
{quote}
java.io.IOException: Failed to run job : User user1 cannot submit applications
to queue root.default
{quote}
We'd better add original user requested queue name to this error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (YARN-3945) maxApplicationsPerUser is wrongly calculated
[
https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106191#comment-15106191
]
Wangda Tan edited comment on YARN-3945 at 1/19/16 3:31 AM:
---
bq. i feel better not to consider userLimit and userLimitFactor at all, to
reduce the confusion for the number of applications per user.
I would prefer this proposal.
bq. IMO numAppsPerUser can be greater than numAppsPerQueue and user-resource
and user-am-resource greater than the queue's resource or queue's AM resource
only when userLimitFactor is of really greater value, so is it actually
required to be greater than 1, Is it sufficient to restrict this to 1 ?
I think it's better to only cap it by the max possible value of queue (queue's
max capacity / queue's max application number). User can still set ULF as he
wants, but we will return capped value to user.
Changing max value of ULF will be an incompatible changes, since lots of
cluster are using very high ULF (e.g. 100).
was (Author: leftnoteasy):
bq. i feel better not to consider userLimit and userLimitFactor at all, to
reduce the confusion for the number of applications per user.
I would prefer this proposal.
bq. IMO numAppsPerUser can be greater than numAppsPerQueue and user-resource
and user-am-resource greater than the queue's resource or queue's AM resource
only when userLimitFactor is of really greater value, so is it actually
required to be greater than 1, Is it sufficient to restrict this to 1 ?
I think it's better to only cap it by the max possible value of queue (queue's
max capacity / queue's max application number). User can still set ULF as he
wants, but we will return capped value to user.
> maxApplicationsPerUser is wrongly calculated
>
>
> Key: YARN-3945
> URL: https://issues.apache.org/jira/browse/YARN-3945
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.7.1
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-3945.20150728-1.patch, YARN-3945.20150729-1.patch,
> YARN-3945.V1.003.patch
>
>
> maxApplicationsPerUser is currently calculated based on the formula
> {{maxApplicationsPerUser = (int)(maxApplications * (userLimit / 100.0f) *
> userLimitFactor)}} but description of userlimit is
> {quote}
> Each queue enforces a limit on the percentage of resources allocated to a
> user at any given time, if there is demand for resources. The user limit can
> vary between a minimum and maximum value.{color:red} The the former (the
> minimum value) is set to this property value {color} and the latter (the
> maximum value) depends on the number of users who have submitted
> applications. For e.g., suppose the value of this property is 25. If two
> users have submitted applications to a queue, no single user can use more
> than 50% of the queue resources. If a third user submits an application, no
> single user can use more than 33% of the queue resources. With 4 or more
> users, no user can use more than 25% of the queues resources. A value of 100
> implies no user limits are imposed. The default is 100. Value is specified as
> a integer.
> {quote}
> configuration related to minimum limit should not be made used in a formula
> to calculate max applications for a user
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3945) maxApplicationsPerUser is wrongly calculated
[
https://issues.apache.org/jira/browse/YARN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106191#comment-15106191
]
Wangda Tan commented on YARN-3945:
--
bq. i feel better not to consider userLimit and userLimitFactor at all, to
reduce the confusion for the number of applications per user.
I would prefer this proposal.
bq. IMO numAppsPerUser can be greater than numAppsPerQueue and user-resource
and user-am-resource greater than the queue's resource or queue's AM resource
only when userLimitFactor is of really greater value, so is it actually
required to be greater than 1, Is it sufficient to restrict this to 1 ?
I think it's better to only cap it by the max possible value of queue (queue's
max capacity / queue's max application number). User can still set ULF as he
wants, but we will return capped value to user.
> maxApplicationsPerUser is wrongly calculated
>
>
> Key: YARN-3945
> URL: https://issues.apache.org/jira/browse/YARN-3945
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.7.1
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: YARN-3945.20150728-1.patch, YARN-3945.20150729-1.patch,
> YARN-3945.V1.003.patch
>
>
> maxApplicationsPerUser is currently calculated based on the formula
> {{maxApplicationsPerUser = (int)(maxApplications * (userLimit / 100.0f) *
> userLimitFactor)}} but description of userlimit is
> {quote}
> Each queue enforces a limit on the percentage of resources allocated to a
> user at any given time, if there is demand for resources. The user limit can
> vary between a minimum and maximum value.{color:red} The the former (the
> minimum value) is set to this property value {color} and the latter (the
> maximum value) depends on the number of users who have submitted
> applications. For e.g., suppose the value of this property is 25. If two
> users have submitted applications to a queue, no single user can use more
> than 50% of the queue resources. If a third user submits an application, no
> single user can use more than 33% of the queue resources. With 4 or more
> users, no user can use more than 25% of the queues resources. A value of 100
> implies no user limits are imposed. The default is 100. Value is specified as
> a integer.
> {quote}
> configuration related to minimum limit should not be made used in a formula
> to calculate max applications for a user
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3692) Allow REST API to set a user generated message when killing an application
[
https://issues.apache.org/jira/browse/YARN-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-3692:
Target Version/s: 2.9.0
> Allow REST API to set a user generated message when killing an application
> --
>
> Key: YARN-3692
> URL: https://issues.apache.org/jira/browse/YARN-3692
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rajat Jain
>Assignee: Rohith Sharma K S
>
> Currently YARN's REST API supports killing an application without setting a
> diagnostic message. It would be good to provide that support.
> *Use Case*
> Usually this helps in workflow management in a multi-tenant environment when
> the workflow scheduler (or the hadoop admin) wants to kill a job - and let
> the user know the reason why the job was killed. Killing the job by setting a
> diagnostic message is a very good solution for that. Ideally, we can set the
> diagnostic message on all such interface:
> yarn kill -applicationId ... -diagnosticMessage "some message added by
> admin/workflow"
> REST API { 'state': 'KILLED', 'diagnosticMessage': 'some message added by
> admin/workflow'}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4596) SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities
[ https://issues.apache.org/jira/browse/YARN-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106240#comment-15106240 ] Li Lu commented on YARN-4596: - I've created YARN-4604 and MAPREDUCE-6610 to trace the two problems raised in the discussion of this issue. Thanks [~Naganarasimha]! > SystemMetricPublisher should not swallow error messages from > TimelineClient#putEntities > --- > > Key: YARN-4596 > URL: https://issues.apache.org/jira/browse/YARN-4596 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.8.0 > > Attachments: YARN-4596-trunk.001.patch, YARN-4596-trunk.002.patch > > > We should report error messages from the returned TimelineResponse when > posting timeline entities through system metric publisher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4557) Improper Queues sorting in PartitionedQueueComparator when accessible node labels is configured as ANY
[ https://issues.apache.org/jira/browse/YARN-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4557: Attachment: YARN-4557.v3.002.patch Thanks for the review [~wangda], yes it was a typo, have corrected it in the latest patch > Improper Queues sorting in PartitionedQueueComparator when accessible node > labels is configured as ANY > -- > > Key: YARN-4557 > URL: https://issues.apache.org/jira/browse/YARN-4557 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Minor > Attachments: YARN-4557.v1.001.patch, YARN-4557.v2.001.patch, > YARN-4557.v2.002.patch, YARN-4557.v3.001.patch, YARN-4557.v3.002.patch > > > * When queue has * as accessibility, then the queue ordering was not > happening properly. > Few Small nits > * In AppSchedulingInfo comparator field doesn't have generics > * TestNodeLabelContainerAllocation.testResourceRequestUpdateNodePartitions > has unused variable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4605) Spelling mistake in the help message of "yarn applicationattempt" command
Manjunath Ballur created YARN-4605: -- Summary: Spelling mistake in the help message of "yarn applicationattempt" command Key: YARN-4605 URL: https://issues.apache.org/jira/browse/YARN-4605 Project: Hadoop YARN Issue Type: Bug Components: client, yarn Reporter: Manjunath Ballur Priority: Trivial Using YARN CLI, when the user types "yarn applicationattempt", the help message for the "applicationattempt" command is shown. Here, the following line, has a spelling mistake. "application" is misspelled as "aplication": -listList application attempts for aplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4603) FairScheduler should mention user requested queuename in error message when failed in queue ACL check
[
https://issues.apache.org/jira/browse/YARN-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106303#comment-15106303
]
Hadoop QA commented on YARN-4603:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 14s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 56s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 149m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782986/YARN-4603.001.patch |
| JIRA
[jira] [Commented] (YARN-4596) SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities
[
https://issues.apache.org/jira/browse/YARN-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106332#comment-15106332
]
Varun Saxena commented on YARN-4596:
[~Naganarasimha], [~gtCarrera9],
Regarding below comment,
bq. but if some one uses rest directly then some entities will get posted and
some will not, hence wanted to know whether its right to break in the middle ?
Not really.
*None of the entities will get posted to backend store.*
*BadRequestException* is thrown during entity preprocessing phase. Not during
store put. We first collect all the applicable entities which can be posted and
then only call {{TimelineStore#put}}. We are not calling put from within the
loop. Refer to {{TimelineDataManager#doPostEntities}}.
It can although be argued that we can post entities which can be and send a put
error(new error type) for the entity which has no entity id or entity type. But
if we do not have entity id or type, how will client recognize which specific
entity has failed from put error ? Maybe they can cache entities which they
have sent and then re-look at entities they attempted to post and find out
which ones had entity id and entity type missing. But will client be doing so ?
Current clients would be looking at entity id and type to find out which
entities failed and why.
Entity ID and type are key pieces of information in ATSv1. If client is not
sending them, it indicates some problem in the client side code. I think
current code can be kept as it is because this will be consistent with
{{TimelineClient}} side code too where none of the entities will be posted if
entity id or type is missing in even one of them.
Thoughts ?
> SystemMetricPublisher should not swallow error messages from
> TimelineClient#putEntities
> ---
>
> Key: YARN-4596
> URL: https://issues.apache.org/jira/browse/YARN-4596
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Reporter: Li Lu
>Assignee: Li Lu
> Fix For: 2.8.0
>
> Attachments: YARN-4596-trunk.001.patch, YARN-4596-trunk.002.patch
>
>
> We should report error messages from the returned TimelineResponse when
> posting timeline entities through system metric publisher.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4604) TimelineDataManager should return gracefully when one entity's id or type is empty
[
https://issues.apache.org/jira/browse/YARN-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106338#comment-15106338
]
Varun Saxena commented on YARN-4604:
Copying over my comment from YARN-4596.
bq. It should at least let the client side know which entities have been posted
to the timeline server, and which ones haven't.
*None of the entities will get posted to backend store.*
*BadRequestException* is thrown during entity preprocessing phase. Not during
store put. We first collect all the applicable entities which can be posted and
then only call {{TimelineStore#put}}. We are not calling put from within the
loop. Refer to {{TimelineDataManager#doPostEntities}}.
It can although be argued that we can post entities which can be and send a put
error(new error type) for the entity which has no entity id or entity type. But
if we do not have entity id or type, how will client recognize which specific
entity has failed from put error ? Maybe they can cache entities which they
have sent and then re-look at entities they attempted to post and find out
which ones had entity id and entity type missing. But will client be doing so ?
Current clients would be looking at entity id and type to find out which
entities failed and why.
Entity ID and type are key pieces of information in ATSv1. If client is not
sending them, it indicates some problem in the client side code. I think
current code can be kept as it is because this will be consistent with
{{TimelineClient}} side code too where none of the entities will be posted if
entity id or type is missing in even one of them.
Thoughts ?
> TimelineDataManager should return gracefully when one entity's id or type is
> empty
> --
>
> Key: YARN-4604
> URL: https://issues.apache.org/jira/browse/YARN-4604
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Li Lu
>Assignee: Li Lu
>
> As discussed in YARN-4596, when the timeline data manager hit one entity
> whose id and/or type fields are empty, it should not directly throw
> exception. It should at least let the client side know which entities have
> been posted to the timeline server, and which ones haven't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4604) TimelineDataManager should return gracefully when one entity's id or type is empty
[ https://issues.apache.org/jira/browse/YARN-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106344#comment-15106344 ] Naganarasimha G R commented on YARN-4604: - [~varun_saxena], Thanks for correcting it out, yes we can keep as it its ! > TimelineDataManager should return gracefully when one entity's id or type is > empty > -- > > Key: YARN-4604 > URL: https://issues.apache.org/jira/browse/YARN-4604 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Li Lu >Assignee: Li Lu > > As discussed in YARN-4596, when the timeline data manager hit one entity > whose id and/or type fields are empty, it should not directly throw > exception. It should at least let the client side know which entities have > been posted to the timeline server, and which ones haven't. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4219) New levelDB cache storage for timeline v1.5
[
https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106326#comment-15106326
]
Hadoop QA commented on YARN-4219:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 3s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 6s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 41s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 13s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 12 new +
225 unchanged - 1 fixed = 237 total (was 226) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 8s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 5s
{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit
[jira] [Assigned] (YARN-4605) Spelling mistake in the help message of "yarn applicationattempt" command
[ https://issues.apache.org/jira/browse/YARN-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang reassigned YARN-4605: - Assignee: Weiwei Yang > Spelling mistake in the help message of "yarn applicationattempt" command > - > > Key: YARN-4605 > URL: https://issues.apache.org/jira/browse/YARN-4605 > Project: Hadoop YARN > Issue Type: Bug > Components: client, yarn >Reporter: Manjunath Ballur >Assignee: Weiwei Yang >Priority: Trivial > Attachments: YARN-4605.001.patch > > > Using YARN CLI, when the user types "yarn applicationattempt", the help > message for the "applicationattempt" command is shown. > Here, the following line has a spelling mistake. "application" is misspelled > as "aplication": > -listList application attempts for aplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4605) Spelling mistake in the help message of "yarn applicationattempt" command
[ https://issues.apache.org/jira/browse/YARN-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manjunath Ballur updated YARN-4605: --- Description: Using YARN CLI, when the user types "yarn applicationattempt", the help message for the "applicationattempt" command is shown. Here, the following line has a spelling mistake. "application" is misspelled as "aplication": -listList application attempts for aplication. was: Using YARN CLI, when the user types "yarn applicationattempt", the help message for the "applicationattempt" command is shown. Here, the following line, has a spelling mistake. "application" is misspelled as "aplication": -listList application attempts for aplication. > Spelling mistake in the help message of "yarn applicationattempt" command > - > > Key: YARN-4605 > URL: https://issues.apache.org/jira/browse/YARN-4605 > Project: Hadoop YARN > Issue Type: Bug > Components: client, yarn >Reporter: Manjunath Ballur >Priority: Trivial > > Using YARN CLI, when the user types "yarn applicationattempt", the help > message for the "applicationattempt" command is shown. > Here, the following line has a spelling mistake. "application" is misspelled > as "aplication": > -listList application attempts for aplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4605) Spelling mistake in the help message of "yarn applicationattempt" command
[ https://issues.apache.org/jira/browse/YARN-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated YARN-4605: -- Attachment: YARN-4605.001.patch I do see several places having such typo, changes is trivial, just submitted a patch to fix this. > Spelling mistake in the help message of "yarn applicationattempt" command > - > > Key: YARN-4605 > URL: https://issues.apache.org/jira/browse/YARN-4605 > Project: Hadoop YARN > Issue Type: Bug > Components: client, yarn >Reporter: Manjunath Ballur >Priority: Trivial > Attachments: YARN-4605.001.patch > > > Using YARN CLI, when the user types "yarn applicationattempt", the help > message for the "applicationattempt" command is shown. > Here, the following line has a spelling mistake. "application" is misspelled > as "aplication": > -listList application attempts for aplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-4604) TimelineDataManager should return gracefully when one entity's id or type is empty
[ https://issues.apache.org/jira/browse/YARN-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu resolved YARN-4604. - Resolution: Won't Fix > TimelineDataManager should return gracefully when one entity's id or type is > empty > -- > > Key: YARN-4604 > URL: https://issues.apache.org/jira/browse/YARN-4604 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Li Lu >Assignee: Li Lu > > As discussed in YARN-4596, when the timeline data manager hit one entity > whose id and/or type fields are empty, it should not directly throw > exception. It should at least let the client side know which entities have > been posted to the timeline server, and which ones haven't. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4604) TimelineDataManager should return gracefully when one entity's id or type is empty
[ https://issues.apache.org/jira/browse/YARN-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106342#comment-15106342 ] Li Lu commented on YARN-4604: - Yes you're right. I missed the preprocess part. Close this as won't fix. > TimelineDataManager should return gracefully when one entity's id or type is > empty > -- > > Key: YARN-4604 > URL: https://issues.apache.org/jira/browse/YARN-4604 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Li Lu >Assignee: Li Lu > > As discussed in YARN-4596, when the timeline data manager hit one entity > whose id and/or type fields are empty, it should not directly throw > exception. It should at least let the client side know which entities have > been posted to the timeline server, and which ones haven't. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4224) Support fetching entities by UID and change the REST interface to conform to current REST APIs' in YARN
[
https://issues.apache.org/jira/browse/YARN-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105906#comment-15105906
]
Hadoop QA commented on YARN-4224:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} feature-YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 28s
{color} | {color:green} feature-YARN-2928 passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 9 new +
48 unchanged - 14 fixed = 57 total (was 62) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License
[jira] [Commented] (YARN-4596) SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities
[ https://issues.apache.org/jira/browse/YARN-4596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105838#comment-15105838 ] Li Lu commented on YARN-4596: - Ah I've got your point. Yes we can definitely improve the logic here. Please feel free to raise a JIRA about it. > SystemMetricPublisher should not swallow error messages from > TimelineClient#putEntities > --- > > Key: YARN-4596 > URL: https://issues.apache.org/jira/browse/YARN-4596 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4596-trunk.001.patch, YARN-4596-trunk.002.patch > > > We should report error messages from the returned TimelineResponse when > posting timeline entities through system metric publisher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4601) HA as a general YARN service to highlighted container by application.
[ https://issues.apache.org/jira/browse/YARN-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105791#comment-15105791 ] Junping Du commented on YARN-4601: -- I believe it is potentially to be critical in prospective of looking at YARN as a competitive distributed OS to get more applications running on top of it. Thoughts? > HA as a general YARN service to highlighted container by application. > - > > Key: YARN-4601 > URL: https://issues.apache.org/jira/browse/YARN-4601 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > > For LRS (long running services) on YARN, get rid of single point failure for > critical container failure may not be necessary. Some applications would like > to build its own HA architecture. However, it would be ideal to provide some > fundamental support to HA service in YARN, like: launching container marked > with active/standby, monitor/trigger out failed over, provide end point for > shring information between active/standby container, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4219) New levelDB cache storage for timeline v1.5
[
https://issues.apache.org/jira/browse/YARN-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105821#comment-15105821
]
Hadoop QA commented on YARN-4219:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 4s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 12 new +
225 unchanged - 1 fixed = 237 total (was 226) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Updated] (YARN-4224) Support fetching entities by UID and change the REST interface to conform to current REST APIs' in YARN
[ https://issues.apache.org/jira/browse/YARN-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4224: --- Attachment: YARN-4224-feature-YARN-2928.04.patch > Support fetching entities by UID and change the REST interface to conform to > current REST APIs' in YARN > --- > > Key: YARN-4224 > URL: https://issues.apache.org/jira/browse/YARN-4224 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Attachments: YARN-4224-YARN-2928.01.patch, > YARN-4224-feature-YARN-2928.04.patch, > YARN-4224-feature-YARN-2928.wip.02.patch, > YARN-4224-feature-YARN-2928.wip.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
