[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236032#comment-16236032
 ] 

Haibo Chen commented on YARN-4511:
--

Checking in this shortly. Thanks [~asuresh] [~miklos.szeg...@cloudera.com] 
[~leftnoteasy] a lot for the reviews!

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch, YARN-4511-YARN-1011.10.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235971#comment-16235971
 ] 

Arun Suresh commented on YARN-4511:
---

+1, The latest patch lgtm

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch, YARN-4511-YARN-1011.10.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234822#comment-16234822
 ] 

Haibo Chen commented on YARN-4511:
--

[~asuresh] [~leftnoteasy] Do you have more comments on the latest patch? I can 
incorporate them along with fixing the checkstyle issue.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch, YARN-4511-YARN-1011.10.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226157#comment-16226157
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 14 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m 
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 
56s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m  
4s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 2s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  9m 
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m  6s{color} | {color:orange} root: The patch generated 7 new + 1358 unchanged 
- 4 fixed = 1365 total (was 1362) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
7m 28s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
8s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 12s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
48s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 51s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.guaranteedContainerReleased(Container)
  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.guaranteedContainerReleased(Container)
  At SchedulerNode.java:[line 383] |
|  |  Increment of volatile field 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226021#comment-16226021
 ] 

Haibo Chen commented on YARN-4511:
--

Filed YARN-7337 for SchedulerNodeReport changes because I think there is more 
work to do there.

Uploaded the patch to include the following changes:
1) remove duplicate isDebugEnabled()
2) replace assert statements with throw YarnRuntimeExceptions
3) rename guaranteedContainerResourceReleased() to guaranteedContainerReleased,
 include all necessary updates. Similarly for 
opportunisticContainerResourceReleased(),
 opportunisticContainerResourceAllocated(), 
guaranteedContainerResourceAllocated()
4) replace containerAllocated(resource, allocatedResourceOpportunistic) with
if (containerAllocated(resource, allocatedResourceOpportunistic)) {
 // nothing else to do
} 
to make it consistent with guaranteed container allocation.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225984#comment-16225984
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks [~asuresh] for your review!
bq. Is there a case where the 'resource' argument might be null in the former 
method call (since that is only case when the containerResourceAllocated method 
can return false)?
Not sure. But this is the same behavior as before, so I figure it's safer to 
preserve it.  The reason why I did not do the same in 
opportunisticContainerResourceAllocated is that 
there is nothing else to do if containerResourceAlllocated returns to true. 
i.e., if we were to make it consistent, it'll be like 
{code}
if (containerAllocated(resource, allocatedResourceOpportunistic)) {
 // nothing else to do
}
{code}
Not sure what's best style here. Open to all suggestions/preferences.

bq. but then in the SchedulerNodeReport, shouldn't this.num = numOpp + 
numGuaranteed ?
Yes. but given how used/avail is assigned in SchedulerNodeReport (used -> 
guaranteedResourceUsed, num-> numGuaranteedContainers),
I created YARN-7337 to augment SchedulerNodeReport with opportunistic container 
stats instead of modifying existing variables and also
make sure it is exposed properly in NodeReport api

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225973#comment-16225973
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks [~leftnoteasy] for the review!
bq. assert in the main code (Such as SchedulerNode) will be removed at runtime. 
Do you want to throw exception instead?
The assert is meant mainly for documentation purpose. I'll replace them with 
throw exception instead.
bq. Moving following statements to guaranteedContainerResourceReleased? 
Good point. Will do that and rename it to guaranteedContainerReleased given its 
semantics are changed, and likewise for guaranteedContainerAllocated.



> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223840#comment-16223840
 ] 

Arun Suresh commented on YARN-4511:
---

Thanks for updating the patch [~haibochen]. I am able to follow this a bit 
better.

Some Comments:
- In SchedulerNode, you are doing an if check inside 
{{guaranteedContainerResourceAllocated}}, but you do not  do the same in 
{{opportunisticContainerResourceAllocated}}. Is there a case where the 
'resource' argument might be null in the former method call (since that is only 
case when the {{containerResourceAllocated}} method can return false)?
{code}
  if (containerResourceAllocated(resource, allocatedResourceGuaranteed)) {
  Resources.subtractFrom(unallocatedResource, resource);
}
{code}
- I see a similar pattern as above in {{guaranteedContainerResourceReleased}} 
and {{opportunisticContainerResourceReleased}}
- Thank you for splitting numContainers into numGuaranteedContainers and 
numAllocatedContainers, but then in the SchedulerNodeReport, shouldn't this.num 
= numOpp + numGuaranteed ?

Everything else looks fine to me.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223824#comment-16223824
 ] 

Wangda Tan commented on YARN-4511:
--

[~haibo.chen], 

Haven't reviewed all details of the patch, but the latest patch looks much 
cleaner than previous one. I took a closer look at SchedulerNode.

Few comments: 
- {{assert}} in the main code (Such as SchedulerNode) will be removed at 
runtime. Do you want to throw exception instead?
- Moving following statements:
{code}
  // do not update allocated containers until the resources of the
  // container are released because we need to check if we need
  // to update resourceAllocatedPendingLaunch in case the container
  // has not been launched on the node.
  allocatedContainers.remove(containerId);
  numGuaranteedContainers--;
{code}
To {{guaranteedContainerResourceReleased}}? And same for the 
{{opportunisticContainerResourceReleased}}
- Duplicated isDebugEnabled: 
{code}
if (LOG.isDebugEnabled()) {
  if (LOG.isDebugEnabled()) {
{code} 

Please proceed if you get a +1 from other committer. (Since I haven't reviewed 
all other logics)

+ [~asuresh] in case he wants to take a look.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222743#comment-16222743
 ] 

Haibo Chen commented on YARN-4511:
--

I manually kicked off jenkins at 
https://builds.apache.org/job/PreCommit-YARN-Build/18198/console. 
There were no unit test failures, so the patch without swapContainer change is 
sufficient.

[~wangda], do you want to take a look the v9 of the patch now that it has no 
swapContainer changes?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch, 
> YARN-4511-YARN-1011.09.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220826#comment-16220826
 ] 

Haibo Chen commented on YARN-4511:
--

Strangely, as I remove changes to swapContainer, the unit tests that I have 
seen previously failing in the first jenkins report are passing now.
I may have misled everyone, my apologies folks. Will upload the patch to 
confirm.

The offline sync is mainly for me to have a better understanding of the 
implementation container resizing/promotion as I can foresee future
 changes may also need to take container resizing into consideration.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219692#comment-16219692
 ] 

Wangda Tan commented on YARN-4511:
--

[~haibo.chen], 

ContainerUpdateContext#swapContainer 
1) Instead of using assert, should we throw exception? Assertions will be 
removed at the runtime.

2) I'm not sure why it need acquire Node and do operations, the original 
purpose of swapContainers/ContainerUpdateContext is (I believe you understand 
the code, just to make sure we have no differences here):

a. When a promotion or demotion request comes (same as increase/decrease 
container), ContainerUpdateContext calculate resource differences (for example, 
promote a 2G opportunistic container means request a 2G node-local request)and 
send request to scheduler.
b. Scheduler handles the increase/decrease request, which creates a new 
Container and AM pulls it. 
- b.1. If it is a increase request, RM changes internal resource accounting 
including SchedulerNode/Queue/Application, etc.
- b.2. If it is a decrease request, RM will not change resource accounting 
immediately. Instead, inside ContainerUpdateContext#swapContainer, it set 
to-be-released resource to tempContainerToKill. Which will be sent to scheduler 
to release async.

So my question is, is it possible to avoid the sync lock of SchedulerNode 
inside swapContainer? Which looks dangerous and potentially make implementation 
complicated. 

I do want to review other part of the code (such as SchedulerNode) in one shot, 
however I found they might be related to the swapContainer implementation.

Please let me know if you want a conf call to discuss this easier.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218975#comment-16218975
 ] 

Haibo Chen commented on YARN-4511:
--

[~leftnoteasy] Would you like to look at the patch as well while I am working 
on the SLS tests?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217791#comment-16217791
 ] 

Jason Lowe commented on YARN-4511:
--

Yeah for a dev branch scenario it greatly reduces the number of impacted people 
and precommit builds, but it still leaves people open to thinking they didn't 
break anything _new_ in their subsequent change because "those tests have 
always been failing."  It's easy for them to overlook tests that could be 
failing in a new way.

Sorry for the interruption, now back to your regularly scheduled program. ;-)


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217727#comment-16217727
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks [~jlowe] for the comment. My apologies for raising alarms by posting 
comments like that. I was more to support my point that this should be part of 
the patch.
Prior to posting the comment though, I did think that given this is going to 
branch YARN-1011 only and only a few of us touch this branch, we'd get away by 
quickly
fixing the failures in a following-up patch. But I guess even that may not be a 
good idea at all.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217700#comment-16217700
 ] 

Jason Lowe commented on YARN-4511:
--

bq. I can the patch up if we are willing to check in the rest of the patch even 
though the change causes new unit test failures. Thoughts?

Apologies for showing up late.  I've not been following this JIRA that closely, 
but I did see this go by and it raised alarms.  We should *never* check in code 
that knowingly breaks unit tests.  It is quite disruptive to other precommit 
builds causing other contributors and reviewers to have to triage the test 
failures to make sure they are not related.  The whole point of having 
automated unit tests is to help know if a change is safe.  If for some reason a 
change just _has_ to go in before the problems with the tests can be properly 
fixed then we might as well just disable the tests and file a followup JIRA to 
fix them rather than let them fail and cause everyone to keep rehashing the 
same test failures over and over.

My apologies if I misunderstood the intentions of that comment, but I'm a 
strong -1 to any change that is known to trigger unit test failures.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217679#comment-16217679
 ] 

Haibo Chen commented on YARN-4511:
--

bq. Have another patch for the changes to the swapContainer etc. 
The early versions of this patch does not include the swap Container changes. 
This is added to address some unit test failures IIUC.
Previously we don't distinguish between opportunistic and guaranteed resources 
when it comes to resource accounting. This is however,
changed, with the latest patch. Hence, the change to swapContainers. I can the 
patch up if we are willing to check in the rest
of the patch even though the change causes new unit test failures. Thoughts?

bq. please make sure to test it either via SLS / running actual jobs.
Good point. Will run some tests in SLS if we decide not to split.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217620#comment-16217620
 ] 

Arun Suresh commented on YARN-4511:
---

So with regard to the extra synchronization in {{swapContainer()}}, please make 
sure to test it either via SLS / running actual jobs - since there can be a 
possibility of deadlock (ok - not dead lock, but quite perceptible slow down) 
if the node in question tries to perform a heartbeat during the allocate call.
I am thinking maybe we split this patch:
* Keep this patch to track modifications required for addition of the 
OverAllocationInfo
* Have another patch for the changes to the {{swapContainer}} etc. That way, we 
can actually see how the modifications to the ContainerUpdateContext are being 
used.
Thoughts?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215729#comment-16215729
 ] 

Miklos Szegedi commented on YARN-4511:
--

[~leftnoteasy], [~asuresh], do you have any more comments on the patch?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch, YARN-4511-YARN-1011.08.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215593#comment-16215593
 ] 

Haibo Chen commented on YARN-4511:
--

Right. Will update the patch real quick

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215566#comment-16215566
 ] 

Miklos Szegedi commented on YARN-4511:
--

Thank you, [~haibochen] for the patch.
{code}
342 // notify schedulerNode of the update to correct resource accounting
343 node.containerUpdated(existingRMContainer, existingContainer);
344 
345 
((RMContainerImpl)tempRMContainer).setContainer(updatedTempContainer);
346 // notify SchedulerNode of the update to correct resource accounting
347 node.containerUpdated(tempRMContainer, tempContainer);
{code}
I think these still need to be atomic from the perspective of the node. It may 
happen that there is an allocation decision before the release and after 
reacquiring the resources here. This may lead to double allocation of the same 
resources on the node.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch, 
> YARN-4511-YARN-1011.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214203#comment-16214203
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  9m  
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 14 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
32s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
48s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
21s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
11s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 52s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 22s{color} | {color:orange} root: The patch generated 10 new + 1406 
unchanged - 4 fixed = 1416 total (was 1410) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  7s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
28s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 46m 
31s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m  
7s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
40s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 43s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:[line 182] |
|  |  Increment of volatile field 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213028#comment-16213028
 ] 

Haibo Chen commented on YARN-4511:
--

YARN-7373 is created for the container update atomicity discussion.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213013#comment-16213013
 ] 

Haibo Chen commented on YARN-4511:
--

bq. If containerResourceAllocated fails in guaranteedContainerResourceAllocated 
we will still call allocatedContainers.put(). I think this may cause some 
inconsistencies in the future. Probably it is better to propagate the false 
return code all the way to the caller.
bq. guaranteedContainerResourceReleased may fail inside but regardless of the 
outcome, we decrease numGuaranteedContainers.
These two are the current behavior without the patch. The resource release can 
fail only if resource is null, in which case is equivalent to releasing a 
zero-sized container, but it won't cause any inconsistency. 

bq.  I think that it would be nicer to lock around these two calls to become 
atomic.
That's a valid concern. container update and thus swap is atomic according to 
[~asuresh]. But that is indeed not very clear. Let's discuss this in another 
jira to see if we can improve it.
 
Will address the rest of your comments in the next patch plus unit tests.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212971#comment-16212971
 ] 

Haibo Chen commented on YARN-4511:
--

bq. however we need to make sure it reflects the state of the object, so for 
example allocateContainer() should set this value as the last step after the 
allocatedContainers.put() call. 
bq. containerResourceReleased should decrease resourceAllocatedPendingLaunch, 
if the container has not been started, yet.
Good points, will address in the next patch.
bq. I think that it would be nicer to lock around these two calls to become 
atomic.
swapContainer() is already protected in a writeLock, so it is already atomic, 
no?

bq. isValidGuaranteedContainer and isValidOpportunisticContainer contain the 
same code. Should they be different? 
I'm inclined to keep both of them. The caller may want to check whether it is a 
guaranteed or opportunistic, not just whether it has been allocated on the node
It just so happens that we are sharing the same map for both OPPORTUNISTIC and 
GUARANTEED containers, hence the code is identical.
I'll add Execution Type check to be more rigorous.

bq. allocatedContainers.remove(containerId); can be placed outside the if.
{code:java}
if (container.getExecutionType() == ExecutionType.GUARANTEED) {
  guaranteedContainerResourceReleased(container);
  numGuaranteedContainers--;
} else {
  opportunisticContainerResourceReleased(container);
  numOpportunisticContainers--;
}
allocatedContainers.remove(containerId);
{code}
The above code will update the num*Containers counter before 
allocatedContainers is updated, so I think we should keep it as it.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212848#comment-16212848
 ] 

Miklos Szegedi commented on YARN-4511:
--

Thank you, [~haibochen] for the patch.
{code}
342 // notify schedulerNode of the update to correct resource accounting
343 node.containerUpdated(existingRMContainer, existingContainer);
344 
345 
((RMContainerImpl)tempRMContainer).setContainer(updatedTempContainer);
346 // notify SchedulerNode of the update to correct resource accounting
347 node.containerUpdated(tempRMContainer, tempContainer);
348 
{code}
I think that it would be nicer to lock around these two calls to become atomic.
{code}
431   public int getNumOpportunisticContainers() {
432 return numOpportunisticContainers;
321   }
{code}
This function takes a sample but does not lock. This is fine, however we need 
to make sure it reflects the state of the object, so for example 
allocateContainer() should set this value as the last step after the 
allocatedContainers.put() call.
If containerResourceAllocated fails in guaranteedContainerResourceAllocated we 
will still call allocatedContainers.put(). I think this may cause some 
inconsistencies in the future. Probably it is better to propagate the false 
return code all the way to the caller.
isValidGuaranteedContainer and isValidOpportunisticContainer contain the same 
code. Should they be different? Would an isValidContainer function be 
sufficient?
{code}
294 Container container = rmContainer.getContainer();
295 if (container.getExecutionType() == ExecutionType.GUARANTEED) {
296   guaranteedContainerResourceReleased(container);
297   allocatedContainers.remove(containerId);
298   numGuaranteedContainers--;
299 } else {
300   opportunisticContainerResourceReleased(container);
301   numOpportunisticContainers--;
302   allocatedContainers.remove(containerId);
303 }
{code}
allocatedContainers.remove(containerId); can be placed outside the if.

containerResourceReleased should decrease resourceAllocatedPendingLaunch, if 
the container has not been started, yet.

guaranteedContainerResourceReleased may fail inside but regardless of the 
outcome, we decrease numGuaranteedContainers.
{{ + ", which has " + getNumGuaranteedContainers() + " containers, "}} should 
be {{ + ", which has " + getNumGuaranteedContainers() + " guaranteed 
containers, "}}
I do not see unit tests added for getNumOpportunisticContainers() and 
opportunistic container code paths added in general.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212335#comment-16212335
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
 0s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
11s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 6s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
12s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 47s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
14s{color} | {color:green} root: The patch generated 0 new + 535 unchanged - 2 
fixed = 535 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  2s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m  2s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
58s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 23s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 |
|   | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | YARN-4511 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12893173/YARN-4511-YARN-1011.06.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 18e59deb6425 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-1011 / e5a996e 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212171#comment-16212171
 ] 

Haibo Chen commented on YARN-4511:
--

Upload a new patch now that YARN-1011 is rebased on top of latest trunk which 
has YARN-7112.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch, YARN-4511-YARN-1011.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212150#comment-16212150
 ] 

Haibo Chen commented on YARN-4511:
--

TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable() is a very 
known failure.
The sls test failures are likely due to YARN-7112. Will rebase YARN-1011 branch 
to see if they go away.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch, 
> YARN-4511-YARN-1011.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212124#comment-16212124
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  6m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
 6s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 
21s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
19s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
22s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
51s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
12s{color} | {color:green} root: The patch generated 0 new + 535 unchanged - 2 
fixed = 535 total (was 537) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 57s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
22s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 37s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 49s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:[line 177] |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numOpportunisticContainers
 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212121#comment-16212121
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
29s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
42s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
52s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 43s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
38s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 9s{color} | {color:green} root: The patch generated 0 new + 534 unchanged - 2 
fixed = 534 total (was 536) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 25s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
22s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 19s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 46s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
38s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 22s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:[line 177] |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numOpportunisticContainers
 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211631#comment-16211631
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks [~miklos.szeg...@cloudera.com] for the review! I will address your 
comments in the following patch,
but will wait until the HADOOP-14816 is resolved so that the jenkins can give 
some feedback.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Miklos Szegedi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211480#comment-16211480
 ] 

Miklos Szegedi commented on YARN-4511:
--

Thank you, [~haibochen] for the patch. I have a few comments.
{code}
112   public synchronized void updateTotalResource(Resource resource){
{code}
I think we need to clone resource before assigning.
{code}
308   public RMContainer swapContainer(RMContainer tempRMContainer,
{code}
The two node updates should be atomic, and I would release the resources first 
and assign second.
{code}
177   allocatedContainers.put(
178   container.getId(),
179   new ContainerInfo(rmContainer, launchedOnNode));
{code}
I think this can be pulled outside the if.
{code}
203   public synchronized void guaranteedContainerResourceAllocated(
{code}
It might be helpful, to throw if the unallocated resource is less than the 
request.
{code}
419 getNumOpportunisticContainers()+ " available=" +
{code}
There is a missing space, also we mention containers in the row above instead 
of guaranteed containers.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210615#comment-16210615
 ] 

Haibo Chen commented on YARN-4511:
--

Looks like the build has been failing due to 404 when downloading oracle jdk

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210576#comment-16210576
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  0m 
10s{color} | {color:red} Docker failed to build yetus/hadoop:71bbb86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-4511 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892474/YARN-4511-YARN-1011.04.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/18023/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210571#comment-16210571
 ] 

Haibo Chen commented on YARN-4511:
--

The findbug warnings are bogus in that all the increments/decrements of the 
volatile variable are protected by synchronized. Not sure what is going on with 
the build error that I could not reproduce locally. Will retrigger the job 
manually

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206815#comment-16206815
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  6m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
49s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
10s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
52s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  9s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
54s{color} | {color:green} root: The patch generated 0 new + 534 unchanged - 2 
fixed = 534 total (was 536) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 22s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
20s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
49s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 7 new + 297 unchanged - 0 fixed = 304 total (was 297) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
41s{color} | {color:red} hadoop-tools_hadoop-sls generated 4 new + 20 unchanged 
- 0 fixed = 24 total (was 20) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 29s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 21s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206632#comment-16206632
 ] 

Haibo Chen commented on YARN-4511:
--

Patch updated to address the follow major issues:
1) rename SchedulerNode.launchedContainers to allocatedContainers to avoid 
confusion with the launchedOnNode tag.
2) rename allocationInThisHeart  to resourcesOfContainersPendingLaunch which is 
incremented when containers are allocated and decremented when containers are 
launched on the node. It is now detached from node heartbeat, should also be 
fine in the case of global scheduler.
3) replace two nodes passed to ContainerUpdateContext.swap() with a single 
node, given Arun's comment that container update is done locally on a node.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206358#comment-16206358
 ] 

Haibo Chen commented on YARN-4511:
--

Yep, I have created YARN-7334 as a blocker to document the API/proto changes as 
well as user guide. Will upload a patch that addresses the rest of your 
comments soon.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206297#comment-16206297
 ] 

Wangda Tan commented on YARN-4511:
--

Thanks [~haibo.chen] for the explanation.

bq. I am thinking of decoupling allocationInThisHeart from node heartbeat by 
renaming it to resourcesOfContainersPendingLaunch and update it in 
containerStarted()
method instead of resetting every node heartbeat. Let me know what you think.
Sounds good to me. 

bq. The launchedContainer should probably be renamed to allocatedContainer and 
launchedOnNode is to track whether the allocated
container is actually launched on the node. This piece code already exists. I 
can do the renaming if you are fine with it.
+1 to rename it now. 

bq. Very much for the same reason you are thinking of here, I am more inclined 
to keep OverAllocationInfo for now. I am not sure if we just have 
ResourceThresholds,
how we can keep backward compatibility in a clean way if we ever want to 
include more for NM overallocation configs. I agree we should do the 
consolidating
with resource profiles before the release, I think we can revisit this topic 
then.
Make sense if we can have a clean API/proto change before release.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202335#comment-16202335
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks for the background on YARN-5139, [~leftnoteasy]. 

My understanding of SchedulerNode from scheduler's perspective is that it keeps 
track of the set of allocated containers on a given node and how much
resources of the node are being in use or left for allocation. The 
SchedulerNode is notified whenever there is a container allocated, launched or 
released
on that node to update its bookkeeping. The major change of SchedulerNode in 
this patch is to account for Opportunistic containers in a different way
than we do for Guaranteed containers. Specifically, we don't include resources 
of Opportunistic container in SchedulerNode.allocatedResource. A quick
look at Capacity Scheduler shows me that SchedulerNode is notified of container 
allocation only when allocation proposal is accepted, so I believe this
patch won't change how YARN-5139 behaves.  

{code:java}allocationInThisHeartbeat{code}, however, does need to be changed 
given the way scheduling is not driven by node heartbeat in YARN-5139.
The purpose of this variable is to track how much resources allocated 
containers that have not yet launched are going to use (based on resource 
request,
since they can use all resources they have requested in the worst case if they 
were to run on the node). To illustrate the workflow of this patch and what
allocationInThisHeartbeat is for, let's say on a node of 10 GB of memory, there 
are already 10 containers running (in aggregate requested 10GB of memory)
and the resource utlization reported in the node heartbeat is 5GB of memory, 
there are 2 containers that are just allocated but not yet launched and they
two together request 2GB of memory. In the case of oversubscription, scheduler 
will try to allocate Opportunistic containers based on node resource 
utilization.
5GB is what the running containers are using and 2GB is probably soon to be 
utilized, so the scheduler will think I'd better assume that the resource 
utilization
is 7GB and so only 3GB is left, then decide whether to continue to allocate 
OPPORTUNISTIC containers given the node's overallocation threshold. How the 
3GB is calculated is done by allowedResourceForOverAllocation() and 
allocationInThisHeartbeat.

I am thinking of decoupling allocationInThisHeart from node heartbeat by 
renaming it to resourcesOfContainersPendingLaunch and update it in 
containerStarted()
method instead of resetting every node heartbeat. Let me know what you think.

bq. I'm not sure why we need a separate launchedOnNode flag because we already 
have a launchedContainer map.
This is indeed confusing. The launchedContainer should probably be renamed to 
allocatedContainer and launchedOnNode is to track whether the allocated
container is actually launched on the node. This piece code already exists. I 
can do the renaming if you are fine with it.

bq.  otherwise it gonna be very hard to modify defined protos in a future 
release.
Very much for the same reason you are thinking of here, I am more inclined to 
keep OverAllocationInfo for now. I am not sure if we just have 
ResourceThresholds,
how we can keep backward compatibility in a clean way if we ever want to 
include more for NM overallocation configs. I agree we should do the 
consolidating
with resource profiles before the release, I think we can revisit this topic 
then.


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201275#comment-16201275
 ] 

Wangda Tan commented on YARN-4511:
--

Thanks Haibo, applogize for my late responses, was busy with other tasks.

Regarding to: {{allocationInThisHeartbeat}} discussion. The related JIRA is 
YARN-5139, in short, which makes scheduler allocation to two separate phases:
Phase #1, Scheduler look at existing scheduler states (queue/node/app, etc.) 
and make allocation proposal (on which node, allocate container). This could be 
done in multiple threads.
Phase #2, There's another thread (now is single thread), look at allocation 
proposal and try to accept/reject them. 
Under the context of YARN-5139, we cannot assume an allocation proposal will be 
accepted. I'm not sure how this impact your approach.

To your proposal:
bq. we'd do allocation of guaranteed containers first followed by opportunistic 
containers. W need to consider the just-allocated-yet-to-launch guaranteed 
containers to project how much resource we have left to allocate opportunistic 
containers.
I'm still not quite sure about how it works: just-allocated-yet-to-launch 
guaranteed containers could be allocated in different heartbeats, correct? It 
is possible that AM acquires an guaranteed container and wait for serveral 
minutes to launch it, I'm not sure if recording total allocated in a single 
node update event is enough. 

bq. I only try to preserve the containerLaunched flag. Can you be more specific 
about what you're referring to in the patch?
I'm talking about below method in SchedulerNode: (it seems renamed in the 
latest patch)
{code}
/**
   * Inform the node that a container has launched.
   * @param containerId ID of the launched container
   */
  public synchronized void containerStarted(ContainerId containerId) {
ContainerInfo info = launchedContainers.get(containerId);
if (info != null) {
  info.launchedOnNode = true;
}
  }
{code}
I'm not sure why we need a separate launchedOnNode flag because we already have 
a launchedContainer map.

bq. There is a jira open to consolidate with Resource Profiles (YARN-6690). Is 
that a good place to do the work to accommodate other resources?
I'm fine with moving this to a separate JIRA, but we need to do this before 
release, otherwise it gonna be very hard to modify defined protos in a future 
release. 

I'm not sure if I asks too much: could you include a summary of workflow of 
this patch and how schedulers will use them. I found there're lots of changes 
(especially inside SchedulerNode) but I cannot see the full picture of how 
scheduler will use them. A workflow can help reviews a lot. 


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198915#comment-16198915
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks a lot for the review, [~asuresh]!

bq. in the context of container updates, generally, a container stays on the 
same node - so was surprised that you added a newNode and oldNode arguments to 
the method 
I was not sure about whether container update is always node-local, hence I 
added a newNode and another oldNode argument. Will remove one in the follow up 
patch. The change to swapContainer(), IIRC, is necessary to fix some of the 
unit tests that the rest of the patch causes. The revelant change in the patch 
is that SchedulerNode is now modified to distinguish between Opportunistic and 
Guaranteed containers. Since swapContainer only swaps the internal container 
objects rather than RMContainers which SchedulerNod gets notified of, we need 
to update the resource accounting explicitly on the node every time there is a 
container swap. Otherwise, the incorrect resource accounting would cause 
existing unit tests to fail. I hope it makes sense to you.



> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198077#comment-16198077
 ] 

Arun Suresh commented on YARN-4511:
---

[~haibochen], took a quick look at the patch.
With regard to the changes to the {{swapContainer}} method, the original 
intent, was as you guessed to swap the internal container objects of two 
RMContainers. Generally, in the context of container updates, the new container 
is killed off immediately after the swap. We assume this swap happens somewhat 
transitionally (given the locking order etc.), and allows us to side-step 
having to deal with intermediate events the original RMContainer might have to 
deal with during the update. Also, in the context of container updates, 
generally, the a container stays on the same node - so was surprised that you 
added a newNode and oldNode arguments to the method, since I felt it does not 
make sense. Maybe if you could point me to how you are going to ultimately use 
the changes - I might have more context ?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197989#comment-16197989
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m  
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m 
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
40s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 
40s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
26s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
21s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 12s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 22s{color} | {color:orange} root: The patch generated 4 new + 565 unchanged 
- 2 fixed = 569 total (was 567) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 17s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
45s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
43s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 2 new + 297 unchanged - 0 fixed = 299 total (was 297) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 51m 11s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 57s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
50s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 36s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.allocateContainer(RMContainer,
 boolean)  At SchedulerNode.java:[line 182] |
|  

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197757#comment-16197757
 ] 

Haibo Chen commented on YARN-4511:
--

Uploaded a new patch that based on Wangda's review
1) removes allowedResourceForOverAllocation() from SchedulerNode 
2) passes SchedulerNode from SchedulerAppAttempt to 
ContainerUpdateContext.swapContainers() in the parameter to avoid plumbing 
context.
3) consolidates two separate maps of  into one
 
[~asuresh], can you give some insight into the swapContainer change and a look 
at the patch ?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197368#comment-16197368
 ] 

Haibo Chen commented on YARN-4511:
--

Forget to mention why allowedResourceForOverAllocation() is there even though 
it is not used by anyone.
This can be technically included in the Fair Scheduler Oversubscription patch, 
but I though capacity scheduler will probably use it as well. hence, it is 
included in this patch.
Let me remove it from the this patch and include it in the following fair 
scheduler one. You guys can always decide if you can reuse it in capacity 
scheduler.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193897#comment-16193897
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks for the reviews, [~leftnoteasy]!

The scope of the variable allocationInThisHeartbeat is supposed to be a round 
of allocation on a given node. In the case of fair scheduler, it is every 
nodeUpdate() method call. Can you point out where else do we also do allocation 
that is decoupled from heartbeat? Here is what this variable is used for in the 
case of fair scheduler: every time there is a node update, we'd do allocation 
of guaranteed containers first followed by opportunistic containers. W need to 
consider the just-allocated-yet-to-launch guaranteed containers to project how 
much resource we have left to allocate opportunistic containers. Any guidance 
on how to handle allocation decouple from heartbeat is appreciated.  

bq. In what case the oldNode and newNode will be null and should we throw 
exception when this happens?
This is more to avoid NPE defensively. I guess you are right, we should 
probably throw exception since it indicates errors and will lead to problems if 
we update one but not the other.

bq. This part of logic looks confusing, since old containers will be finished 
inside pullNewlyUpdatedContainers, do we really need this method? 
This arises from the fact that RMContainer events trigger update of 
schedulerNode, whereas the current implementation of container 
incr/decr/promotion/demotion only swaps the underly Container instance for 
RMContainers, which does not trigger schedulerNode updates, so the resulting 
accounting on that node is incorrect. [~asuresh] any suggestion to simply this?

bq. Why containerLaunched is added? Should we just increase allocated 
opportunistic/guaranteed resource?
I only try to preserve the containerLaunched flag. Can you be more specific 
about what you're referring to in the patch?

bq. I think one of the ResourceThresholds and OverAllocationInfo should be 
removed, they're kind of duplicated. We should try to reduce unnecessary 
#PB-records.
Good point. The initial intension was to include more than resourceThresholds 
in OverAllocationInfo, such as per node max allocation in terms of a 
percentage. We may be able to get rid of OverAllocationInfo down the road if 
there is no such need any more.

There is a jira open to consolidate with Resource Profiles (YARN-6690). Is that 
a good place to do the work to accommodate other resources?


> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177390#comment-16177390
 ] 

Wangda Tan commented on YARN-4511:
--

Thanks [~haibo.chen], 

1) SchedulerNode#resetAllocationThisHeartbeat:
I'm not sure why we do this, since allowedResourceForOverAllocation is not used 
by anyone. Secondly, since allocation can be decoupled from heartbeat, this 
looks confusing as well. Could you add more contexts here?

2) Changes to pass RMContext to ContainerUpdateContext: can we get 
SchedulerNodes insider SchedulerAppAttempt#pullNewlyUpdatedContainers and pass 
to swapContainer?

3) ContainerUpdateContext#swapContainer:
- In what case the oldNode and newNode will be null and should we throw 
exception when this happens? Existing logic could cause nodeUpdate called on 
one node but not on the other.
- And related: {{SchedulerNode#containerUpdated}} This part of logic looks 
confusing, since old containers will be finished inside 
{{pullNewlyUpdatedContainers}}, do we really need this method? I would like to 
see thoughts from [~asuresh] for this part as well.

4) SchedulerNode:
- Instead of having two separate Map, could we just 
leave one? It makes logics such as {{getContainer(ContainerId)}} simpler as 
well. We can get container's executionType from RMContainer in any case. 
- Why {{containerLaunched}} is added? Should we just increase allocated 
opportunistic/guaranteed resource?

Not related to this patch but also important: 
1) I think one of the ResourceThresholds and OverAllocationInfo should be 
removed, they're kind of duplicated. We should try to reduce unnecessary 
#PB-records.
2) Should we consider all resource types for configurations / internal 
calculation? My expectation is, if we want to add different resource 
overallocation like disk resource, we don't have to change all the places. So 
probably it's better to convert configs / fields from individual resource types 
to vector and avoid logics like: 
{code}
ResourceThresholds thresholds =
overAllocationInfo.getOverAllocationThresholds();
Resource overAllocationThreshold = Resources.createResource(
(long) (capacity.getMemorySize() * thresholds.getMemoryThreshold()),
(int) (capacity.getVirtualCores() * thresholds.getCpuThreshold())); 
{code}

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177369#comment-16177369
 ] 

Arun Suresh commented on YARN-4511:
---

Would like to take a closer look at this at well - should be able to give it a 
rev over the weekend or early next week.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177367#comment-16177367
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  2m  
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  6m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
39s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
35s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
15s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
51s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 12s{color} | {color:orange} root: The patch generated 1 new + 565 unchanged 
- 2 fixed = 566 total (was 567) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m  5s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 12s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
37s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m  5s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
|   | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-4511 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12888581/YARN-4511-YARN-1011.02.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f9b26e09ccf9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-1011 / 6d9500c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/17601/artifact/patchprocess/diff-checkstyle-root.txt
 |
| unit | 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177333#comment-16177333
 ] 

Haibo Chen commented on YARN-4511:
--

Thanks [~leftnoteasy] for reviewing. I will be on vacations for the next a few 
days, so will probably address your comments late next week.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177285#comment-16177285
 ] 

Wangda Tan commented on YARN-4511:
--

[~haibo.chen], thanks for working on the patch, I'm reviewing the patch now and 
should be able to finish it by end of today, could you wait several hours 
before committing the patch?

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166994#comment-16166994
 ] 

Haibo Chen commented on YARN-4511:
--

TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable is 
unrelated, and being tracked at YARN-7044

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166957#comment-16166957
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m 
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  4m 
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
55s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
12s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 8s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
10s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
49s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 12s{color} | {color:orange} root: The patch generated 13 new + 612 unchanged 
- 0 fixed = 625 total (was 612) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
33s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 1 new + 348 unchanged - 0 fixed = 349 total (was 348) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 18s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
59s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-4511 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12887156/YARN-4511-YARN-1011.01.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 9f27160c6c98 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | YARN-1011 / 843c993 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/17459/artifact/patchprocess/diff-checkstyle-root.txt
 |
| javadoc | 

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144371#comment-16144371
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  5m  
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
46s{color} | {color:green} YARN-1011 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  4m 
57s{color} | {color:red} root in YARN-1011 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
43s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 23s{color} 
| {color:red} root generated 539 new + 778 unchanged - 0 fixed = 1317 total 
(was 778) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 48s{color} | {color:orange} root: The patch generated 11 new + 559 unchanged 
- 0 fixed = 570 total (was 559) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
12s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
28s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 1 new + 348 unchanged - 0 fixed = 349 total (was 348) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 39s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 51s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.guaranteedContainerAllocated(RMContainer,
 boolean)  At SchedulerNode.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.guaranteedContainerAllocated(RMContainer,
 boolean)  At SchedulerNode.java:[line 207] |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.guaranteedContainerReleased(RMContainer,
 boolean)  At SchedulerNode.java:in