[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805485#comment-15805485 ] Arun Suresh commented on YARN-6041: --- Cherry-picking the JIRAs mentioned in the description to branch-2 shortly. Raised YARN-6066 to commit the changes suggested by [~kasha] and [~leftnoteasy] > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802796#comment-15802796 ] Karthik Kambatla commented on YARN-6041: The changes to config names and methods look good to me. > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802732#comment-15802732 ] Wangda Tan commented on YARN-6041: -- +1 to latest patch, [~asuresh] please wait another day before cherry-picking it to see is there any other comments. > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802212#comment-15802212 ] Wangda Tan commented on YARN-6041: -- [~asuresh], so it will generate 10 commits (plus one for suggestions from this JIRA), correct? It will be better to create a separate JIRA to track kasha's suggestions and commit it separately (so we will have a JIRA number) > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802195#comment-15802195 ] Arun Suresh commented on YARN-6041: --- [~leftnoteasy], So what I plan to do was, like you mentioned, cherry-pick the 10 JIRAs specified in the description (I actually created the patch by doing just that.. and then doing a "git diff ").. Then commit JUST the changes Karthik suggested as "YARN-6041: .." which I will cherry-pick on to trunk as well. > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802169#comment-15802169 ] Wangda Tan commented on YARN-6041: -- [~asuresh], How you plan to merge these changes to branch-2? I think it might be better to do cherry-pick one-by-one, and file a separate JIRA to address new comments from [~kasha] and me. Commit this huge patch to branch-2 creates troubles for future maintenance. Thoughts? > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801509#comment-15801509 ] Arun Suresh commented on YARN-6041: --- As mentioned earlier, the javac, javadoc and whitespace errors are better left unfixed to retain the style of the existing files. The Testcase errors are not related. [~kasha] / [~leftnoteasy], do let me know if this is good for checkin > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801234#comment-15801234
]
Hadoop QA commented on YARN-6041:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 40 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
28s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
30s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
8s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
8s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
54s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
23s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
27s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 27s{color}
| {color:red} root-jdk1.8.0_111 with JDK v1.8.0_111 generated 1 new + 860
unchanged - 1 fixed = 861 total (was 861) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 27s{color} | {color:orange} root: The patch generated 19 new + 3469
unchanged - 43 fixed = 3488 total (was 3512) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796705#comment-15796705 ] Arun Suresh commented on YARN-6041: --- Thanks for taking a look [~kasha], bq. Assuming MRJobConfig. MR_NUM_OPPORTUNISTIC_MAPS_PER_100 was not part of any release. Can you confirm the same? I confirm this hasn't seen light of day in any previous release :) bq. Assuming the incompatible proto changes to yarn_server_common_protos and yarn_server_common_service_protos are okay because the previous versions didn't make it to a release. Can you confirm the same? Ditto.. bq. Nit: For scheduler-related configs, should we use yarn.scheduler.* like other scheduler configs? And, yarn.nodemanager.* for all node-specific opportunistic container configs? Making sure the configs are consistent and simple is a good exercise before we include these in a release. Hmm... I guess you are referring to YARN-5646. Looks like the patch was modified to incorporate your suggestions wrt the parameter names. Also, since this doesn't affect the scheduler per se.. the RM side changes are all yarn.resourcemanager.* and the nodemanager ones are yarn.nodemanager.* Good catch on the method names... will update patch shortly > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796676#comment-15796676 ] Karthik Kambatla commented on YARN-6041: Thanks for putting this up on JIRA, [~asuresh]. I skimmed through the patch, paying attention to compatibility and user-facing configs. The patch looks mostly good. # Assuming MRJobConfig. MR_NUM_OPPORTUNISTIC_MAPS_PER_100 was not part of any release. Can you confirm the same? # Should the corresponding methods in RMContainerAllocator be also renamed to use percentage instead of per_100? Also, should the config name be PERCENT instead of PERCENTAGE to be consistent with other configs in MR? # Nit: For scheduler-related configs, should we use yarn.scheduler.* like other scheduler configs? And, yarn.nodemanager.* for all node-specific opportunistic container configs? Making sure the configs are consistent and simple is a good exercise before we include these in a release. # Assuming the incompatible proto changes to yarn_server_common_protos and yarn_server_common_service_protos are okay because the previous versions didn't make it to a release. Can you confirm the same? > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15789079#comment-15789079
]
Hadoop QA commented on YARN-6041:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 40 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
17s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
39s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
55s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
39s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
8s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
28s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 28s{color}
| {color:red} root-jdk1.8.0_111 with JDK v1.8.0_111 generated 1 new + 860
unchanged - 1 fixed = 861 total (was 861) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 19s{color} | {color:orange} root: The patch generated 18 new + 3470
unchanged - 42 fixed = 3488 total (was 3512) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787934#comment-15787934
]
Arun Suresh commented on YARN-6041:
---
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
> Opportunistic containers : Combined patch for branch-2
> ---
>
> Key: YARN-6041
> URL: https://issues.apache.org/jira/browse/YARN-6041
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6041-branch-2.001.patch
>
>
> This is a combined patch targeting branch-2 of the following JIRAs which have
> already been committed to trunk :
> YARN-5938. Refactoring OpportunisticContainerAllocator to use
> SchedulerRequestKey instead of Priority and other misc fixes
> YARN-5646. Add documentation and update config parameter names for scheduling
> of OPPORTUNISTIC containers.
> YARN-5982. Simplify opportunistic container parameters and metrics.
> YARN-5918. Handle Opportunistic scheduling allocate request failure when NM
> is lost.
> YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager
> container lifecycle.
> YARN-5823. Update NMTokens in case of requests with only opportunistic
> containers.
> YARN-5377. Fix
> TestQueuingContainerManager.testKillMultipleOpportunisticContainers.
> YARN-2995. Enhance UI to show cluster resource utilization of various
> container Execution types.
> YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http
> Address.
> YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method
> to handle OPPORTUNISTIC container requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787181#comment-15787181
]
Hadoop QA commented on YARN-6041:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
50s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 40 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
44s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
34s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
52s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
25s{color} | {color:green} branch-2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
59s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
28s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 28s{color}
| {color:red} root-jdk1.8.0_111 with JDK v1.8.0_111 generated 1 new + 860
unchanged - 1 fixed = 861 total (was 861) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 20s{color} | {color:orange} root: The patch generated 19 new + 3469
unchanged - 42 fixed = 3488 total (was 3511) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} |
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786416#comment-15786416 ] Wangda Tan commented on YARN-6041: -- Removed 2.9.0 from fixVersion, it should be target version. Just do a very quick scan of the patch, it looks changes are isolated to distributed scheduling logic inside RM, should be relatively safe. But I hope to get someone more familiar with the overall opportunistic scheduling logic to make sure it has less impact to existing logics (especially NM). ([~kasha]?) Two minor comments for compatibility: 1) ContainerState.java: Mark SCHEDULED to unstable? 2) All new configurations to YarnConfigration.java should be @Unstable. And not sure if there's incomplete changes to public (regardless of stablebility) APIs / configs, we should not merge any API-only changes to branch-2. > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
