[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271186#comment-17271186 ] zhuqi commented on YARN-10531: -- Thanks [~wangda] for final review. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch, > YARN-10531.006.patch, YARN-10531.007.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269469#comment-17269469 ] Wangda Tan commented on YARN-10531: --- +1, thanks [~zhuqi], will get it in later today. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch, > YARN-10531.006.patch, YARN-10531.007.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269053#comment-17269053 ] Hadoop QA commented on YARN-10531: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 30s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 12s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 41s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 55s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 52s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 7s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green}{color} | {c
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269052#comment-17269052 ] Hadoop QA commented on YARN-10531: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 40s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 58s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 54s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 47s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 44s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 59s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green}{color} | {col
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269009#comment-17269009 ] zhuqi commented on YARN-10531: -- [~wangda] I have fixed above, in latest patch. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch, > YARN-10531.006.patch, YARN-10531.007.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268755#comment-17268755 ] Wangda Tan commented on YARN-10531: --- Thanks [~zhuqi], Two minor comments: 1) ParentQueue.java: - We can remove: FIXME: Ideally we should disable user limit factor, see YARN-10531 2) AbstractCSQueue: Nit: Let's breakdown {code} 1542 int maxApplicationsPerUser = 1543 leafQueue.getUsersManager().getUserLimitFactor() != -1 1544 ? Math.min(maxApplications, 1545 (int) (maxApplications 1546 * (leafQueue.getUsersManager().getUserLimit() / 100.0f) 1547 * leafQueue.getUsersManager().getUserLimitFactor())) 1548 : maxApplications; {code} Into multiple statements for better readability. Thoughts? [~sunilg], [~shuzirra], [~snemeth], [~pbacsko] > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268457#comment-17268457 ] zhuqi commented on YARN-10531: -- cc [~wangda] [~sunilg] If you any other advice for this? Thanks. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267960#comment-17267960 ] Hadoop QA commented on YARN-10531: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 7s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 42s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 47s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 23s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green}{color} | {col
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267852#comment-17267852 ] Hadoop QA commented on YARN-10531: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 2s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 22s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 46s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 39s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/516/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 556 unchanged - 0 fixed = 557 total (was 556) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 45s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {co
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267771#comment-17267771 ] zhuqi commented on YARN-10531: -- [~BilwaST] Thanks a lot for your review, fix checkstyle in latest patch. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch, YARN-10531.004.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267763#comment-17267763 ] Bilwa S T commented on YARN-10531: -- Hi [~zhuqi] Thanks for the patch. patch looks good to me. +1(Non-binding). Please check findbug and checkstyle issues > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266853#comment-17266853 ] Hadoop QA commented on YARN-10531: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 12s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 54s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 45s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/497/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 1 extant findbugs warnings. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 44s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/497/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 555 unchanged - 1 fixed = 557 total (was 556) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{colo
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266799#comment-17266799 ] zhuqi commented on YARN-10531: -- [~wangda] I have rebased it, i have also added a new auto created based test in latest patch. If you any other advice? Thanks. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch, > YARN-10531.003.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266294#comment-17266294 ] Wangda Tan commented on YARN-10531: --- [~zhuqi], I haven't reviewed details of the patch yet, I'm not sure if we need rebase after YARN-10506. Please share your thoughts so we can move it forward. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248823#comment-17248823 ] zhuqi commented on YARN-10531: -- [~wangda] Attach a latest patch, to make unit test more clear. And change the logic: {code:java} // If user-limit-factor set to -1, we should disabled user limit. if (getUserLimitFactor() != -1) { maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, getUserLimitFactor()); } else { maxUserLimit = lQueue. getEffectiveMaxCapacityDown(nodePartition, lQueue.getMinimumAllocation()); } {code} Other changes is related to maxPerUserApplications and so on. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch, YARN-10531.002.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248318#comment-17248318 ] Hadoop QA commented on YARN-10531: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m 43s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 12s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 45s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 45s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 43s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/383/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 13 new + 486 unchanged - 0 fixed = 499 total (was 486) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 43s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {c
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248297#comment-17248297 ] zhuqi commented on YARN-10531: -- [~wangda] [~pbacsko] [~tangzhankun] I submit a patch to disable user limit factor for capacityScheduler leaf queue. If you can review it. Thanks. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10531.001.patch > > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248260#comment-17248260 ] zhuqi commented on YARN-10531: -- [~wangda] I'm glad to take this. Thanks. > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Priority: Major > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[ https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248149#comment-17248149 ] Wangda Tan commented on YARN-10531: --- [~zhuqi] do you want to take a try on this one? Thanks, > Be able to disable user limit factor for CapacityScheduler Leaf Queue > - > > Key: YARN-10531 > URL: https://issues.apache.org/jira/browse/YARN-10531 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Priority: Major > > User limit factor is used to define max cap of how much resource can be > consumed by single user. > Under Auto Queue Creation context, it doesn't make much sense to set user > limit factor, because initially every queue will set weight to 1.0, we want > user can consume more resource if possible. It is hard to pre-determine how > to set up user limit factor. So it makes more sense to add a new value (like > -1) to indicate we will disable user limit factor > Logic need to be changed is below: > (Inside LeafQueue.java) > {code} > Resource maxUserLimit = Resources.none(); > if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) { > maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity, > getUserLimitFactor()); > } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) > { > maxUserLimit = partitionResource; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org