[jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506818#comment-16506818 ] Botong Huang edited comment on YARN-7708 at 6/9/18 5:01 AM: Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: _git apply -p0 v2.patch_, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? was (Author: botong): Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: _git apply -p0 v2.patch_, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch, > YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch, > YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch, > YARN-7708-YARN-7402.04.cumulative.patch, > YARN-7708-YARN-7402.05.cumulative.patch, > YARN-7708-YARN-7402.06.cumulative.patch, > YARN-7708-YARN-7402.07.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506818#comment-16506818 ] Botong Huang edited comment on YARN-7708 at 6/9/18 5:01 AM: Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: _git apply - p0 v2.patch_, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? was (Author: botong): Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: _git apply -p0 v2.patch_, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch, > YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch, > YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch, > YARN-7708-YARN-7402.04.cumulative.patch, > YARN-7708-YARN-7402.05.cumulative.patch, > YARN-7708-YARN-7402.06.cumulative.patch, > YARN-7708-YARN-7402.07.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506818#comment-16506818 ] Botong Huang edited comment on YARN-7708 at 6/9/18 5:00 AM: Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: _git apply -p0 v2.patch_, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? was (Author: botong): Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: git apply -p0 v2.patch, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch, > YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch, > YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch, > YARN-7708-YARN-7402.04.cumulative.patch, > YARN-7708-YARN-7402.05.cumulative.patch, > YARN-7708-YARN-7402.06.cumulative.patch, > YARN-7708-YARN-7402.07.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506818#comment-16506818 ] Botong Huang edited comment on YARN-7708 at 6/9/18 4:59 AM: Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields Also when I apply your patch with: git apply -p0 v2.patch, I got some warning like: ../YARN-7708-YARN-7402.02.patch:240: new blank line at EOF. Can you please fix this as well? was (Author: botong): Thanks [~youchen] for the patch! A few comments in LoadBasedGlobalPolicy: In getTargetWeights, when there's three subclusters and all of their pending is above maxPending, we will get zero weight (default min weight) for all of them? Can you please change *Map* registerPaths() to *Map, String>*, and in all other necessary places? This will remove all the raw type warnings. line 48-69: move these to YarnConfiguration, add some comments about them line 53: default min pending to something smaller, say 100? max pending to say 1000? line 72-76: some comments here as well to explain the fields > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch, > YARN-7708-YARN-7402.01.patch, YARN-7708-YARN-7402.02.cumulative.patch, > YARN-7708-YARN-7402.02.patch, YARN-7708-YARN-7402.03.cumulative.patch, > YARN-7708-YARN-7402.04.cumulative.patch, > YARN-7708-YARN-7402.05.cumulative.patch, > YARN-7708-YARN-7402.06.cumulative.patch, > YARN-7708-YARN-7402.07.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370783#comment-16370783 ] Young Chen edited comment on YARN-7708 at 2/21/18 12:24 AM: The rationale for this policy is to lower load on sub clusters that are receiving an abnormal amount of jobs or are unable to keep up for performance reasons. To alleviate job failures/excessive delays, this policy attempts to redirect load away from highly loaded sub clusters by updating a WeightedLocalityPolicyManager with modified weights. This patch introduces the LoadBasedGlobalPolicy. It's configurable in the following ways: * MAX_EDIT: The maximum number of sub clusters weights this policy will edit. The top N clusters by pending load will have their policy weights scaled down. * MIN_PENDING: The minimum number of pending applications before a sub cluster weight qualifies for editing. * MAX_PENDING: The maximum number of pending applications in the scalable range. Sub clusters exceeding this will have their policy weights set to MIN_WEIGHT * MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster. * SCALING: The scaling method that maps pending load to sub cluster policy weight. Currently there are three scaling methods: quadratic, log, and linear. was (Author: youchen): The rationale for this policy is to lower load on sub clusters that are receiving an abnormal amount of jobs or are unable to keep up for performance reasons. To alleviate job failures/excessive delays, this policy attempts to redirect load away from highly loaded sub clusters by updating a WeightedLocalityPolicyManager with modified weights. This patch introduces the LoadBasedGlobalPolicy. It's configurable in the following ways: * MAX_EDIT: The maximum number of sub clusters this policy will throttle. The top N clusters by pending load will have their policy weights scaled down. * MIN_PENDING: The minimum number of pending applications before a sub cluster weight qualifies for editing. * MAX_PENDING: The maximum number of pending applications in the scalable range. Sub clusters exceeding this will have their policy weights set to MIN_WEIGHT * MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster. * SCALING: The scaling method that maps pending load to sub cluster policy weight. Currently there are three scaling methods: quadratic, log, and linear. > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7708) [GPG] Load based policy generator
[ https://issues.apache.org/jira/browse/YARN-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370783#comment-16370783 ] Young Chen edited comment on YARN-7708 at 2/21/18 12:24 AM: The rationale for this policy is to lower load on sub clusters that are receiving an abnormal amount of jobs or are unable to keep up for performance reasons. To alleviate job failures/excessive delays, this policy attempts to redirect load away from highly loaded sub clusters by updating a WeightedLocalityPolicyManager with modified weights. This patch introduces the LoadBasedGlobalPolicy. It's configurable in the following ways: * MAX_EDIT: The maximum number of sub clusters this policy will throttle. The top N clusters by pending load will have their policy weights scaled down. * MIN_PENDING: The minimum number of pending applications before a sub cluster weight qualifies for editing. * MAX_PENDING: The maximum number of pending applications in the scalable range. Sub clusters exceeding this will have their policy weights set to MIN_WEIGHT * MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster. * SCALING: The scaling method that maps pending load to sub cluster policy weight. Currently there are three scaling methods: quadratic, log, and linear. was (Author: youchen): This patch introduces the LoadBasedGlobalPolicy. It's configurable in the following ways: * MAX_EDIT: The maximum number of sub clusters this policy will throttle. The top N clusters by pending load will have their policy weights scaled down. * MIN_PENDING: The minimum number of pending applications before a sub cluster weight qualifies for editing. * MAX_PENDING: The maximum number of pending applications in the scalable range. Sub clusters exceeding this will have their policy weights set to MIN_WEIGHT * MIN_WEIGHT: The minimum weight possible for a highly loaded sub cluster. * SCALING: The scaling method that maps pending load to sub cluster policy weight. Currently there are three scaling methods: quadratic, log, and linear. > [GPG] Load based policy generator > - > > Key: YARN-7708 > URL: https://issues.apache.org/jira/browse/YARN-7708 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Attachments: YARN-7708-YARN-7402.01.cumulative.patch > > > This policy reads load from the "pendingQueueLength" metrics and provides > scaling into a set of weights that influence the AMRMProxy and Router > behaviors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
