[jira] [Assigned] (HBASE-25460) Expose drainingServers as cluster metric
[ https://issues.apache.org/jira/browse/HBASE-25460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Kumar reassigned HBASE-25460: --- Assignee: Rahul Kumar (was: Abhey Rana) > Expose drainingServers as cluster metric > > > Key: HBASE-25460 > URL: https://issues.apache.org/jira/browse/HBASE-25460 > Project: HBase > Issue Type: New Feature > Components: metrics >Affects Versions: 1.6.0, 2.4.0 >Reporter: Viraj Jasani >Assignee: Rahul Kumar >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > > Due to some reason, we had significantly high number of servers put in > decommissioned mode and for significant time, they continued being in the > same state serving no regions at all. This put heavy load on rest of live > servers and it was too late before one could recognize the issues with > improper balancing of the cluster. And as expected, balancing such cluster > with/without *runMaxSteps* can bring up sudden spike of RITs in proportion to > the degree of imbalanced regions in the cluster. > Although running into such situation is rare, we can take some precautions by > exposing metric. We should expose list of draining RegionServers as jmx > metrics just like we expose _*liveRegionServers*_ and _*deadRegionServers*_. > Such metric can help configure alerts with threshold on % of total RS that > are allowed to go in draining mode (e.g during rolling upgrades) in any > circumstances. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25460) Expose drainingServers as cluster metric
[ https://issues.apache.org/jira/browse/HBASE-25460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhey Rana reassigned HBASE-25460: -- Assignee: Abhey Rana > Expose drainingServers as cluster metric > > > Key: HBASE-25460 > URL: https://issues.apache.org/jira/browse/HBASE-25460 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.6.0, 2.4.0 >Reporter: Viraj Jasani >Assignee: Abhey Rana >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > > Due to some reason, we had significantly high number of servers put in > decommissioned mode and for significant time, they continued being in the > same state serving no regions at all. This put heavy load on rest of live > servers and it was too late before one could recognize the issues with > improper balancing of the cluster. And as expected, balancing such cluster > with/without *runMaxSteps* can bring up sudden spike of RITs in proportion to > the degree of imbalanced regions in the cluster. > Although running into such situation is rare, we can take some precautions by > exposing metric. We should expose list of draining RegionServers as jmx > metrics just like we expose _*liveRegionServers*_ and _*deadRegionServers*_. > Such metric can help configure alerts with threshold on % of total RS that > are allowed to go in draining mode (e.g during rolling upgrades) in any > circumstances. -- This message was sent by Atlassian Jira (v8.3.4#803005)
