[
https://issues.apache.org/jira/browse/HBASE-29933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-29933:
-----------------------------------
Labels: pull-request-available (was: )
> update_all_config hangs indefinitely when balancing event is in progress
> ------------------------------------------------------------------------
>
> Key: HBASE-29933
> URL: https://issues.apache.org/jira/browse/HBASE-29933
> Project: HBase
> Issue Type: Bug
> Reporter: Dev Hingu
> Assignee: Dev Hingu
> Priority: Major
> Labels: pull-request-available
>
> update_all_config command hangs indefinitely if HMaster.balance() event is
> going on.
> When HMaster.balance() is running it acquires lock on CacheAwareLoadBalancer
> and HMaster thread goes to sleep due to throttling in CacheAwareLoadBalancer.
> Now, CacheAwareLoadBalancer.onConfigurationChange() waits to acquire the same
> lock
> Attaching stack traces for both thread
> 1. HMaster Thread :
> {code:java}
> #355 daemon prio=5 os_prio=0 cpu=2014.35ms elapsed=19554.89s
> tid=0x00007f3fd018b310 nid=0x5fda waiting on condition [0x00007f3f6a3f1000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep([email protected]/Native Method)
> at
> org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer.throttle(CacheAwareLoadBalancer.java:197)
> at
> org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:2164)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:2122)
> - locked <0x000000070b7d1438> (a
> org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1998)
> at
> org.apache.hadoop.hbase.master.HMaster.balanceOrUpdateMetrics(HMaster.java:2010)
> - locked <0x000000070b7d1438> (a
> org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:47)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161){code}
> 2. update configuration RPC thread :
> {code:java}
> #96 daemon prio=5 os_prio=0 cpu=523.03ms elapsed=19854.17s
> tid=0x00007f3ffbb47ed0 nid=0x48a4 waiting for monitor entry
> [0x00007f3f785fe000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.onConfigurationChange(BaseLoadBalancer.java:785)
>
> - waiting to lock <0x000000070b7d1438> (a
> org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
> at
> org.apache.hadoop.hbase.conf.ConfigurationManager.notifyAllObservers(ConfigurationManager.java:110)
>
> - locked <0x000000070c9e2440> (a java.util.Collections$SetFromMap)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.updateConfiguration(HRegionServer.java:3927)
>
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.updateConfiguration(RSRpcServices.java:3902)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
