Enis Soztutar created PHOENIX-3072:
--------------------------------------
Summary: Deadlock on region opening with secondary index recovery
Key: PHOENIX-3072
URL: https://issues.apache.org/jira/browse/PHOENIX-3072
Project: Phoenix
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 4.9.0
There is a distributed deadlock happening in clusters with some moderate number
of regions for the data tables and secondary index tables and cluster and it is
cluster restart or some large failure. We have seen this in a couple of
production cases already.
Opening of regions in hbase is performed by a thread pool with 3 threads by
default. Every regionserver can open 3 regions at a time. However, opening data
table regions has to write to multiple index regions during WAL recovery. All
other region open requests are queued up in a single queue. This causes a
deadlock, since the secondary index regions are also opened by the same thread
pools that we do the work. So if there is greater number of data table regions
then available number of region opening threads from regionservers, the
secondary index region open requests just wait to be processed in the queue.
Since these index regions are not open, the region opening of data table
regions just block the region opening threads for a long time.
One proposed fix is to use a different thread pool for opening regions of the
secondary index tables so that we will not deadlock. See HBASE-16095 for the
HBase-level fix. In Phoenix, we just have to set the priority for secondary
index tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
