Nick Dimiduk created PHOENIX-2940:
-------------------------------------
Summary: Remove CATALOG,STATS RPCs from rowlock
Key: PHOENIX-2940
URL: https://issues.apache.org/jira/browse/PHOENIX-2940
Project: Phoenix
Issue Type: Improvement
Environment: HDP 2.3 + Apache Phoenix 4.6.0
Reporter: Nick Dimiduk
We have an unfortunate situation wherein we potentially execute many RPCs while
holding a row lock. This is problem is discussed in detail on the user list
thread ["Write path blocked by MetaDataEndpoint acquiring region
lock"|http://search-hadoop.com/m/9UY0h2qRaBt6Tnaz1&subj=Write+path+blocked+by+MetaDataEndpoint+acquiring+region+lock].
During some situations, the
[MetaDataEndpoint|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L492]
coprocessor will attempt to refresh it's view of the schema definitions and
statistics. This involves [taking a
rowlock|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L2862],
executing a scan against the [local
region|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L542],
and then a scan against a [potentially
remote|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L964]
statistics table.
This issue is apparently exacerbated by the use of user-provided timestamps (in
my case, the use of the ROW_TIMESTAMP feature, or perhaps as in PHOENIX-2607).
When combined with other issues (PHOENIX-2939), we end up with total gridlock
in our handler threads -- everyone queued behind the rowlock, scanning and
rescanning SYSTEM.STATS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
