[
https://issues.apache.org/jira/browse/PHOENIX-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340344#comment-15340344
]
Hudson commented on PHOENIX-2940:
---------------------------------
FAILURE: Integrated in Phoenix-master #1272 (See
[https://builds.apache.org/job/Phoenix-master/1272/])
PHOENIX-2940 Query the stats table and cache stats in the client (elserj: rev
e53e3bbb660769cde11d77ed557b8d305926edf3)
* phoenix-core/src/test/java/org/apache/phoenix/util/TestUtil.java
*
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java
*
phoenix-core/src/test/java/org/apache/phoenix/query/PhoenixStatsCacheRemovalListenerTest.java
*
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java
* phoenix-core/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java
*
phoenix-core/src/test/java/org/apache/phoenix/filter/SkipScanBigFilterTest.java
* phoenix-core/src/main/java/org/apache/phoenix/schema/PTable.java
* phoenix-core/src/main/java/org/apache/phoenix/execute/ScanPlan.java
* phoenix-core/src/main/java/org/apache/phoenix/query/QueryServices.java
* phoenix-core/src/main/java/org/apache/phoenix/query/TableStatsCache.java
* phoenix-core/src/main/java/org/apache/phoenix/schema/DelegateTable.java
*
phoenix-core/src/main/java/org/apache/phoenix/query/DelegateConnectionQueryServices.java
*
phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServices.java
* phoenix-core/src/main/java/org/apache/phoenix/schema/PTableImpl.java
* phoenix-core/src/it/java/org/apache/phoenix/rpc/PhoenixServerRpcIT.java
* phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java
* phoenix-protocol/src/main/PTable.proto
* phoenix-core/src/main/java/org/apache/phoenix/schema/MetaDataClient.java
*
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
> Remove STATS RPCs from rowlock
> ------------------------------
>
> Key: PHOENIX-2940
> URL: https://issues.apache.org/jira/browse/PHOENIX-2940
> Project: Phoenix
> Issue Type: Improvement
> Environment: HDP 2.3 + Apache Phoenix 4.6.0
> Reporter: Nick Dimiduk
> Assignee: Josh Elser
> Fix For: 4.8.0
>
> Attachments: PHOENIX-2940.001.patch, PHOENIX-2940.002.patch,
> PHOENIX-2940.003.patch, PHOENIX-2940.004.patch
>
>
> We have an unfortunate situation wherein we potentially execute many RPCs
> while holding a row lock. This is problem is discussed in detail on the user
> list thread ["Write path blocked by MetaDataEndpoint acquiring region
> lock"|http://search-hadoop.com/m/9UY0h2qRaBt6Tnaz1&subj=Write+path+blocked+by+MetaDataEndpoint+acquiring+region+lock].
> During some situations, the
> [MetaDataEndpoint|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L492]
> coprocessor will attempt to refresh it's view of the schema definitions and
> statistics. This involves [taking a
> rowlock|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L2862],
> executing a scan against the [local
> region|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L542],
> and then a scan against a [potentially
> remote|https://github.com/apache/phoenix/blob/10909ae502095bac775d98e6d92288c5cad9b9a6/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L964]
> statistics table.
> This issue is apparently exacerbated by the use of user-provided timestamps
> (in my case, the use of the ROW_TIMESTAMP feature, or perhaps as in
> PHOENIX-2607). When combined with other issues (PHOENIX-2939), we end up with
> total gridlock in our handler threads -- everyone queued behind the rowlock,
> scanning and rescanning SYSTEM.STATS. Because this happens in the
> MetaDataEndpoint, the means by which all clients refresh their knowledge of
> schema, gridlock in that RS can effectively stop all forward progress on the
> cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
