[
https://issues.apache.org/jira/browse/PHOENIX-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tanuj Khurana resolved PHOENIX-8023.
------------------------------------
Fix Version/s: 5.4.0
5.3.3
Resolution: Fixed
> SYSTEM.CATALOG region unavailable for entire scan duration when closing
> -----------------------------------------------------------------------
>
> Key: PHOENIX-8023
> URL: https://issues.apache.org/jira/browse/PHOENIX-8023
> Project: Phoenix
> Issue Type: Bug
> Reporter: Tanuj Khurana
> Priority: Major
> Fix For: 5.4.0, 5.3.3
>
>
> _MetaDataEndpointImpl.buildTable_ holds the SYSTEM.CATALOG region operation
> (read lock)
> while building a PTable, and on that path the TTL resolution helpers
> (getTTLFromHierarchy / getTTLForParentTable) introduced as part of
> PHOENIX-6978 open a HBase client scan of SYSTEM.CATALOG itself via
> {_}ServerUtil.getHTableForCoprocessorScan(env, SYSTEM.CATALOG){_}. Once a
> close sets the
> closing flag, those nested scans get '{*}NotServingRegionException ... is
> closing{*}' and the HBase client *retries for the full 60 s scanner timeout*
> while the outer handler still holds the region read lock. The close cannot
> take the write lock until they give up and the scan operation times out.
>
> {code:java}
> java.net.SocketTimeoutException: callTimeout=60000, callDuration=68468:
> SYSTEM.CATALOG,,1787178855993.695f91ca... is closing ...
> hostname=regionserver-sys-2...
> at
> o.a.h.h.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(...:294)
> at o.a.h.h.client.ScannerCallableWithReplicas.call(...:256)
> at o.a.h.h.client.ClientScanner.next(ClientScanner.java:623)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTTLFromHierarchy(MetaDataEndpointImpl.java:1837)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1571)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1154)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1149)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:809)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:4085)
> at
> o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:728)
> at ...MetaDataProtos$MetaDataService.callMethod(...)
> at o.a.h.h.regionserver.HRegion.execService(HRegion.java:8011)
> ...
> at o.a.h.h.ipc.RpcHandler.run(RpcHandler.java:82)
> Caused by: java.net.SocketTimeoutException: callTimeout=60000,
> callDuration=68468: ... is closing
> at
> o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157)
> at o.a.h.h.client.ResultBoundedCompletionService$QueueingFuture.run(...:74)
> Caused by: org.apache.hadoop.hbase.NotServingRegionException:
> SYSTEM.CATALOG,... is closing
> at o.a.h.h.regionserver.HRegion.startRegionOperation(HRegion.java:8130)
> at o.a.h.h.regionserver.HRegion.getScanner(HRegion.java:3059)
> at
> o.a.h.h.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3205)
> at o.a.h.h.regionserver.RSRpcServices.scan(RSRpcServices.java:3578)
> at
> o.a.h.h.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.scan(...:149)
> at o.a.h.h.client.ScannerCallable.openScanner(ScannerCallable.java:363)
> ...
> at
> o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:103){code}
>
> This happens twice because on a graceful shutdown syscat region first moves
> from the original regionserver to another regionserver and then again moves
> back to the original regionserver.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
