Tanuj Khurana created PHOENIX-8023:
--------------------------------------
Summary: SYSTEM.CATALOG region unavailable for entire scan
duration when closing
Key: PHOENIX-8023
URL: https://issues.apache.org/jira/browse/PHOENIX-8023
Project: Phoenix
Issue Type: Bug
Reporter: Tanuj Khurana
_MetaDataEndpointImpl.buildTable_ holds the SYSTEM.CATALOG region operation
(read lock)
while building a PTable, and on that path the TTL resolution helpers
(getTTLFromHierarchy / getTTLForParentTable) introduced as part of PHOENIX-6978
open a HBase client scan of SYSTEM.CATALOG itself via
{_}ServerUtil.getHTableForCoprocessorScan(env, SYSTEM.CATALOG){_}. Once a close
sets the
closing flag, those nested scans get '{*}NotServingRegionException ... is
closing{*}' and the HBase client *retries for the full 60 s scanner timeout*
while the outer handler still holds the region read lock. The close cannot take
the write lock until they give up and the scan operation times out.
{code:java}
java.net.SocketTimeoutException: callTimeout=60000, callDuration=68468:
SYSTEM.CATALOG,,1787178855993.695f91ca... is closing ...
hostname=regionserver-sys-2...
at
o.a.h.h.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(...:294)
at o.a.h.h.client.ScannerCallableWithReplicas.call(...:256)
at o.a.h.h.client.ClientScanner.next(ClientScanner.java:623)
at
o.a.p.coprocessor.MetaDataEndpointImpl.getTTLFromHierarchy(MetaDataEndpointImpl.java:1837)
at
o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1571)
at
o.a.p.coprocessor.MetaDataEndpointImpl.getTableFromCells(MetaDataEndpointImpl.java:1154)
at
o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1149)
at
o.a.p.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:809)
at
o.a.p.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:4085)
at
o.a.p.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:728)
at ...MetaDataProtos$MetaDataService.callMethod(...)
at o.a.h.h.regionserver.HRegion.execService(HRegion.java:8011)
...
at o.a.h.h.ipc.RpcHandler.run(RpcHandler.java:82)
Caused by: java.net.SocketTimeoutException: callTimeout=60000,
callDuration=68468: ... is closing
at
o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157)
at o.a.h.h.client.ResultBoundedCompletionService$QueueingFuture.run(...:74)
Caused by: org.apache.hadoop.hbase.NotServingRegionException:
SYSTEM.CATALOG,... is closing
at o.a.h.h.regionserver.HRegion.startRegionOperation(HRegion.java:8130)
at o.a.h.h.regionserver.HRegion.getScanner(HRegion.java:3059)
at
o.a.h.h.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:3205)
at o.a.h.h.regionserver.RSRpcServices.scan(RSRpcServices.java:3578)
at
o.a.h.h.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.scan(...:149)
at o.a.h.h.client.ScannerCallable.openScanner(ScannerCallable.java:363)
...
at
o.a.h.h.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:103){code}
This happens twice because on a graceful shutdown syscat region first moves
from the original regionserver to another regionserver and then again moves
back to the original regionserver.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
