[
https://issues.apache.org/jira/browse/HBASE-27761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705284#comment-17705284
]
Nick Dimiduk commented on HBASE-27761:
--------------------------------------
thread dump before killing the region server thread,
{noformat}
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
CacheReplicationMonitor(986445302)
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-EventThread
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-EventThread
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-SendThread(127.0.0.1:60205)
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-SendThread(127.0.0.1:60205)
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-0
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-0
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-1
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-1
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-2
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-2
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.shipper.scipio%2C56368%2C1679912275932,region_replica_replication
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.shipper.scipio%2C56370%2C1679912275959,region_replica_replication
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.wal-reader.scipio%2C56368%2C1679912275932,region_replica_replication
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.wal-reader.scipio%2C56370%2C1679912275959,region_replica_replication
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REGION_REPLICA_FLUSH_OPS-regionserver/scipio:0-0
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REGION_REPLICA_FLUSH_OPS-regionserver/scipio:0-0
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-0
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-1
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-2
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-3
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-4
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-5
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-6
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56366
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56368
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56370
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56366
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56368
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56370
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56366
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56368
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56370
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56368Replication Statistics #0
2023-03-27T12:17:59,917 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56370Replication Statistics #0
{noformat}
thread listing after killing a region server thread and waiting 30 seconds
{noformat}
2023-03-27T12:18:29,940 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
CacheReplicationMonitor(986445302)
2023-03-27T12:18:29,940 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_CLAIM_REPLICATION_QUEUE-regionserver/scipio:0-0
2023-03-27T12:18:29,940 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-EventThread
2023-03-27T12:18:29,940 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-EventThread
2023-03-27T12:18:29,940 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-SendThread(127.0.0.1:60205)
2023-03-27T12:18:29,940 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-SendThread(127.0.0.1:60205)
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-1
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-1
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-2
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-2
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.shipper.scipio%2C56368%2C1679912275932,region_replica_replication
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.wal-reader.scipio%2C56368%2C1679912275932,region_replica_replication
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.wal-reader.scipio%2C56370%2C1679912275959,region_replica_replication
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REGION_REPLICA_FLUSH_OPS-regionserver/scipio:0-0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REGION_REPLICA_FLUSH_OPS-regionserver/scipio:0-0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-1
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-1
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-2
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-2
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-3
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-3
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-4
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-5
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-6
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56366
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56368
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56366
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56368
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56366
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56368
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56368Replication Statistics #0
2023-03-27T12:18:29,941 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56370Replication Statistics #0
{noformat}
threads after starting a new region server thread,
{noformat}
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
CacheReplicationMonitor(986445302)
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS:2;scipio:56748.replicationSource,region_replica_replication
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_CLAIM_REPLICATION_QUEUE-regionserver/scipio:0-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-EventThread
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-EventThread
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-SendThread(127.0.0.1:60205)
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-SendThread(127.0.0.1:60205)
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-1
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-1
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-2
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication-Writer-2
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.shipper.scipio%2C56368%2C1679912275932,region_replica_replication
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.wal-reader.scipio%2C56368%2C1679912275932,region_replica_replication
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REFRESH_PEER-regionserver/scipio:0-0.replicationSource,region_replica_replication.replicationSource.wal-reader.scipio%2C56370%2C1679912275959,region_replica_replication
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REGION_REPLICA_FLUSH_OPS-regionserver/scipio:0-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RS_REGION_REPLICA_FLUSH_OPS-regionserver/scipio:0-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-0
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-1
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-1
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-2
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-2
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-3
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-3
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-4
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-5
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RegionReplicaReplicationEndpoint-rpc-shared-pool-6
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56366
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56368
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=56748
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56366
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56368
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=56748
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56366
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56368
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=56748
2023-03-27T12:18:30,060 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56368Replication Statistics #0
2023-03-27T12:18:30,061 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56370Replication Statistics #0
2023-03-27T12:18:30,061 INFO [Listener at localhost/56361]
stream.ForEachOps$ForEachOp$OfRef(183): system > main > FailOnTimeoutGroup >
scipio:56748Replication Statistics #0
{noformat}
> Replication threads not attached to their parent process
> --------------------------------------------------------
>
> Key: HBASE-27761
> URL: https://issues.apache.org/jira/browse/HBASE-27761
> Project: HBase
> Issue Type: Task
> Components: read replicas, regionserver, Replication
> Affects Versions: 2.5.4
> Reporter: Nick Dimiduk
> Priority: Major
>
> While debugging HBASE-27707 in a unit test, I see behaviour that I cannot
> explain. My test uses a minicluster, enables read replica replication, writes
> some data, concurrently kills a region server thread hosting a primary
> region, and then verifies that all replicas eventually show all data.
> Inspecting logs, noticed that replication source threads seem to continue
> working even after their associated region server is killed. Interspersing
> some thread dumps and sleeps, I can see that replication threads associated
> with the condemned region server are not being removed after it is killed. I
> think that this behaviour will render unreliably any replication test that
> relies on killing a source or sink region server. It also implies to me that
> the minicluster leaks replication threads and cannot be reliably recycled
> within a single jvm process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
