[ https://issues.apache.org/jira/browse/HBASE-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610262#comment-15610262 ]
Guanghao Zhang commented on HBASE-16947: ---------------------------------------- A example for our cluster. Skip the host:port info. {format} Found 4 deleted queues, run hbck -fixReplication in order to remove the deleted replication queues "host","port",1476784747260/70-"host","port",,1476779010928 "host","port",1476784747260/70-"host","port",1470421093464-"host","port",1476773708939-"host","port",1476779010928 "host","port",1476784763605/70-"host","port",1470418208092-"host","port",1476773709589 "host","port",1476784763605/70-"host","port",1476773709589 Found 2 dead regionservers, restart one regionserver to transfer the queues of dead regionservers "host","port",1476784747260 "host","port",1476784763605 Dumping all peers's number of WALs in replication queue PeerId: 80 , sizeOfLogQueue: 1352 PeerId: 30 , sizeOfLogQueue: 202 PeerId: 20 , sizeOfLogQueue: 202 PeerId: 10 , sizeOfLogQueue: 202 PeerId: 70 , sizeOfLogQueue: 31724 PeerId: 17000 , sizeOfLogQueue: 775 PeerId: 50 , sizeOfLogQueue: 4490 Total size of WALs on HDFS: 61.6G ERROR: There are 120 WALs not found!!! {format} > Some improvements for DumpReplicationQueues tool > ------------------------------------------------ > > Key: HBASE-16947 > URL: https://issues.apache.org/jira/browse/HBASE-16947 > Project: HBase > Issue Type: Improvement > Components: Replication > Affects Versions: 2.0.0, 1.4.0 > Reporter: Guanghao Zhang > Assignee: Guanghao Zhang > Attachments: HBASE-16947.patch > > > Recently we met too many replication WALs problem in our production cluster. > We need the DumpReplicationQueues tool to analyze the replication queues info > in zookeeper. So I backport HBASE-16450 to our branch based 0.98 and did some > improvements for it. > 1. Show the dead regionservers under replication/rs znode. When there are too > many WALs under znode, it can't be atomic transferred to new rs znode. So the > dead rs znode will be leaved on zookeeper. > 2. Make a summary about all the queues that belong to peer has been deleted. > 3. Aggregate all regionservers' size of replication queue. Now the > regionserver report ReplicationLoad to master, but there were not a aggregate > metrics for replication. > 4. Show how many WALs which can not found on hdfs. But the reason (WAL Not > Found) need more time to dig. -- This message was sent by Atlassian JIRA (v6.3.4#6332)