Duo Zhang created HBASE-21323: --------------------------------- Summary: The holdingCleanupTracker is not updated properly and causes too many procedure wal files Key: HBASE-21323 URL: https://issues.apache.org/jira/browse/HBASE-21323 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang
Keep seeing this {noformat} 2018-10-16,20:03:02,027 WARN [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=340 above the warning threshold 10. check running procedures to see if something is stuck. 2018-10-16,20:03:02,027 INFO [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=343 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,027 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,870 WARN [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=341 above the warning threshold 10. check running procedures to see if something is stuck. 2018-10-16,20:03:02,870 INFO [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=344 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:02,870 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:03,816 WARN [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=342 above the warning threshold 10. check running procedures to see if something is stuck. 2018-10-16,20:03:03,816 INFO [WALProcedureStoreSyncThread] org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=345 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=991, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=992, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=994, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. 2018-10-16,20:03:03,816 DEBUG [Force-Update-PEWorker-0] org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Procedure pid=995, ppid=990, state=SUCCESS; org.apache.hadoop.hbase.master.replication.RefreshPeerProcedure has already been finished, skip force updating. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)