[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305037#comment-16305037 ]
stack commented on HBASE-18309: ------------------------------- Cleaner is also removing these: 2017-12-27 19:59:02,261 DEBUG [ForkJoinPool-1-worker-17] cleaner.CleanerChore: CleanerTask 391 starts cleaning dirs and files under hdfs://ve0524.halxg.cloudera.com:8020/hbase/oldWALs and itself. 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000001.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000002.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000003.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000004.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000005.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000006.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000007.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000008.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000009.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000010.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000011.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000012.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000013.log 2017-12-27 19:59:02,279 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000014.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000015.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000016.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000017.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000018.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000019.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000020.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000021.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000022.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000023.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000024.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000025.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000026.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000027.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000028.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000029.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000030.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000031.log 2017-12-27 19:59:02,280 DEBUG [ForkJoinPool-1-worker-17] master.ReplicationLogCleaner: Didn't find this log in ZK, deleting: pv2-00000000000000000032.log .... Is that right [~reidchan] ? (These are master procedure v2 logs that get moved to oldWALs when done...) > Support multi threads in CleanerChore > ------------------------------------- > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement > Reporter: binlijin > Assignee: Reid Chan > Fix For: 3.0.0, 2.0.0-beta-1 > > Attachments: HBASE-18309.addendum.patch, > HBASE-18309.master.001.patch, HBASE-18309.master.002.patch, > HBASE-18309.master.004.patch, HBASE-18309.master.005.patch, > HBASE-18309.master.006.patch, HBASE-18309.master.007.patch, > HBASE-18309.master.008.patch, HBASE-18309.master.009.patch, > HBASE-18309.master.010.patch, HBASE-18309.master.011.patch, > HBASE-18309.master.012.patch, space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. > What's more, currently there's only one thread iterating the archive > directory, and we could use multiple threads cleaning sub directories in > parallel to speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)