[ https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948201#comment-15948201 ]
huaxiang sun commented on HBASE-17215: -------------------------------------- Thanks [~carp84] for the patch! The patch looks great. I went through the patch and left couple comments there. Maybe as a followup, can these two threads share a LinkedBlockingDeque? So large-size file is added to the head of the queue and small-size files are added to the tail. the large-file-thread poll from the head and the small-file-thread poll from the end. In this sense, no thread will be idle when there are works to do. What do you think? > Separate small/large file delete threads in HFileCleaner to accelerate > archived hfile cleanup speed > --------------------------------------------------------------------------------------------------- > > Key: HBASE-17215 > URL: https://issues.apache.org/jira/browse/HBASE-17215 > Project: HBase > Issue Type: Improvement > Reporter: Yu Li > Assignee: Yu Li > Attachments: HBASE-17215.patch > > > When using PCIe-SSD the flush speed will be really quick, and although we > have per CF flush, we still have the > {{hbase.regionserver.optionalcacheflushinterval}} setting and some other > mechanism to avoid data kept in memory for too long to flush small hfiles. In > our online environment we found the single thread cleaner kept cleaning > earlier flushed small files while large files got no chance, which caused > disk full then many other problems. > Deleting hfiles in parallel with too many threads will also increase the > workload of namenode, so here we propose to separate large/small hfile > cleaner threads just like we do for compaction, and it turned out to work > well in our cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)