[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245467#comment-16245467
]
Yu Li commented on HBASE-18309:
-------------------------------
bq. I may misunderstand binlijin's original intent, which is to focus on
deletion files as HFileCleaner does
Yes, it's different, in our case there's no HBASE-14247 and no subdir in
oldWALs, only files. Please note that HBASE-14247 only added an option to use
different directories for oldWALs, and if not configured, the implementation in
patch here cannot speed up the deletion of oldWALs
bq. I'd suggest you take a look at the patch.
OK, checked the patch and got your point, mainly using one thread to scan and
clean a (sub)directory (if still free thread in pool). It applies for both
hfile and wal after HBASE-14247 (if configured). But it should still follow the
previous logic of {{sortByConsumedSpace}} to delete from the large files.
Please check more carefully on the methods removed in patch.
The main difference between your idea and ours is that you choose to use
multiple threads to scan the directories and we delete files. It should be ok
that the two policies apply in combination or separately through configuration,
but we should document it clearly so user won't misunderstand and cause
unnecessary thread flood in production. Not sure whether you also applied
HBASE-18083 online and checked the pressure on NN, it would be good if we could
give some recommended values on configuration of the combination.
Good on you [~reidchan] and better if could consider more about existing impl
in trunk. And the suggestion of making thread pool size online configurable is
still valid (smile).
> Support multi threads in CleanerChore
> -------------------------------------
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Reporter: binlijin
> Assignee: Reid Chan
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
