[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247086#comment-16247086
]
Reid Chan commented on HBASE-18309:
-----------------------------------
New simulation first creates 10 dirs under oldWALs (hbase.rootdir not tmp this
time), then at least 10 subdirs, at least 10 files under one subdir, file's
size is between 1M~10M. The reason of "at least" is to make size different
between each other by adding some random factor, and for sorting.
Repeat 3 more times. That's to say, 10 dirs under oldWALs, each dir contains at
least 1110 subdirs and 1110 files.
Time is measured when (sub)dirs and files are cleaned up. Following is the
result table(in ms), FYI [~tedyu], [~busbey], [~carp84]:
||Test case||avg||min||max||
|original version|31672.2|22877|36676|
|4 threads with sorting|21200.2|15815|28060|
|4 threads without sorting|19123.2|15269|21968|
Since there's not much improvement on not sorting, i will keep the sorting in
next patch if folks think it should.
[~carp84]
bq. if not configured, the implementation in patch here cannot speed up the
deletion of oldWALs
Yes, i will add the multi threads version to delete oldWALs as well, thanks for
reminder. Do you think it is ok to borrow the idea of {{HFileCleaner}}? Or a
general way would work?
bq. It should be ok that the two policies apply in combination or separately
through configuration, but we should document it clearly so user won't
misunderstand and cause unnecessary thread flood in production.
Yes, i'm trying to combine them.
bq. thread pool size online configurable is still valid
Marked, still working on that.
> Support multi threads in CleanerChore
> -------------------------------------
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Reporter: binlijin
> Assignee: Reid Chan
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
