[
https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16242675#comment-16242675
]
Sean Busbey commented on HBASE-18309:
-------------------------------------
bq. In production, the number of directories would be greater than the number
of threads.
this. much much greater.
for a reasonable test please use a larger scale and include your reasoning, 10
doesn't seem like enough to simulate what will happen in a deployment. e.g. X
regions per server, Y servers means Z directories to clean up.
when do we worry about perf across 1000 directories? 10000?
At what point will tuning this parameter cause a NameNode to fall over? How do
we stop folks from doing that accidentally?
{code}
52 private static final String CLEANER_POOL_SIZE =
"hbase.cleaner.threadpool.size";
53 private static final String DEFAULT_CLEANER_POOL_SIZE = "0.5";
{code}
this config needs to be documented.
{code}
102 * Calculate size for cleaner pool.
103 * If poolSize >= 1, it would be the size of pool;
104 * if 0.0 < poolSize <= 1.0, size of pool would be available
processors * poolSize.
105 * @param poolSize size from configuration
106 * @return size of pool after calculation
{code}
These details should probably be in the documentation about the config.
> Support multi threads in CleanerChore
> -------------------------------------
>
> Key: HBASE-18309
> URL: https://issues.apache.org/jira/browse/HBASE-18309
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Reporter: binlijin
> Assignee: Reid Chan
> Attachments: HBASE-18309.master.001.patch,
> HBASE-18309.master.002.patch
>
>
> There is only one thread in LogCleaner to clean oldWALs and in our big
> cluster we find this is not enough. The number of files under oldWALs reach
> the max-directory-items limit of HDFS and cause region server crash, so we
> use multi threads for LogCleaner and the crash not happened any more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
