[
https://issues.apache.org/jira/browse/HBASE-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750666#comment-13750666
]
Dave Latham commented on HBASE-9208:
------------------------------------
Thanks for the attention Stack, and JD.
{quote}Trunk patch looks good. Does what you have deployed look like this patch
Dave Latham?{quote}
What we deployed is a bit different. Added a startBatch and endBatch to the
FileCleanerDelegate. That made it a bit less clean of an interface, which also
has some hidden dependencies on client behavior (otherwise would have race
conditions). However it was a smaller change that we needed rapidly that
didn't require a refactor of CleanerChore like this patch does. I like this
patch better and look forward to running it if it looks good to others.
{quote}It doesn't have unit tests. You are relying on existing tests to ensure
that base functionality remains undisturbed?{quote}
That's right.
{quote}Rather than indent whole method, just reverse this test and return
early: if (entries != null) { {quote}
I guess it depends on whether you prefer having indentation or an extra return
statement. I'll add the extra return since you spoke up.
{quote}
Is this right? Delete all files if not configured?
...
What it means is that this log cleaner says ok for those files, but other log
cleaners can say otherwise. As long as one cleaner in the chain says "keep it",
we keep it.
But I'm not sure about this WARN, it could just be removed:
LOG.warn("Not configured - allowing all hlogs to be deleted");
{quote}
This case existed in the previous code with the same behavior that JD explains.
However it can only occur if someone manually configures HBase to use the
ReplicationLogCleaner but has replication disabled (normally the
ReplicationLogCleaner is added programmatically in
Replication.decorateMasterConfiguration when replication is enabled). The
warning could prove useful to someone in such a case. However I will move it
to the setConf method so it is only logged once at startup if there is concern
over its level and verbosity.
> ReplicationLogCleaner slow at large scale
> -----------------------------------------
>
> Key: HBASE-9208
> URL: https://issues.apache.org/jira/browse/HBASE-9208
> Project: HBase
> Issue Type: Improvement
> Components: Replication
> Reporter: Dave Latham
> Assignee: Dave Latham
> Fix For: 0.94.12, 0.96.0
>
> Attachments: HBASE-9208-0.94.patch, HBASE-9208.patch,
> HBASE-9208-v2.patch
>
>
> At a large scale the ReplicationLogCleaner fails to clean up .oldlogs as fast
> as the cluster is producing them. For each old HLog file that has been
> replicated and should be deleted the ReplicationLogCleaner checks every
> replication queue in ZooKeeper before removing it. This means that as a
> cluster scales up the number of files to delete scales as well as the time to
> delete each file so the cleanup chore scales quadratically. In our case it
> reached the point where the oldlogs were growing faster than they were being
> cleaned up.
> We're now running with a patch that allows the ReplicationLogCleaner to
> refresh its list of files in the replication queues from ZooKeeper just once
> for each batch of files the CleanerChore wants to evaluate.
> I'd propose updating FileCleanerDelegate to take a List<FileStatus> rather
> than a single one at a time. This would allow file cleaners that check an
> external resource for references such as ZooKeeper (for
> ReplicationLogCleaner) or HDFS (for SnapshotLogCleaner which looks like it
> may also have similar trouble at scale) to load those references once per
> batch rather than for every log.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
