I agree that it could be done more efficiently. Instead of sending files one-by-one, plugins could process WAL archive directory content (all files). For very large clusters this would save a lot of time for BackupLogCleaner, but because it is a background thread, the performance benefit is not clear for client applications. If we run cleaners once a day, it does not matter will they finish in 10 min or in 10 hours.
-Vlad On Fri, Jul 17, 2015 at 11:38 PM, rahul gidwani <[email protected]> wrote: > Do you think its time to stop sending files one by one to the cleaners and > have all the cleaners take batches of files (like replication and snapshot > cleaners)? As we add more cleaners for large clusters and certain > scenarios the cleaner chain comes to a crawl? > > On Fri, Jul 17, 2015 at 3:28 PM, Vladimir Rodionov (JIRA) <[email protected] > > > wrote: > > > > > [ > > > https://issues.apache.org/jira/browse/HBASE-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > > ] > > > > Vladimir Rodionov resolved HBASE-14036. > > --------------------------------------- > > Resolution: Implemented > > > > This feature is part of patch v4. See parent JIRA. > > > > > HBase Backup/Restore Phase 1: Custom WAL archive cleaner > > > -------------------------------------------------------- > > > > > > Key: HBASE-14036 > > > URL: https://issues.apache.org/jira/browse/HBASE-14036 > > > Project: HBase > > > Issue Type: Task > > > Reporter: Vladimir Rodionov > > > Assignee: Vladimir Rodionov > > > Fix For: 2.0.0 > > > > > > > > > Custom WAL archive cleaner (BackupLogCleaner). We need to keep WAL > > files in archive until they either get copied over to backup destination > > during an incremental backup or full backup (for ALL tables) happens. > This > > is tricky, but is doable. Backup-aware WAL archiver cleaner should > consult > > hbase:backup to determine if WAL file is safe to purge. > > > > > > > > -- > > This message was sent by Atlassian JIRA > > (v6.3.4#6332) > > >
