Sid Khillon created HBASE-30535:
-----------------------------------
Summary: BackupLogCleaner keeps WALs forever after a table with
its own backup root is deleted
Key: HBASE-30535
URL: https://issues.apache.org/jira/browse/HBASE-30535
Project: HBase
Issue Type: Bug
Components: backup&restore
Affects Versions: 2.6.5, 3.0.0
Reporter: Sid Khillon
Assignee: Sid Khillon
*Problem*
If a table that has backups is deleted, and it is the only table in its backup
root, BackupLogCleaner never lets any WAL newer than that table's last backup
be deleted. oldWALs grows without limit.
*Cause*
BackupLogCleaner takes the newest completed backup in each backup root and
keeps every WAL newer than the oldest of those. It does not check whether the
tables in that backup still exist. After the table is deleted, its root can
never take another backup, so that root's newest backup never moves forward and
it holds back every WAL written after it.
Nothing else cleans this up: BackupMasterObserver#postDeleteTable only removes
the table's bulk-load rows. The only way out today is "hbase backup delete" on
the deleted table's backups, which also deletes the backup data.
*What we saw*
We back up each table into its own root. One table was deleted a few hours
after its last backup. From then on, no WAL on the cluster could be cleaned,
even though every other table kept taking full and incremental backups. oldWALs
grew from about 0.3 TB to 1.2 TB in 4 days, and incremental backups got slower
because each one read all of those WALs.
*Steps to reproduce*
1. Take a FULL backup of table A into root R1.
2. Take a FULL backup of table B into root R2.
3. Delete table B.
4. Take an INCREMENTAL backup of table A into R1.
5. Ask BackupLogCleaner which WALs are deletable.
Expected: every WAL from before step 4 is deletable.
Actual: WALs written after step 2 are kept, and stay kept forever.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
