Dieter De Paepe created HBASE-30542:
---------------------------------------

             Summary: Recreating a table can corrupt incremental backups
                 Key: HBASE-30542
                 URL: https://issues.apache.org/jira/browse/HBASE-30542
             Project: HBase
          Issue Type: Bug
          Components: backup&restore
            Reporter: Dieter De Paepe


**Summary:** Incremental backup after a table is deleted and recreated restores 
data of the deleted table

```
h3. Problem

The backup system does not notice when a table is deleted and recreated with 
the same name. The next incremental backup treats the new table as a 
continuation of the old one, so restoring that backup image restores data of 
the deleted table.

Found while working on HBASE-29905, by code inspection. I have not reproduced 
this with a test yet.

h3. Scenario

# A full backup F1 includes table T in backup root R. This adds T to the 
incremental backup set of R (\{{incrbackupset:}} row) and stores the WAL 
timestamps of T in its \{{trslm:}} row.
# Rows are written to T.
# T is deleted. \{{BackupMasterObserver#postDeleteTable}} only removes bulk 
load rows, so T stays in the incremental backup set of R and its \{{trslm:}} 
row stays as well.
# T is recreated with the same name and the same column families, and new rows 
are written to it.
# An incremental backup I1 runs on R. Incremental backups cover the whole 
incremental backup set, and T exists again, so I1 includes T. The column family 
check in \{{IncrementalTableBackupClient}} only compares column family names, 
so it passes.
# T is restored from I1.

h3. Expected

Either the restored T only contains the rows of the recreated table, or I1 
refuses to back up T until a new full backup of T is taken.

h3. Actual

Restoring I1 starts from the snapshot of the deleted table in F1 and then 
applies the incremental data. Deleting a table produces no delete markers, so 
the restored T contains:
* all rows of the deleted table from F1,
* the edits made to the deleted table between F1 and its deletion, which I1 
reads from the WALs because the \{{trslm:}} timestamps of T still point to F1,
* the rows of the recreated table.

If the recreated table has different column families, I1 fails with 
\{{ColumnFamilyMismatchException}} instead, so that variant does not corrupt 
data.

{\{postTruncateTable}} has the same gap: truncating also removes data without 
delete markers. I have not verified the truncate case.

h3. Possible fix

Treat a deleted table as no longer backed up. In 
\{{BackupMasterObserver#postDeleteTable}}, remove the table from the 
incremental backup set of every backup root and delete its \{{trslm:}} rows. 
The next incremental backup then skips the recreated table, and a request that 
names it explicitly fails with "haven't gone through full backup". The existing 
backups of the deleted table stay restorable: restore reads the backup images 
and manifests, and point-in-time restore reads the backup history and the 
continuous backup table set, so neither uses these rows. As a side effect, this 
also removes the stale \{{trslm:}} rows that HBASE-29905 works around in 
BackupLogCleaner.
```



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to