[ 
https://issues.apache.org/jira/browse/HBASE-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dieter De Paepe updated HBASE-30542:
------------------------------------
    Affects Version/s: 3.0.0
                       2.6.0

> Recreating a table can corrupt incremental backups
> --------------------------------------------------
>
>                 Key: HBASE-30542
>                 URL: https://issues.apache.org/jira/browse/HBASE-30542
>             Project: HBase
>          Issue Type: Bug
>          Components: backup&restore
>    Affects Versions: 2.6.0, 3.0.0
>            Reporter: Dieter De Paepe
>            Priority: Critical
>
> Incremental backup after a table is deleted and recreated restores data of 
> the deleted table
> h3. Problem
> The backup system does not notice when a table is deleted and recreated with 
> the same name. The next incremental backup treats the new table as a 
> continuation of the old one, so restoring that backup image restores data of 
> the deleted table.
> Found while working on HBASE-29905, by code inspection. I have not reproduced 
> this with a test yet.
> h3. Scenario
>  # A full backup F1 includes table T in backup root R. This adds T to the 
> incremental backup set of R ({{{}incrbackupset:{}}} row) and stores the WAL 
> timestamps of T in its {{trslm:}} row.
>  # Rows are written to T.
>  # T is deleted. {{BackupMasterObserver#postDeleteTable}} only removes bulk 
> load rows, so T stays in the incremental backup set of R and its {{trslm:}} 
> row stays as well.
>  # T is recreated with the same name and the same column families, and new 
> rows are written to it.
>  # An incremental backup I1 runs on R. Incremental backups cover the whole 
> incremental backup set, and T exists again, so I1 includes T. The column 
> family check in {{IncrementalTableBackupClient}} only compares column family 
> names, so it passes.
>  # T is restored from I1.
> h3. Expected
> Either the restored T only contains the rows of the recreated table, or I1 
> refuses to back up T until a new full backup of T is taken.
> h3. Actual
> Restoring I1 starts from the snapshot of the deleted table in F1 and then 
> applies the incremental data. Deleting a table produces no delete markers, so 
> the restored T contains:
>  * all rows of the deleted table from F1,
>  * the edits made to the deleted table between F1 and its deletion, which I1 
> reads from the WALs because the {{trslm:}} timestamps of T still point to F1,
>  * the rows of the recreated table.
> If the recreated table has different column families, I1 fails with 
> {{ColumnFamilyMismatchException}} instead, so that variant does not corrupt 
> data.
> {{postTruncateTable}} has the same gap: truncating also removes data without 
> delete markers. I have not verified the truncate case.
> h3. Possible fix
> Treat a deleted table as no longer backed up. In 
> {{{}BackupMasterObserver#postDeleteTable{}}}, remove the table from the 
> incremental backup set of every backup root and delete its {{trslm:}} rows. 
> The next incremental backup then skips the recreated table, and a request 
> that names it explicitly fails with "haven't gone through full backup". The 
> existing backups of the deleted table stay restorable: restore reads the 
> backup images and manifests, and point-in-time restore reads the backup 
> history and the continuous backup table set, so neither uses these rows. As a 
> side effect, this also removes the stale {{trslm:}} rows that HBASE-29905 
> works around in BackupLogCleaner.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to