Dieter De Paepe created HBASE-28083:
---------------------------------------
Summary: Failed backups block backup merges
Key: HBASE-28083
URL: https://issues.apache.org/jira/browse/HBASE-28083
Project: HBase
Issue Type: Bug
Components: backup&restore
Environment: Encountered on HBase
a2e7d2015e9f603e46339d0582e29a86843b9324 (branch-2), running in Kubernetes.
Reporter: Dieter De Paepe
I was doing some experiments with HBase backup, and came to the following
situation:
{noformat}
user@hadoop-client-769bc9946-g6p59:/$ hbase backup history
{ID=backup_1694682979468,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start
time=Thu Sep 14 09:16:20 UTC 2023,End time=Thu Sep 14 09:16:45 UTC
2023,Progress=100%}
{ID=backup_1694682942306,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start
time=Thu Sep 14 09:15:43 UTC 2023,End time=Thu Sep 14 09:16:11 UTC
2023,Progress=100%}
{ID=backup_1694682906267,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start
time=Thu Sep 14 09:15:08 UTC 2023,End time=Thu Sep 14 09:15:34 UTC
2023,Progress=100%}
{ID=backup_1694682869991,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=COMPLETE,Start
time=Thu Sep 14 09:14:31 UTC 2023,End time=Thu Sep 14 09:14:58 UTC
2023,Progress=100%}
{ID=backup_1694681054166,Type=INCREMENTAL,Tables={foo:bar,foo:baz},State=FAILED,Start
time=Thu Sep 14 08:44:16 UTC 2023,Failed message=REPAIR status: repaired after
failure:
backup_1694681054166,Progress=0%}
{ID=backup_1694681006360,Type=FULL,Tables={foo:bar,foo:baz},State=COMPLETE,Start
time=Thu Sep 14 08:43:27 UTC 2023,End time=Thu Sep 14 08:43:56 UTC
2023,Progress=100%}
{noformat}
Even though I was able to create new incremental backups, I am not able to
merge the old backups, due to the failed (& repaired) backup:
{code:java}
# Failed incremental + completed incremental
user@hadoop-client-769bc9946-g6p59:/$ hbase backup merge
backup_1694681054166,backup_1694682869991
23/09/14 09:17:57 ERROR org.apache.hadoop.hbase.backup.BackupDriver: Error
running command-line tool
java.io.IOException: Backup image backup_1694681054166 can not be merged
becuase of its state: FAILED
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.checkIfValidForMerge(BackupAdminImpl.java:664)
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.mergeBackups(BackupAdminImpl.java:619)
at
org.apache.hadoop.hbase.backup.impl.BackupCommands$MergeCommand.execute(BackupCommands.java:834)
at
org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:134)
at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:169)
at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:199)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:177)
{code}
{code:java}
# Completed full backup + failed incremental
user@hadoop-client-769bc9946-g6p59:/$ hbase backup merge
backup_1694681006360,backup_1694681054166
23/09/14 09:19:40 ERROR org.apache.hadoop.hbase.backup.BackupDriver: Error
running command-line tool
java.io.IOException: FULL backup image can not be merged for:
backup_1694681006360
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.checkIfValidForMerge(BackupAdminImpl.java:659)
at
org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.mergeBackups(BackupAdminImpl.java:619)
at
org.apache.hadoop.hbase.backup.impl.BackupCommands$MergeCommand.execute(BackupCommands.java:834)
at
org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:134)
at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:169)
at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:199)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:177)
{code}
I'm not sure why merging a failed backup is not possible, given that it is
possible to create new incremental backups on top of it. Either the data in
that failed backup is reliable (meaning it should be merge-able), or not
reliable (meaning it should not be possible to execute incremental backups on
top of it).
If I were to delete the failed backup, all other incremental backups would also
be deleted. This made me assume failed backups still keep track of reliable
data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
