Hernan Romer created HBASE-30499:
------------------------------------
Summary: A bypassed SnapshotProcedure leaves its tmp working
directory behind, which pins archived HFiles until the HMaster restarts
Key: HBASE-30499
URL: https://issues.apache.org/jira/browse/HBASE-30499
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 3.0.0, 2.6.0, 2.0.0
Reporter: Hernan Romer
Bypassing a SnapshotProcedure before SNAPSHOT_COMPLETE_SNAPSHOT skips both the
commit and the rollback. The directory <snapshot dir>/.tmp/<snapshot name>
stays on disk, and nothing in the running master tracks it any more.
list_snapshots and list_procedures don't show it, and deleteSnapshot doesn't
remove it. The only cleanup is SnapshotManager#resetTempDir(), which only runs
when the master starts.
Until then, SnapshotFileCache#getSnapshotsInProgress() treats the directory as
a live snapshot and keeps every HFile its manifest references. As the table
compacts, those files pile up in the archive.
Consequence: the archive can grow to roughly the size the table had when the
procedure was bypassed, and stays there until the master restarts. The only
hint is CleanerChore TRACE logging not deletable according to:
SnapshotHFileCleaner, which doesn't name the snapshot. We hit this in
production: the master stayed up for about a month after a bypass, and the
archive grew to about the size of the table's live data.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
