Shrinidhi Shridhar Talpankar created HBASE-30154:
----------------------------------------------------
Summary: HFiles Deleted While Still Referenced by HFileLinks in
Cloned Table
Key: HBASE-30154
URL: https://issues.apache.org/jira/browse/HBASE-30154
Project: HBase
Issue Type: Bug
Components: master, regionserver
Affects Versions: 2.6.2
Reporter: Shrinidhi Shridhar Talpankar
There is a race condition where Hfiles referenced by cloned table can be
permanently deleted by HFileLinkCleaner causing data loss. We've seen this
happening on HBase 2.6.2, but looking at the latest code it seems that the race
can still happen there.
The sequence of operations that leads to this issue:
# Regions merged in the parent table.
# Snapshot taken while a merged region still holds reference files, and that
snapshot is then cloned. RestoreSnapshotHelper.restoreReferenceFile() creates
HFileLink Reference files in the clone (e.g.
srcTable=srcRegion-hfile.cloneRegion) but does not write back-references to the
archive directory.
# Parent table compacted. CatalogJanitor GCs the pre-merge regions, archiving
their HFiles.
# HFileLinkCleaner sees no back-references for those archived HFiles and
deletes them.
# Every subsequent region open/scan on the cloned table fails with
FileNotFoundException.
Logs evidence supporting the above flow:
h4. 07:41:16Z — Merge 1 Initiated (RegionNormalizer)
The {{RegionNormalizerWorker}} identified regions {{e95408fd}} (4511 MB) and
{{565ae77f}} (128 MB) as candidates for merging due to size imbalance. It
submitted {{MergeTableRegionsProcedure}} (pid=2307105).
INFO [normalizer-worker-0] master.HMaster: merge regions
[e95408fd3383e18df6f3a171a6bfc05d, 565ae77f2c2e16d6ee539dd95540a240] INFO
[PEWorker-9] procedure.MasterProcedureScheduler: Took xlock for pid=2307105,
state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE
Source regions unassigned from regionserver-11 and regionserver-13.
h4. 07:41:18Z — Merge 1 Completes → {{43bb39bf}} created
INFO [PEWorker-14] procedure2.ProcedureExecutor: Finished pid=2307105,
state=SUCCESS; MergeTableRegionsProcedure table=aeris_v2,
regions=[e95408fd3383e18df6f3a171a6bfc05d, 565ae77f2c2e16d6ee539dd95540a240],
force=false in 2.0710 sec
Merged region {{43bb39bf414f2c2bb4c453a7c4ace246}} is assigned to
regionserver-11. Its store directory contains HFileLinks pointing back to the
source region directories.
h4. 09:41:16Z — Merge 2 Initiated (RegionNormalizer)
The normalizer identified regions {{449b39800}} (2964 MB) and {{c8bcf55f}}
(1694 MB) and submitted {{MergeTableRegionsProcedure}} (pid=2307118).
INFO [normalizer-worker-0] master.HMaster: merge regions
[449b39800c6efb9ce3eca36410d292c1, c8bcf55fde67e62655c51caec1a2ca96]
h4. 09:41:19Z — Merge 2 Completes → {{d5243cea}} created
INFO [PEWorker-9] procedure2.ProcedureExecutor: Finished pid=2307118,
state=SUCCESS; MergeTableRegionsProcedure table=aeris_v2,
regions=[449b39800c6efb9ce3eca36410d292c1, c8bcf55fde67e62655c51caec1a2ca96],
force=false in 2.7930 sec
Merged region {{d5243cea04f0a22e4292c1d006580385}} is assigned to
regionserver-88. Contains HFileLinks to {{449b39800}} and {{c8bcf55f}} source
HFiles.
h4. 14:02:59Z — Major Compaction Starts on {{d5243cea}} (regionserver-88)
The merged region immediately triggers a high-priority major compaction
(labeled as "recently split daughter region" — this is the HBase labeling for
merged regions). The compaction includes 9 files, among which are HFileLinks to
the source regions:
INFO [longCompactions-0] regionserver.HStore: Starting compaction of
d5243cea.../e...
hdfs://.../d5243cea.../e/d2d48db764e645d1ad5eb2bb1d1409c0.c8bcf55fde67e62655c51caec1a2ca96->...
hdfs://.../d5243cea.../e/3c9628b6aeaa4ae5944667d35ddf0fc3.c8bcf55fde67e62655c51caec1a2ca96->...
totalSize=4.6 G
*This compaction will take 44 minutes and 53 seconds.*
h4. ~14:24Z — ⚠️ Snapshot Taken WHILE Compaction is Running
A snapshot {{aeris_v2_triforce_snapshot_1773843841734812535}} is taken of
{{{}aeris_v2{}}}. The snapshot name encodes a Unix timestamp of approximately
{{1773843841}} seconds, corresponding to ~14:24 UTC March 18, 2026.
At this moment: - {{d5243cea}} still contains HFileLinks to
{{{}449b39800{}}}/{{{}c8bcf55f{}}} (compaction has NOT finished) - {{43bb39bf}}
still contains HFileLinks to {{{}e95408fd{}}}/{{{}565ae77f{}}}
The snapshot manifest captures this state — the merged regions' HFileLinks to
the source regions are part of the snapshot.
h4. 14:24:15Z - 14:28:05Z — Clone Created:
{{aeris_v2_triforce_archive_1773843841734812535}}
{{CloneSnapshotProcedure}} runs, invoking {{RestoreSnapshotHelper}} which
creates HFileLinks in the new archive table pointing directly to the source
region directories (the ones being cleaned up after merge). Key HFileLinks
added:
INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper: Adding HFileLink
37211e9b8f2744f3a7e2707594b39fbb.565ae77f2c2e16d6ee539dd95540a240 from cloned
region in snapshot aeris_v2_triforce_snapshot_1773843841734812535 to
table=aeris_v2_triforce_archive_1773843841734812535 INFO
[RestoreSnapshot-pool-7] snapshot.RestoreSnapshotHelper: Adding HFileLink
09aa8849c2d64984942a80b026e4a19e.449b39800c6efb9ce3eca36410d292c1 from cloned
region in snapshot aeris_v2_triforce_snapshot_1773843841734812535 to
table=aeris_v2_triforce_archive_1773843841734812535
Clone procedure completes at 14:28:05Z:
INFO [PEWorker-16] procedure.CloneSnapshotProcedure: Clone
snapshot=aeris_v2_triforce_snapshot_1773843841734812535 on
table=aeris_v2_triforce_archive_1773843841734812535 completed!
h4. 14:47:52Z — ⚠️ Compaction of {{d5243cea}} Completes
After 44 minutes and 53 seconds, the major compaction finishes:
INFO [longCompactions-0] regionserver.CompactSplit: Completed compaction
region=aeris_v2,...d5243cea04f0a22e4292c1d006580385.,
storeName=d5243cea04f0a22e4292c1d006580385/e, duration=44mins, 53sec INFO
[longCompactions-0] regionserver.HStore: Completed compaction of 9 (all)
file(s) in d5243cea04f0a22e4292c1d006580385/e into
9e904c6e8b2247cf9c0463aecc1d67b1 (size=4.4 G)
The HFileLinks to {{449b39800}} and {{c8bcf55f}} are *removed* from
{{{}d5243cea{}}}'s store directory. These source region files are no longer
referenced by any live region in {{{}aeris_v2{}}}. This is the gate condition
for {{GCMultipleMergedRegionsProcedure}} to proceed.
h4. 14:51:50Z — 💥 GCMultipleMergedRegionsProcedure Deletes Source Regions
(Group 2)
{{GCMultipleMergedRegionsProcedure}} (pid=2326702) runs immediately (only 93ms
total execution):
INFO [PEWorker-11] procedure.MasterProcedureScheduler: Took xlock for
pid=2326702, state=RUNNABLE:GC_MERGED_REGIONS_PREPARE;
GCMultipleMergedRegionsProcedure child=d5243cea04f0a22e4292c1d006580385,
parents:[449b39800c6efb9ce3eca36410d292c1], [c8bcf55fde67e62655c51caec1a2ca96]
INFO [PEWorker-4] hbase.MetaTableAccessor: Deleted
aeris_v2,...449b39800c6efb9ce3eca36410d292c1. INFO [PEWorker-11]
hbase.MetaTableAccessor: Deleted aeris_v2,...c8bcf55fde67e62655c51caec1a2ca96.
INFO [PEWorker-10] procedure2.ProcedureExecutor: Finished pid=2326702,
state=SUCCESS; GCMultipleMergedRegionsProcedure
child=d5243cea04f0a22e4292c1d006580385,
parents:[449b39800c6efb9ce3eca36410d292c1], [c8bcf55fde67e62655c51caec1a2ca96]
in 93 msec
The HDFS directories for {{449b39800c6efb9ce3eca36410d292c1}} and
{{c8bcf55fde67e62655c51caec1a2ca96}} — including both {{/hbase/data/}} and
{{/hbase/archive/}} paths — are deleted. The HFileLinks in
{{aeris_v2_triforce_archive}} now point to non-existent paths.
h4. 15:27:37Z — Compaction Triggered on {{43bb39bf}}
A flush on {{43bb39bf}} completes, requesting compaction:
INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of
43bb39bf414f2c2bb4c453a7c4ace246 in 73ms, compaction requested=true
h4. 17:15:36Z — First FileNotFoundException (Group 2: {{{}449b39800{}}})
{{aeris_v2_triforce_archive}} attempts to read through HFileLinks, fails:
java.io.FileNotFoundException: HFileLink locations=[
hdfs://aeris/hbase/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e,
hdfs://aeris/hbase/archive/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e,
hdfs://aeris/hbase/.tmp/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e,
hdfs://aeris/hbase/mobdir/data/default/aeris_v2/449b39800c6efb9ce3eca36410d292c1/e/09aa8849c2d64984942a80b026e4a19e]
All 4 candidate locations checked by {{HFileLink.open()}} are missing.
h4. 18:06:05Z — Compaction of {{43bb39bf}} Completes (regionserver-11)
INFO [longCompactions-0] regionserver.CompactSplit: Completed compaction
region=aeris_v2,...43bb39bf414f2c2bb4c453a7c4ace246., storeName=e, fileCount=6,
fileSize=4.5 G, duration=1mins, 35sec
HFileLinks to {{e95408fd}} and {{565ae77f}} are removed from {{{}43bb39bf{}}}.
GC gate condition met for Group 1.
h4. 18:11:52Z — 💥 GCMultipleMergedRegionsProcedure Deletes Source Regions
(Group 1)
INFO [PEWorker-13] hbase.MetaTableAccessor: Deleted
aeris_v2,...565ae77f2c2e16d6ee539dd95540a240. INFO [PEWorker-7]
hbase.MetaTableAccessor: Deleted aeris_v2,...e95408fd3383e18df6f3a171a6bfc05d.
INFO [PEWorker-12] procedure2.ProcedureExecutor: Finished pid=2357877,
state=SUCCESS; GCMultipleMergedRegionsProcedure
child=43bb39bf414f2c2bb4c453a7c4ace246,
parents:[e95408fd3383e18df6f3a171a6bfc05d], [565ae77f2c2e16d6ee539dd95540a240]
in 73 msec
HDFS directories for {{e95408fd3383e18df6f3a171a6bfc05d}} and
{{565ae77f2c2e16d6ee539dd95540a240}} are deleted.
h4. 21:16:49Z — First FileNotFoundException (Group 1: {{{}565ae77f{}}})
java.io.FileNotFoundException: HFileLink locations=[
hdfs://aeris/hbase/data/default/aeris_v2/565ae77f2c2e16d6ee539dd95540a240/e/37211e9b8f2744f3a7e2707594b39fbb,
...]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
