Xiao Liu created HBASE-30525:
--------------------------------
Summary: MOB cleaner archives MOB files still referenced after
region split or merge
Key: HBASE-30525
URL: https://issues.apache.org/jira/browse/HBASE-30525
Project: HBase
Issue Type: Bug
Components: mob
Reporter: Xiao Liu
Fix For: 4.0.0-alpha-1, 2.7.0, 3.1.0, 3.0.1, 2.6.8, 2.5.17
{\{MobFileCleanupUtil}} is run by the master \{{MobFileCleanerChore}}. It
archives a MOB file when two conditions hold:
* the region named in the file name suffix no longer exists in the table
directory;
* the file is older than \{{hbase.mob.min.age.archive}}.
It collects the MOB files referenced by the current store files
(\{{allActiveMobFileName}}), but only logs them. HBASE-26969 replaced the
reference check with the region check.
After a split or merge, the store files of the new regions still reference the
MOB files of the parent regions. The compaction that removes the reference
files is a system compaction, and a system compaction copies MOB references
unchanged. Only a user major compaction rewrites the MOB cells, for example one
run by \{{MobFileCompactionChore}}. Split, merge and the parent region GC do
not handle MOB files.
The parent MOB files are archived when all of the following hold:
* the parent region directory has been GCed;
* the new regions still reference the parent MOB files;
* the files are older than \{{hbase.mob.min.age.archive}} (1 hour by default);
* the cleaner runs before the MOB cells are rewritten.
With the default settings this is likely: the cleaner runs daily, while the MOB
compaction chore runs weekly. In I/O optimized mode, MOB files larger than
\{{hbase.mob.compactions.max.file.size}} are never rewritten, so the new
regions keep referencing them indefinitely.
The HFileCleaner deletes the archived files after
\{{hbase.master.hfilecleaner.ttl}} (5 minutes by default). After that, reads of
the affected rows fail with \{{DoNotRetryIOException}} caused by
\{{FileNotFoundException}}. MOB compactions of the new regions fail as well.
Steps to reproduce, as in the attached test:
# Write and flush MOB data into a single region table.
# Split the region, and wait for the daughters to compact away their reference
files.
# Run the CatalogJanitor so that it GCs the parent region.
# Run \{{MobFileCleanupUtil.cleanupObsoleteMobFiles}}, then the HFileCleaner.
# Scan the table: it fails with the exception above.
Merging two regions leads to the same result.
The fix is to archive a MOB file only when it is also not referenced by any
store file.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
