[
https://issues.apache.org/jira/browse/HBASE-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Liu resolved HBASE-30525.
------------------------------
Resolution: Duplicate
> MOB cleaner archives MOB files still referenced after region split or merge
> ---------------------------------------------------------------------------
>
> Key: HBASE-30525
> URL: https://issues.apache.org/jira/browse/HBASE-30525
> Project: HBase
> Issue Type: Bug
> Components: mob
> Reporter: Xiao Liu
> Priority: Critical
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.1.0, 3.0.1, 2.6.8, 2.5.17
>
>
> {\{MobFileCleanupUtil}} is run by the master \{{MobFileCleanerChore}}. It
> archives a MOB file when two conditions hold:
> * the region named in the file name suffix no longer exists in the table
> directory;
> * the file is older than \{{hbase.mob.min.age.archive}}.
> It collects the MOB files referenced by the current store files
> (\{{allActiveMobFileName}}), but only logs them. HBASE-26969 replaced the
> reference check with the region check.
> After a split or merge, the store files of the new regions still reference
> the MOB files of the parent regions. The compaction that removes the
> reference files is a system compaction, and a system compaction copies MOB
> references unchanged. Only a user major compaction rewrites the MOB cells,
> for example one run by \{{MobFileCompactionChore}}. Split, merge and the
> parent region GC do not handle MOB files.
> The parent MOB files are archived when all of the following hold:
> * the parent region directory has been GCed;
> * the new regions still reference the parent MOB files;
> * the files are older than \{{hbase.mob.min.age.archive}} (1 hour by default);
> * the cleaner runs before the MOB cells are rewritten.
> With the default settings this is likely: the cleaner runs daily, while the
> MOB compaction chore runs weekly. In I/O optimized mode, MOB files larger
> than \{{hbase.mob.compactions.max.file.size}} are never rewritten, so the new
> regions keep referencing them indefinitely.
> The HFileCleaner deletes the archived files after
> \{{hbase.master.hfilecleaner.ttl}} (5 minutes by default). After that, reads
> of the affected rows fail with \{{DoNotRetryIOException}} caused by
> \{{FileNotFoundException}}. MOB compactions of the new regions fail as well.
> Steps to reproduce, as in the attached test:
> # Write and flush MOB data into a single region table.
> # Split the region, and wait for the daughters to compact away their
> reference files.
> # Run the CatalogJanitor so that it GCs the parent region.
> # Run \{{MobFileCleanupUtil.cleanupObsoleteMobFiles}}, then the HFileCleaner.
> # Scan the table: it fails with the exception above.
> Merging two regions leads to the same result.
> The fix is to archive a MOB file only when it is also not referenced by any
> store file.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
